When a team tells us reads are fine at p50 and terrible at p99, the disk is the suspect and the JVM is usually the culprit. Cassandra is a Java process holding a large heap under sustained allocation pressure; every stop-the-world pause shows up as a coordinator timeout, a dropped mutation, or — at the far tail — a node flapping out of the cluster because gossip missed its window. This post is the tuning we actually apply, and the order we apply it in.
A note on scope: nothing here rescues a bad data model. If reads scan 40,000 tombstones or partitions are 800 MB, no collector will save you. Fix shape first, then JVM.
Sizing the heap
The defaults in jvm-server.options and cassandra-env.sh are conservative and workload-blind. Our starting points on a modern node with 64–128 GB of RAM:
- G1, 16–31 GB heap. Stay under the ~32 GB threshold where the JVM drops compressed object pointers; crossing it costs you effective heap for nothing. 16 GB is a reasonable floor for a busy node, 31 GB a practical ceiling.
- Set
-Xmsequal to-Xmx. A growing heap means repeated resizing and full-GC surprises under load. Fix it. - Leave real memory for the page cache. Cassandra reads go through the OS page cache; a heap that consumes 70% of RAM converts cheap cached reads into disk reads. Half of RAM to the heap is already aggressive.
- Off-heap matters as much as heap. Bloom filters, compression metadata, index summaries, and (depending on configuration) memtables live outside the heap. Budget them. A node that OOMs with a half-empty heap is almost always an off-heap accounting failure, frequently bloom filters on tables with billions of small rows.
If you only change one thing on an untuned cluster, make it heap size plus -Xms = -Xmx.
G1 or ZGC
Cassandra 5 runs on JDK 11 and JDK 17, and JDK 17 is where the interesting collectors live. The honest trade-off:
G1 is the safe default. It is what most production Cassandra runs, what most tuning advice assumes, and what your vendor will ask about first. With a 16–31 GB heap, -XX:MaxGCPauseMillis=300 (not 50 — unreachable targets make G1 shrink young gen and collect constantly), and otherwise stock settings, G1 produces predictable tens-of-milliseconds young collections and occasional longer mixed ones.
ZGC is the tail-latency play. Generational ZGC on JDK 17+ targets sub-millisecond pauses and scales past 32 GB heaps without the compressed-pointer cliff. When p99.9 read latency is the product requirement and you have CPU headroom to spare — ZGC trades throughput and extra cores for pause time — it is worth a controlled test. Two cautions: give it more heap than you gave G1 (ZGC needs allocation headroom; starving it produces allocation stalls that look worse than the pauses you were fixing), and treat it as a change you benchmark on one rack before it becomes cluster policy.
Parallel GC still wins on pure throughput for batch-ingest-only clusters where nobody serves interactive reads. That is a narrow case, and if you are not sure it is yours, it is not.
What we do not recommend: copying a long list of -XX: flags from a blog post. Most per-region and per-thread G1 flags in circulation were tuned for JDK 8 heaps and now fight the collector's own ergonomics. Change heap size, collector, and pause target; measure; stop.
Memtables, and what Cassandra 5 changed
Memtable allocation is the dominant source of heap churn on a write-heavy node. Three settings govern it: memtable_allocation_type, memtable_heap_space, and memtable_offheap_space.
For high-write workloads, offheap_objects moves the bulk of memtable data out of the heap, so GC has far less to trace. That has been the standard recommendation for years and it still holds.
Cassandra 5 adds a better option for many tables: trie memtables (memtable_allocation_type: trie via a memtable configuration, paired with the BTI SSTable format). The trie structure stores partition and clustering prefixes once instead of repeatedly, which in practice means substantially more rows resident per byte of memory and markedly less garbage produced per write. The operational effects we care about: fewer flushes for the same ingest rate, larger and better-organized SSTables, and less GC pressure to tune around in the first place.
Trie memtables are opt-in per table, and BTI is a separate sstable_format decision that applies to newly written SSTables. That makes them a good candidate for exactly the migration pattern we like — enable on one high-write table, watch flush frequency, GC time, and write latency for a week, then widen. Do not flip both settings cluster-wide during an upgrade window; you will not be able to attribute what changed.
Proving a change worked
GC tuning invites superstition, so hold every change to the same evidence bar. Before and after, from the same load profile:
- GC time per minute and pause distribution, not just pause count. Export
jvm.gc.*metrics; read the p99 pause, not the mean. - Client-visible p99 and p99.9 read latency per table (
nodetool tablehistograms), compared against coordinator latency to separate replica pauses from queueing. - Dropped mutations and hints created. These rise when nodes go unresponsive long enough for peers to notice, which is the failure mode a pause becomes at scale.
- Flush frequency and pending flushes (
nodetool tpstats) — the metric that tells you whether a memtable change helped. - Allocation stalls, specifically, if you are testing ZGC.
Enable GC logging permanently (-Xlog:gc*:file=... with rotation). It costs almost nothing and it is the difference between diagnosing a pause storm from evidence and guessing at it after the fact.
Roll changes one rack at a time with a canary node, and keep one node on the old settings long enough to compare like-for-like under real traffic. Restarting every node in an hour to apply a flag you have not validated is how a tuning exercise becomes an incident.
The short version
Set -Xms = -Xmx, keep the heap under 32 GB on G1, leave RAM for the page cache, account for off-heap, and move memtables off heap. Consider generational ZGC when tail latency is a product requirement and you have cores to spend. Evaluate trie memtables and BTI per table on Cassandra 5 — they attack the allocation problem at the source rather than asking the collector to clean up faster. Then stop tuning and go fix the data model, because that is where the larger win usually is.
If you are staring at unexplained p99 spikes and want a second pair of eyes on the heap dumps and GC logs, that is the kind of question our cluster review is built around.