When we audit a struggling cluster, the first two questions are always the same: when did repairs last complete on every table, and is compaction keeping up. The answers predict most of what the rest of the review will find. Repairs and compaction are Cassandra's housekeeping — invisible when healthy, and the root cause behind a remarkable share of "sudden" incidents when not.
Why repair is not optional
Cassandra accepts writes when replicas are down; that's the availability promise. Hinted handoff patches short outages, but hints expire (default window: three hours). Past that, the only mechanism that reconciles replicas is anti-entropy repair — the Merkle-tree comparison you trigger with nodetool repair or schedule with a tool.
Two consequences make repair a hard requirement, not a nice-to-have:
- Consistency drift. Un-repaired replicas diverge, and reads at
LOCAL_QUORUMcan return different results depending on which replicas answer. - The tombstone contract. Compaction may only purge a tombstone after
gc_grace_seconds(default ten days) — a grace period that exists specifically to give repair time to propagate the delete to every replica. If repairs don't complete withingc_grace_seconds, deleted data can resurrect: a replica that missed the delete re-spreads the "live" row after the tombstone is purged elsewhere. Resurrection bugs are among the most confusing incidents Cassandra can produce — records customers deleted reappearing weeks later.
So the rule is mechanical: every table, fully repaired, on a cycle strictly shorter than gc_grace_seconds. Ten-day grace means a repair cycle of at most about a week, with margin for failures and reruns.
In practice that means incremental repairs on a schedule (or a repair orchestrator — rolling your own cron rarely survives contact with node replacements), staggered so validation compaction doesn't stack up across the cluster, and monitored for completion — a repair that starts nightly and silently fails nightly is the most common version of "we run repairs" we encounter. Track repair age per table as a first-class metric with an alert well inside the grace window.
Compaction: the write path's other half
Every flush writes an immutable SSTable; compaction merges them so reads touch few files. When compaction keeps up, reads are cheap. When it falls behind, each read consults more SSTables, page cache efficiency drops, and latency climbs — gradually, then suddenly.
Watch three numbers per node:
- Pending compactions (
nodetool compactionstats). Sustained pending counts that grow through your daily peak and never drain are the classic early warning — weeks of warning, usually, before users notice. - SSTables per read (
nodetool tablehistograms): the p95 should be a small single digit for most workloads. Rising SSTables-per-read is compaction debt measured where it hurts. - Disk headroom. Size-tiered compaction can transiently need free space on the order of the data it's merging. Nodes past ~60–70% disk on STCS tables are running out of room to fix themselves.
When compaction lags chronically, resist the urge to simply raise compaction_throughput and walk away. Find out why: an over-aggressive flush cadence, a strategy mismatched to the workload (time-series on STCS instead of TWCS is the perennial offender), too few compaction threads for the hardware, or a burst of upgradesstables/repair-triggered work competing for the same I/O. Throughput raises help when I/O headroom exists; on saturated disks they just move the bottleneck.
Strategy fit is hygiene too, revisited as workloads drift: STCS for general write-heavy tables, LCS where read latency justifies its write amplification, TWCS for TTL'd time-series — and on Cassandra 5, UCS consolidates these trade-offs into one tunable strategy worth evaluating per table.
The hygiene dashboard
If you build one dashboard for a Cassandra cluster, put these on it: repair age per table (alert < gc_grace_seconds with margin) · pending compactions per node · SSTables-per-read p95 · tombstones scanned per read · disk usage per node · dropped mutations · hint accumulation. Every one of these predicts an incident category; together they are most of the difference between operating Cassandra and being surprised by it.
None of this work is glamorous, which is exactly why it decays — the engineer who set it up leaves, an alert gets silenced during an unrelated incident, and eighteen months later the cluster is a mystery again. That maintenance-of-the-maintenance is precisely what our hourly operations retainer exists to hold steady; whether you run the chores or we do, make sure someone provably does.