Sizing questions usually arrive in one line: "we have about 40 TB of telemetry, how many nodes?" That number is the least interesting input. Raw data size tells you almost nothing until you have multiplied it by replication, added compaction headroom, subtracted the capacity you are holding back for a dead node, and checked that one node can still bootstrap in a time you are willing to wait for.
This is the arithmetic we walk through on design-and-build engagements, in the order we walk through it.
Step 1: replicated bytes, not logical bytes
Start with the on-disk size of one copy of your data — after compression. Cassandra compresses SSTables by default, and for telemetry and JSON-ish payloads a 2x to 4x ratio is common. Measure it; do not guess. Load a representative day into a single-node test instance and read nodetool tablestats for compressed size and compression ratio.
Then multiply:
replicated_bytes = compressed_bytes_per_copy x RF x number_of_datacenters
RF=3 in two datacenters means six copies. The 40 TB question is really a 240 TB question, and that is before any headroom. Teams routinely forget the second datacenter in the budget, then discover it when someone asks for a DR region.
Step 2: compaction and repair headroom
Compaction rewrites SSTables, which means it needs free space to write the output before it can delete the input. How much depends on strategy:
- STCS is the greedy one. A major compaction of a large table can transiently need space equal to the table itself. In practice, plan for substantial free space and avoid major compactions on big tables entirely.
- LCS is well behaved — it works in small fixed-size sstables, so transient overhead is modest, but it costs far more write amplification.
- UCS (Cassandra 5's unified strategy) lets you pick a point on that curve per table, and is the one we now default to for new builds.
Repair adds its own transient cost: streamed partitions land as new SSTables before compaction merges them away, and a repair of a badly diverged table can add a double-digit percentage to a node's footprint for hours.
Our working rule: keep steady-state disk utilization at or below 50-60% per node, with 70% as an alarm threshold and 80% as an incident. That is not conservatism for its own sake. A full disk on a Cassandra node is not a degraded node, it is a node that stops compacting, falls behind, and then takes the read latency of the whole token range with it.
Step 3: per-node density, and why it is a time limit in disguise
The common guidance — roughly 1 TB per node on spinning disks, 2-4 TB on NVMe, and more on Cassandra 4.x+ with fast local storage — is not a hard architectural limit. It is a proxy for operational time. The real constraints are:
- Bootstrap and replacement streaming. Replacing a dead node means streaming its entire dataset from its replicas. At a realistic sustained 150-300 MB/s (throttled by
stream_throughput_outbound, network, and compaction on the receiving node), 2 TB takes hours; 8 TB takes most of a day, during which you are running at reduced redundancy. - Repair duration. Repair cost scales with data per node. If a full repair cycle cannot finish inside
gc_grace_seconds, you are risking deleted-data resurrection — see our post on repair and compaction hygiene. - Compaction throughput. Compaction must keep up with the write rate on top of a larger existing dataset. Pending compactions that only ever grow are the first symptom of nodes that are too dense for their CPU.
So the right question is not "how many TB per node is allowed" but "how long am I willing to run a node short, and what streaming rate have I actually measured?" Pick a tolerable replacement window — four hours is a common answer — measure your real streaming throughput, and let that set maximum node density. Cassandra 4.0's zero-copy streaming moved this number meaningfully for whole-SSTable transfers; measure on your hardware rather than inheriting a number from a 2017 blog post.
Step 4: write path, not just storage
Storage sizing can produce a cluster that is large enough and still too slow. Check the write side separately:
- Coordinator fan-out. Every client write at RF=3 becomes three replica writes. A 100k writes/sec application is a 300k replica-writes/sec cluster.
- Cores, not GHz. Cassandra scales with cores — mutation stages, compaction, and streaming all want their own threads. 8-16 cores per node is a reasonable floor for production; under-provisioning CPU shows up as pending compactions and GC pressure long before it shows up as CPU saturation.
- Heap. Keep heaps in the 8-31 GB range (stay under the compressed-oops boundary), and leave the rest of RAM to the page cache. Cassandra 5's trie memtables and trie-indexed SSTables reduce heap pressure, which helps, but does not change the shape of the rule.
- Disk latency. Local NVMe is the default answer. Network-attached volumes work, but you are paying for provisioned IOPS and you must size them for compaction's sequential throughput, not just the application's random reads.
Step 5: node count floor
After all that, node count is the max of three numbers:
replicated_bytes / (usable_bytes_per_node)where usable is disk size x 0.5-0.6.- Enough nodes to serve peak throughput with one node down — size for N-1, not N.
- A minimum of RF nodes per datacenter, and realistically 5-6 per DC so that losing one node is a 20% capacity event rather than a 33% one.
Three-node datacenters are legal and we deploy them for small workloads, but be clear-eyed: at RF=3 with LOCAL_QUORUM, one node down leaves no slack for a second failure or a rolling restart that goes wrong.
Worked example
IoT telemetry, 90-day TTL, 1.2 TB/day logical, compressing 3:1 to 400 GB/day. One copy at steady state: 36 TB. RF=3 in a single DC: 108 TB replicated. At 55% target utilization, that is roughly 196 TB of raw disk. On 4 TB NVMe nodes, that is 49 nodes; measured streaming of 250 MB/s puts a 4 TB replacement at about 4.5 hours, which the team accepted.
Add a second DC for DR and every number doubles. That is the conversation to have before the first node is provisioned, not after.
What we would check before you buy hardware
Sizing errors are expensive in both directions — an oversized cluster burns budget quietly, an undersized one pages people. Before committing, we usually want to see: measured compression ratio on real data, the TTL and deletion pattern (TWCS with aligned TTLs changes the storage curve substantially), peak versus average write rate, and whether the access patterns actually need all the data online. If the last 7 days answer 99% of queries, 90 days of hot Cassandra storage may be the wrong design entirely.
If you are standing up a new cluster or re-sizing one that outgrew its original plan, tell us about the workload — shape of the data, write rate, retention, and what you have measured so far. We will tell you what the math says, including when the answer is that Cassandra is not the right store for it.