RapidMindLabs | Replacing a Dead Node and Scaling Out: Streaming, Bootstrap, and the Steps People Skip

Adding a datacenter is a project with a change window. Replacing a single dead node is not — it happens at 03:00, usually to whoever is on call, usually on a cluster they did not build. That asymmetry is why node replacement is the operation we most often find done wrong: not catastrophically, but in ways that leave a cluster with under-replicated ranges, a node owning tokens it never streamed, or resurrected data a week later.

This is the sequence we use, and the reasoning behind each step. It assumes NetworkTopologyStrategy, RF 3 per DC, and vnodes.

First: decide whether the node is dead

The cheapest replacement is the one you do not do. Before anything else, answer two questions.

Is the data intact? If the host is fine and the process is down, or the host rebooted and came back with its data directories, restarting is dramatically cheaper than streaming hundreds of gigabytes. A node that has been down for less than max_hint_window (default 3 hours) catches up from hints. A node down longer than that is consistent-ish but stale: start it, then repair it. It is still cheaper than a replace.

How long has it been down relative to gc_grace_seconds? This is the step people skip. If a node has been down longer than gc_grace_seconds (default 10 days) and you start it with its old data, it can re-propagate rows that were deleted while it was away — the tombstones that shadowed them have already been purged on the live replicas. Do not start a long-dead node with its data. Wipe it and replace it, or do not bring it back at all.

Write that threshold down somewhere the on-call engineer will see it. "Down longer than 10 days? Wipe, don't restart" is a one-line rule that prevents a class of incident that takes days to diagnose.

Replacing a dead node

The supported path is to bring up a fresh node that claims the dead node's tokens:

Confirm the node is really down and will stay down. nodetool status from two other nodes should show DN for that address. If it flaps back up mid-replace you get two nodes claiming the same tokens.
Build the replacement with empty data directories, the same Cassandra version, same cluster_name, same seeds, same snitch, and the correct rack in the topology. Rack matters: put the replacement in a different rack from the dead node and you change which replicas hold which ranges, which is a much bigger operation than you intended.
Set the replace flag in jvm.options (or cassandra-env.sh):
```
-Dcassandra.replace_address_first_boot=<dead_node_ip>
```
Use replace_address_first_boot, not the older replace_address. The _first_boot variant is ignored on subsequent restarts, so a routine restart weeks later does not attempt a second replacement. The number of clusters where the old flag is still sitting in a config file is not small.
Do not add the replacement to the seed list, and do not give it the dead node's IP if that IP is in the seed list on other nodes. Seeds do not bootstrap. A node that skips bootstrap joins the ring owning ranges whose data it never received — it will answer reads with nothing. If the dead node was a seed, first update the seed list on the surviving nodes to point at a different live node, then replace.
Start it and watch streaming, not the log tail: nodetool netstats on the joining node, and nodetool status elsewhere, where it should appear as UJ (joining) until it flips to UN.
Repair when it finishes. Streaming gives the replacement the data its source replicas had; it does not reconcile divergence among them. Run a repair on the new node's ranges before you consider the operation done.
Remove the replace flag from config. It is harmless with _first_boot, but the next engineer should not have to reason about why it is there.

If the replacement fails partway through — and it does, on large nodes, usually to a network blip or a full disk — stop the node, wipe its data, commitlog, hints, and saved caches directories, and start over. Resuming a half-streamed bootstrap is possible in some versions, but a clean restart is what we recommend on a cluster you do not know intimately.

Making streaming survive

A replacement streams roughly the node's full data footprint. On a 1 TB node at an effective 50 MB/s, that is about six hours if nothing interferes. Two knobs and one habit decide whether it finishes.

stream_throughput_outbound_megabits_per_sec caps outbound streaming per node, and it is a megabits setting — misreading it as megabytes is the single most common tuning error we see on this path. The default is conservative because streaming competes with the read path; raising it is reasonable when the source nodes have I/O and network headroom, and counter-productive when they do not. Change it live with nodetool setstreamthroughput and watch p99 read latency on the sources while you do.
streaming_keep_alive_period_in_secs exists because long streams die silently behind load balancers and stateful firewalls that time out idle connections. If your streams consistently stall at the same elapsed time, this — not throughput — is your problem.
Check disk headroom on the joining node before you start. It needs room for the streamed SSTables plus the compaction that follows them. Starting a bootstrap at 70% disk is how a six-hour operation becomes a two-day one.

Wide partitions hurt here too. Streaming works at the partition level; a multi-gigabyte partition is an indivisible unit of work that will not parallelize and will not resume gracefully. Bootstrap times are one more argument for the partition-size budget.

Scaling out is a different operation

Adding capacity is not replacement, and the two get conflated. A new node bootstraps, takes ownership of a slice of every existing node's token ranges, and streams that data in. Three rules:

Add one node at a time, and wait for UN. Concurrent bootstraps in the same DC can produce overlapping range ownership. Some versions allow it with care; we do not recommend it on production clusters that matter. Patience costs hours. The alternative costs a repair of the entire cluster.

Run nodetool cleanup on the other nodes afterwards — one at a time, off-peak. Until you do, the old owners still hold the data for ranges they no longer own. That data is not served, but it is on disk, it is compacted, and it is repaired. Teams add three nodes, see no disk relief, and conclude scaling out "didn't help." Cleanup is the missing step. It is an I/O-heavy rewrite, so treat it like compaction work, not like a quick command.

Expect the cluster to feel worse before it feels better. Bootstrap plus cleanup is sustained extra I/O across every node in the DC. If you are scaling out because the cluster is already saturated, you are adding load to a saturated system — plan for the latency bump, or do it during your lowest-traffic window, or both. The time to add nodes is before you need them; capacity planning that triggers at 50-60% steady-state disk gives you room to do this calmly.

Decommission, don't remove. Taking a node out of a live cluster is nodetool decommission on that node, which streams its data to the new owners before it leaves. nodetool removenode is for nodes that are already dead and cannot stream; it makes the surviving replicas rebuild the missing ranges from each other, which is more expensive and requires that they actually have consistent copies. Use assassinate only when you understand exactly why the other two failed.

Verify, then close the ticket

An operation is not finished because a command returned zero. Before you go back to bed:

nodetool status — every node UN, ownership percentages roughly even across the DC, no leftover entries for the dead node. Gossip state that still references a removed node is worth chasing down now rather than during the next replacement.
nodetool describecluster — one schema version. Schema disagreement after a topology change is a real and confusing failure mode.
Repair the affected ranges and confirm it completed. Not started: completed.
Cleanup done on the nodes that gave up ranges, and disk usage reflecting it.
Client metrics. Drivers rediscover topology through events, but a cluster that just changed shape is a good moment to check that no application is still opening connections to a host that no longer exists, and that speculative retry settings are not now pointed at a node that is busy compacting.

A note on where this is heading

Most of the sharp edges above trace back to the same root: cluster metadata — token ownership and schema — has historically been agreed via gossip, which is eventually consistent. That is why concurrent bootstraps are risky, why schema disagreement happens, and why so much of the runbook is "do one thing at a time and verify."

The Cassandra project's answer is Transactional Cluster Metadata (CEP-21), which puts ownership and schema changes behind a linearizable log rather than gossip convergence. It is a genuine improvement to exactly these operations, and it is not in the 5.0 line — it lands in the release after it. Follow it, but do not plan a topology change around it until it is running in a version you are willing to put in production. Until then, the runbook above is the runbook.

If you are staring at a cluster that needs nodes replaced, rebalanced, or grown — or one where an earlier topology change left ownership looking wrong — get in touch. A cluster review will tell you what the ring actually looks like today, and an operations retainer means the next 03:00 replacement is a routine one.