Adding a Datacenter to a Live Cassandra Cluster

Adding a datacenter is the most useful trick Cassandra gives you. It is how you move a cluster to another region, how you move from self-managed hardware to cloud instances, how you stand up an analytics or search footprint that does not compete with production reads, and how a surprising number of migrations and upgrades actually get executed. It is also the procedure we most often see done in the wrong order, with the same result each time: a window where the new datacenter is advertised as a replica, has no data, and happily answers reads with nothing.

This is the runbook we follow. The order matters more than any individual command.

Before you start: the two prerequisites

Your snitch must be topology-aware. SimpleSnitch has no concept of datacenters. If the cluster is still on it, you cannot add a DC without first migrating to GossipingPropertyFileSnitch (or a cloud snitch) and setting cassandra-rackdc.properties on every node. On a single-DC cluster you can do this as a rolling change provided the existing DC name stays identical — renaming a datacenter in place is not supported and will strand your data under a replication setting that no longer refers to anything.

Your keyspaces must already use NetworkTopologyStrategy. SimpleStrategy ignores topology and will place replicas across your new DC the moment it appears. Convert first, on the single-DC cluster, with the same replica count you have today:

ALTER KEYSPACE app WITH replication =
  {'class': 'NetworkTopologyStrategy', 'dc1': 3};

This is a metadata-only change when the numbers match — no streaming, no data movement. Then run a full repair before you go further. Everything downstream assumes replicas are consistent.

Step 1: pin the application to its current DC

Do this first, and verify it, before a single new node exists.

In the drivers, set the local datacenter explicitly (basic.load-balancing-policy.local-datacenter in the Java driver; equivalents exist in every modern driver) and use LOCAL_QUORUM / LOCAL_ONE consistency levels — never plain QUORUM, which counts replicas cluster-wide and will start requiring acknowledgements from the new DC across a WAN link as soon as it has replicas. Confirm in production traffic, not in config review: a single service still on QUORUM is enough to turn your expansion into an incident.

While you are there, make sure no client is using a contact point in the new DC.

Step 2: bring up the new nodes, empty and unreplicated

Stand up the new DC's nodes with:

• cluster_name identical to the existing cluster,
• seeds listing a couple of nodes from each DC,
• auto_bootstrap: false in cassandra.yaml,
• correct dc and rack values in cassandra-rackdc.properties,
• the same Cassandra version as the existing cluster (expand first, upgrade later — never both at once),
• connectivity on the storage port between all nodes in both DCs, and a WAN path you have actually measured for latency and throughput.

auto_bootstrap: false is the point of this step. No keyspace yet replicates to the new DC, so there is nothing legitimate to stream; the nodes join gossip, take ownership of their token ranges in a DC that holds no replicas, and sit idle. nodetool status should show them UN with essentially zero load.

Use the same number of racks as the existing DC where you can, and distribute nodes evenly across them. Uneven racks produce uneven replica placement, and the imbalance is tedious to unwind later.

Step 3: extend replication

Now — and only now — tell the keyspaces about the new DC:

ALTER KEYSPACE app WITH replication =
  {'class': 'NetworkTopologyStrategy', 'dc1': 3, 'dc2': 3};

The instant this statement commits, dc2 is a replica for app. New writes replicate there immediately. Historical data does not. This is the dangerous window, and it is why step 1 exists: any read routed to dc2 right now returns partial data at best.

Repeat for every keyspace you intend to serve from the new DC, and do not skip these:

• system_auth — miss it and logins fail in the new DC the moment you point anything at it. Replicate it to every DC (three replicas per DC is the usual choice).
• system_distributed and system_traces — repair state and tracing live here.
• Any DSE or add-on keyspaces your deployment created.

The system, system_schema, and other local keyspaces are node-local and need nothing.

Step 4: rebuild

On each new node, stream the historical data in from the source DC:

nodetool rebuild -- dc1

The -- dc1 argument names the source datacenter and is not optional in practice; without it the node chooses sources itself, which on a multi-DC cluster can mean streaming across the wrong link.

Run it on a few nodes at a time, not all of them. Rebuild saturates the inter-DC link and the source nodes' disks; parallel rebuilds across a whole DC are a reliable way to degrade the production side of a cluster you were trying not to touch. Throttle with nodetool setstreamthroughput and watch source-node read latency as the real signal. On Cassandra 4.0 and later, zero-copy streaming makes this dramatically faster for tables on compatible compaction strategies, but the WAN link is still the ceiling.

Rebuild is resumable in the sense that you can re-run it: if a node fails partway, run nodetool rebuild on it again. It streams what is missing. Budget hours to days depending on data size — we plan rebuilds in terabytes-per-hour of measured link throughput, not in optimism.

When every node reports the expected load in nodetool status and nodetool netstats shows no active streams, the new DC holds the data.

Step 5: repair, then verify

Run a full repair across the new DC before you trust it. Rebuild streams from one replica per range; repair is what reconciles everything and, importantly, establishes the repaired state that tombstone purging depends on. If your repair schedule is orchestrated, add the new DC to it now rather than after cutover — repair age per table must stay inside gc_grace_seconds in the new DC exactly as it does in the old one.

Verification before traffic:

• nodetool status — all nodes UN, load per node within a few percent of its rack peers.
• Row-count or checksum spot checks on your largest tables against the source DC.
• A canary client pointed at the new DC with LOCAL_QUORUM, reading known-good keys.
• Inter-DC latency and dropped-mutation counters under production write volume.

Step 6: cut over, deliberately

Move traffic by changing the drivers' local datacenter, service by service, starting with the most read-heavy and least write-critical. Keep consistency at LOCAL_QUORUM throughout. Watch p99 read latency, dropped mutations, and hint accumulation on both sides.

The rollback is the reason this sequence is worth following exactly: until you remove dc1 from the replication map, the old DC is still a full, live, current replica set. Rolling back is a driver configuration change, not a data recovery. Keep both DCs replicating for at least a full business cycle — a week is a reasonable floor — before you consider decommissioning the old one. When you do, alter replication to drop the old DC first, then nodetool decommission its nodes one at a time.

What goes wrong

In the field, four failures account for nearly all of the pain:

1. Bootstrapping with replication already extended. Nodes stream during join, the cluster spends hours in an unpredictable state, and the operator loses the clean rebuild checkpoint. auto_bootstrap: false first, ALTER KEYSPACE second.
2. A client on QUORUM. Latency doubles the moment the new DC becomes a replica, because acknowledgements now cross a WAN.
3. Forgetting system_auth. Everything looks fine until the first application connects to the new DC and cannot authenticate.
4. Parallel rebuilds. The expansion that was supposed to be invisible shows up as elevated read latency in production for an afternoon.

None of these are subtle once you know them. All of them are easy to hit at 2 a.m. on a change window.

When this is the wrong approach

If your goal is a version upgrade, add the DC on the same version and upgrade afterward; mixed-version streaming is unsupported and the failure modes are unpleasant. If your goal is to move to Astra DB, the DC-expansion pattern does not apply — that path is a proxy-mediated dual-write cutover, which is a different runbook. And if the cluster you are expanding is already unhealthy — repairs failing, compaction behind, partitions oversized — fix that first. A second datacenter copies your problems faithfully and then charges you WAN bandwidth for the privilege.

If you are planning a region move, a cloud migration, or a multi-DC topology and want the runbook reviewed against your actual data sizes and link budget before the change window, get in touch — this is core work for our migrations practice, billed hourly.