RapidMindLabs | Changing Schema on a Live Cluster: Safe DDL, Schema Disagreement, and What TCM Fixes

Teams rehearse upgrades and node replacements. Almost nobody rehearses an ALTER TABLE. Yet schema changes are one of the few routine operations where a careless command, or two commands issued a second apart, can leave a running cluster with nodes that disagree about what the data looks like. The failure is quiet at first and expensive later.

This is the procedure we use for DDL on production clusters, the specific changes that are safe versus the ones that need a migration plan, and what changes once Transactional Cluster Metadata (TCM) lands in your version.

How schema propagates (pre-TCM)

Through Cassandra 4.x and 5.0, schema is distributed state agreed on by gossip, not by a consensus protocol. When you issue DDL, the coordinator writes the new definition into its local system_schema keyspace, computes a schema version (a UUID digest of all schema tables), and gossips it. Other nodes see a version they do not have, pull the mutations, apply them, and converge on the same UUID.

Convergence is normally sub-second. The important word is normally. You can check it directly:

nodetool describecluster

A healthy cluster reports exactly one schema version with every node listed under it. Two or more entries is schema disagreement, and until it resolves, different coordinators are serving different definitions of your tables.

Three things reliably cause it:

Concurrent DDL. Two ALTER/CREATE statements issued from different coordinators at nearly the same moment can produce conflicting digests. The classic version is an automated deploy applying migrations from several app pods at once.
DDL during a partition or heavy GC. A node that misses the gossip round, or is paused long enough to be marked down, applies the change late — or not at all.
DDL during a rolling upgrade. Mixed-version clusters may serialize schema differently between major versions. This is why every upgrade runbook says: no schema changes while versions are mixed.

Stuck disagreement is usually resolved by restarting the dissenting node, which forces a fresh schema pull. That is a blunt instrument on a production cluster at peak, which is why the real control is procedural: one DDL at a time, from one coordinator, with verification in between.

The DDL safety list

Not all schema changes carry the same risk. Roughly, in ascending order:

Safe, and genuinely cheap:

CREATE TABLE, CREATE KEYSPACE — new objects, no existing data to reinterpret.
ADD a column of a new name. Cassandra stores columns sparsely; adding one is a metadata change, not a rewrite. Existing rows simply have no cell for it and read back as null.
Most table-property changes: compaction, compression, gc_grace_seconds, default_time_to_live for future writes. These take effect for new SSTables and new writes; they do not retroactively rewrite data.

Safe, but with an operational tail:

Changing compaction strategy. The metadata change is instant; the recompaction that follows can saturate I/O for hours. Do it on one node first (via JMX, so the change is local), watch pending compactions, then apply cluster-wide.
DROP a column. The column disappears from queries immediately, but the data remains on disk until compaction rewrites those SSTables, and Cassandra records a drop timestamp to shadow late-arriving writes.
Replication factor changes — a metadata change that is incorrect until you run nodetool repair (or rebuild for a new DC) to actually place the extra replicas. Changing RF without the follow-up is one of the more common ways teams quietly lose consistency guarantees.

Not safe without a plan:

Re-adding a dropped column with a different type. Old SSTables still hold bytes written under the previous type. Cassandra blocks the obvious cases, but this is a reliable route to corrupt reads if you force it.
Dropping and recreating a table with the same name. The new table has a new ID and new directory, but the old data directory stays on disk, and any in-flight write still referencing the old schema can land in a confusing place. If you must recycle a name, drop, verify on every node, wait past the longest client timeout, then create.
Changing the primary key. You cannot. The partition key and clustering columns define the on-disk layout. Changing them means a new table and a data migration — dual-write to the new table, backfill historical data, shadow-read to compare, then cut over. That is the same pattern as a platform migration, and it deserves the same ceremony.
Adding a secondary index or SAI index to a large table. The index builds against existing data, which is real work on every node. Add it off-peak, and know how you will drop it if the build hurts.

The procedure we use

For any production DDL:

One change, one coordinator, one at a time. Apply migrations from a single place — not from every app instance on startup. If your framework runs migrations at boot, gate it behind a leader election or move it to a deploy job.
Check the cluster is whole first. nodetool status — every node UN. DDL with a node down means that node applies the change when it returns, if it returns cleanly.
Apply, then verify convergence before the next statement. nodetool describecluster and confirm a single schema version. Scripted: poll until there is one entry, fail the deploy if it does not converge within a timeout you choose.
Never during an upgrade, a repair-heavy window, or an incident.
Make it reversible on the client side. Deploy code that tolerates both the old and new schema before the DDL, and remove the old path afterwards. Expand-then-contract is as right in Cassandra as it is in Postgres — more so, because you cannot roll back a primary key.

For adds, that sequence is: deploy code that reads the new column defensively → ALTER TABLE ... ADD → verify convergence → deploy code that writes it → backfill if the column needs history. For drops, reverse it: stop writing, stop reading, wait past client cache TTLs, then DROP.

What TCM changes

The 5.x line replaces gossip-based agreement on cluster metadata with Transactional Cluster Metadata (TCM, CEP-21). Schema, token ownership, and membership move into a linearizable, log-based metadata service: changes are proposed, ordered in a replicated log, and applied by every node in the same order with the same epoch.

The practical consequences are worth understanding before you plan an upgrade:

Schema disagreement stops being a failure mode. Concurrent DDL is serialized rather than raced. The "restart the dissenting node" remedy goes away with the problem.
Membership operations get safer. Bootstraps, decommissions, and replacements are ordered against the same log, which removes the race windows that make concurrent range movements dangerous today. The long-standing rule of one node joining at a time is relaxed on versions with TCM, under the conditions the release notes specify — verify against your exact version rather than assuming.
There is a new component to operate. The metadata log has its own health, its own catch-up behaviour for nodes that were down, and its own upgrade step during the migration to TCM. It is a better design, not a free one.

If you are on 4.x planning a move, TCM is a real argument for the upgrade — the operational classes of risk it removes are exactly the ones that cost teams their weekends. Until then, procedure is the control: one statement, one coordinator, convergence verified every time.

If you are planning a schema change that cannot be expressed as an additive ALTER — a primary key change, a table split, a re-typed column — that is a data migration, not a DDL statement, and we would rather help you plan it before the dual-write starts than debug it after. Our cluster and data-model review covers exactly this ground; tell us about the change you are planning.