Five years ago, "we're putting Cassandra on Kubernetes" was a sentence that made us ask a lot of follow-up questions. It still prompts questions, but they are different ones now. The operator pattern is mature: cass-operator (the DataStax-originated operator that K8ssandra builds on) handles the lifecycle mechanics competently, and we run and review production clusters on it.
What has not changed is the split in the work. An operator automates the part of Cassandra operations that is a state machine — join, drain, restart, scale, upgrade. It does not automate the part that is judgment: storage, scheduling, topology, and the hygiene we have written about elsewhere. Teams that expect the operator to close the whole gap end up with a cluster that is easier to create and no easier to run.
This is what the operator genuinely gives you, what it hands back to you, and how we decide whether a workload belongs on Kubernetes at all.
What the operator actually does
You declare a CassandraDatacenter — server version, node count, rack list, resource requests, storage class, config overrides — and the operator reconciles reality toward it. The valuable parts, in the order they matter operationally:
Ordered lifecycle operations. Adding a node means one StatefulSet replica at a time, waiting for the bootstrap to finish before the next starts. Removing one means nodetool decommission first, then pod and PVC teardown. Those orderings are where hand-rolled automation on VMs usually breaks, and the operator gets them right by default.
Rolling restarts and config changes. A config edit triggers a controlled rolling restart with readiness gating, one rack at a time. On VMs this is an Ansible playbook someone wrote in 2021 and is now afraid of.
Rack awareness wired into scheduling. Racks in the CRD map to node labels — normally the cloud availability-zone label — and the operator sets cassandra-rackdc.properties to match. You get topology-correct replica placement without hand-maintaining a properties file per node.
Version upgrades as a field change. Bump serverVersion and the operator performs the rolling replacement. The caveat is large and we will come back to it.
Supporting services, if you run K8ssandra. Reaper for repair orchestration, Medusa for snapshot and restore to object storage, Stargate or the data API if you want them, and a metrics endpoint wired for Prometheus. This is the part with real leverage: repairs and backups are the two things most self-managed clusters are missing, and here they arrive as part of the install rather than as two projects nobody has budget for.
What it hands back to you
Storage is still your decision, and it is the big one
Cassandra's write path assumes fast local durability. The storage class you choose determines whether that assumption holds.
Local NVMe (via a local-persistent-volume provisioner) gives you the latency profile Cassandra was designed for. The trade-off is that the volume is pinned to a physical node: if that node dies, the data dies with it, and recovery is a replace-and-stream operation rather than a reschedule. That is acceptable — it is exactly what replication factor 3 exists for — but only if you have measured how long streaming a full node takes. A 2 TB node at 100 MB/s is around six hours of degraded capacity. Know that number before you need it.
Network-attached volumes (EBS gp3, PD-SSD, or equivalent) let a pod reschedule with its data intact, which is genuinely convenient. You pay in latency and in IOPS ceilings that show up under compaction rather than under normal writes. If you go this route, provision IOPS for compaction peaks, not for your average write rate, and keep commitlog on a volume that is not fighting the data directory for the same budget.
Two settings deserve explicit attention regardless of class: volumeBindingMode: WaitForFirstConsumer (without it, volumes get provisioned in zones the scheduler then cannot use) and allowVolumeExpansion: true (because the alternative to growing a volume is replacing a node).
Scheduling: one Cassandra pod per Kubernetes node
Set anti-affinity so that two pods from the same datacenter never land on the same physical node. cass-operator does this by default; the mistake we see is teams relaxing it to fit a cluster into fewer machines. Two replicas of the same token range on one kernel is a replication factor of 3 that behaves like 2 when the machine reboots.
Run the pods with requests equal to limits for CPU and memory, which places them in the Guaranteed QoS class. Cassandra is not a workload you want the kubelet to consider for eviction, and burstable CPU produces GC pauses that look like network problems. Leave the JVM heap sized the same way you would on a VM — a fraction of the container memory limit, with the rest available for page cache and off-heap structures — and remember that the container limit, not the host's RAM, is what the JVM now sees.
Networking is not free
Overlay networks add latency to every inter-node hop, and Cassandra's write path makes several per request at LOCAL_QUORUM. It is usually a modest cost, but measure it rather than assuming it. If you are running multi-DC across Kubernetes clusters, the storage-port path between them is a real design problem: node ports, a service mesh, or pod-routable networking each have different failure modes, and this is where we see the most unpleasant surprises. Multi-cluster Cassandra on Kubernetes is doable and we have helped teams do it; it is not the thing to attempt on your first deployment.
Hygiene does not become someone else's problem
Repair age per table still has to stay inside gc_grace_seconds. Pending compactions still predict latency. Tombstones still accumulate behind a bad delete pattern. Reaper schedules repairs; it does not verify that they complete, and a Reaper install with a failing schedule is exactly as broken as a cron job with a failing schedule. Alert on the same metrics you would alert on anywhere else.
Upgrades: the one field change to be careful with
Changing serverVersion starts a rolling upgrade immediately, and the operator's readiness checks confirm that pods come back — not that the cluster is ready for the next step. The pre-upgrade discipline is unchanged from the VM runbook: check the version-specific upgrade notes, verify no repairs are running, snapshot first, and understand that you are in a mixed-version cluster until the last pod rolls, during which streaming operations are unsupported. nodetool upgradesstables still has to happen afterwards, on your schedule, and the operator will not do it for you.
Our practice is to treat the operator upgrade and the Cassandra upgrade as two separate changes with a soak period between them. Upgrading cass-operator itself can change reconciliation behavior on resources you have already deployed; doing that in the same window as a server version bump means debugging two systems at once.
When we advise staying on VMs
Honest answer: often, and for one reason above all others.
If your team does not already run Kubernetes well, do not learn it underneath your most stateful workload. Cassandra will survive node failures; it will not save you from a cluster-autoscaler policy that drains three machines in a zone because a node pool was mislabeled. If you do not have working PodDisruptionBudgets, an understood autoscaler configuration, and someone on call who reads Kubernetes events fluently, the database is the wrong place to start.
A few narrower cases:
- One or two static clusters that rarely change. The operator's leverage is in repeated lifecycle operations. If you provision a cluster every eighteen months, there is not much to automate.
- Hard latency budgets with no headroom. Overlay networking and container CPU accounting both cost a little. If your p99 budget has no room for a little, prove it on a benchmark before committing.
- DSE-specific deployments where your licensed features or tooling assume a VM footprint. Check what your support agreement covers before the migration, not after.
- When the real problem is modeling. A cluster that is struggling because of oversized partitions or relational habits in a wide-column store will struggle identically on Kubernetes. Platform moves do not fix schemas.
Conversely, the case for Kubernetes is strong when you create and tear down clusters regularly, when you want dev and staging environments that resemble production without hand-built machines, when your organization's operational muscle memory is already Kubernetes-shaped, or when K8ssandra's bundled repair and backup tooling closes a gap you have not otherwise been able to staff.
A short pre-flight list
Before a production CassandraDatacenter goes live, we want to see all of these confirmed:
- Storage class chosen deliberately, with
WaitForFirstConsumerand volume expansion enabled. - Racks mapped to real availability zones, and pod anti-affinity enforced, not relaxed.
- Guaranteed QoS: requests equal limits, heap sized against the container limit.
- A PodDisruptionBudget the autoscaler and node-upgrade process actually respect.
- Repairs scheduled and completion monitored, with an alert well inside
gc_grace_seconds. - Backups landing in object storage, and one restore rehearsed end to end on this platform.
- A measured node-replacement time, written down.
- Client drivers with
local-datacenterset explicitly and consistency atLOCAL_QUORUM.
None of that is exotic. It is the same operational contract Cassandra has always asked for, expressed in a different substrate.
If you are weighing a move onto Kubernetes, inheriting a cluster somebody else deployed there, or trying to work out whether your latency problem is the platform or the data model, get in touch. We review both, hourly, and we will tell you plainly if the platform is not what is hurting you.