IF NOT EXISTS is four extra characters of CQL and roughly four extra round trips of distributed consensus. That ratio — trivial to write, expensive to run — is why lightweight transactions (LWTs) show up in so many of the clusters we review, and why they are so often the thing holding p99 write latency at 40ms when everything else is at 3ms.
LWTs are not a mistake. Uniqueness on a username, a state machine that must not skip a step, an idempotency token — these need linearizability, and Cassandra offers it. The mistake is using them by reflex, at a rate the cluster was never sized for, on partitions where writers collide.
What actually happens on the wire
A normal LOCAL_QUORUM write is one round trip from coordinator to replicas. An LWT is a Paxos ballot, and in the classic (pre-Accord) implementation it runs four phases against the replicas of that partition:
- Prepare / promise — the coordinator proposes a ballot number; replicas promise not to accept anything older.
- Read — the current value of the row is read at
SERIALconsistency so theIFcondition can be evaluated. - Propose / accept — the new value is proposed and accepted by a quorum.
- Commit — the value is written to the normal storage path and acknowledged.
So: roughly four coordinated round trips instead of one, plus a read, plus writes to the system.paxos table at each step. The practical rule of thumb we use in capacity conversations is that an LWT costs about four times a regular write in latency and rather more than that in coordinator and replica work. On a single-DC cluster with sub-millisecond network, that is often tolerable. Across datacenters it is not.
The consistency levels people get wrong
LWTs have their own consistency dimension. serial_consistency is either SERIAL (a quorum of all replicas, every DC) or LOCAL_SERIAL (a quorum within the local DC). Two things follow, and both bite in production:
SERIALin a multi-DC cluster puts cross-DC round trips inside every LWT. Four phases times an 80ms inter-DC RTT is not a latency budget, it is an outage with good manners. If your correctness requirement is per-region, setLOCAL_SERIALexplicitly; most drivers default toSERIAL.LOCAL_SERIALdoes not survive a DC failover. Linearizability is scoped to the DC that ran the ballots. If traffic fails over and the same logical key is written in another DC, uniqueness is no longer guaranteed. Decide which you can live with before the failover, not during it.
The same care applies to reads. A plain LOCAL_QUORUM read can observe a Paxos ballot that was accepted but not yet committed; reading at SERIAL forces the in-flight round to complete first. Writing conditionally and reading non-serially is a very common way to get "impossible" results in a test suite.
Contention is the failure mode, not latency
Latency is predictable. Contention is what actually causes incidents. Paxos serializes per partition, so two clients doing an LWT against the same partition key at the same moment do not queue politely — one loses the ballot and retries with a higher one. Under load this degrades super-linearly: more retries, more ballots, more contention, and eventually WriteTimeoutException with writeType: CAS.
Signs to look for:
CasWriteTimeoutandCasReadTimeoutcounts rising faster than request rate.nodetool tablestats system.paxosshowing growth — the paxos table holds ballot state and is itself compacted and repaired.- Latency histograms with a long, ragged tail on one table and a clean p50, which is the shape of retry storms rather than slow disks.
The cluster-level knob is small: spread the contention. The design-level fix is to stop concentrating conditional writes on hot partitions. A counter-like row that every request touches with IF is the anti-pattern; a per-entity partition where each key has at most one contending writer is the pattern.
Four ways to need fewer LWTs
Make the write idempotent instead of conditional. Much of what people express as "insert only if absent" is really "insert this exact immutable row, repeatedly, safely." If the value is deterministic, a plain write is already correct and last-write-wins does no harm.
Put uniqueness in the partition key. IF NOT EXISTS on a username column is expensive; a users_by_username table whose partition key is the username reduces the problem to a single conditional insert on a partition with exactly one plausible writer. The ballot is still a ballot, but contention approaches zero.
Bound the scope. Conditional updates on one small row are fine. Conditional updates that also drag a wide partition through a serial read are not. Keep LWT-bearing rows narrow, and keep the columns the IF clause inspects on that same narrow row.
Move the ordering problem out of the database. Some workloads want a distributed lock or a saga, and Cassandra's LWTs are a poor lock manager — no lease, no fencing token, no fairness. If you are building leader election, use something built for it. We say this often enough that it is one of our standing honest answers: Cassandra is the wrong tool for coordination-heavy work.
What Accord changes
The 5.x line's most consequential work is not on the read path — it is CEP-15 (Accord) and CEP-21 (Transactional Cluster Metadata). TCM replaces gossip-propagated schema and topology with a linearizable metadata log, which is the foundation that makes everything else safe. Accord builds on it: a leaderless consensus protocol that, in the common uncontended case, commits in one round trip, and — unlike Paxos-based LWTs — can span multiple partitions and multiple keyspaces in a single transaction, exposed in CQL as BEGIN TRANSACTION ... COMMIT TRANSACTION.
What that means for a team planning today:
- It is a reason to keep your upgrade path current, not a reason to redesign now. Treat multi-partition transactions as a capability arriving on the 5.x line rather than something to build against in a production schema this quarter.
- It does not repeal the modeling rules. A one-round-trip transaction on a hot partition is still a hot partition. Query-first modeling remains the thing that determines whether your cluster is fast.
- It does change the honest answer to "can Cassandra do multi-row atomicity?" from "only within a partition, via batches" to "yes, with real trade-offs to measure." Plan to measure them.
The short version
Count your LWTs. If conditional writes are more than a small percentage of write volume, or if any single partition takes concurrent conditional writes at rate, that is a design finding, not a tuning problem. Set LOCAL_SERIAL deliberately. Read serially what you wrote conditionally. And model so that uniqueness lives in the partition key, where consensus is cheap because nobody is arguing.
If you are staring at a CAS timeout graph and trying to work out whether it is the schema or the topology, that is exactly the sort of thing a cluster and data-model review answers in days rather than quarters.