RapidMindLabs | BATCH, Counters, and the Writes That Punish the Coordinator

A team calls us because p99 writes went from 4 ms to 300 ms and the coordinators are hot while the disks are bored. Nine times out of ten the application is sending batches — and the batch is doing the opposite of what the author expected.

The word is the problem. In a relational database, a batch is a performance optimization: one round trip instead of many. In Cassandra, BEGIN BATCH is an atomicity primitive, and atomicity is not free. Used as a throughput trick, it moves work from clients that scale horizontally onto coordinators that do not.

This post covers what each batch type costs, which thresholds to watch, and how to rewrite the patterns that hurt.

The three batches

CQL has three distinct things wearing the same keyword.

Logged batch (BEGIN BATCH). The coordinator writes the whole batch to the system.batches table on two other nodes in the local datacenter before applying any mutation. If the coordinator dies mid-batch, a surviving replica replays it. That gives you atomicity — all mutations eventually apply — but not isolation: other readers can see some rows updated and others not, and there is no rollback. You pay for that guarantee with extra writes, extra fsyncs, and a replay path that has to be cleaned up afterward.

Unlogged batch (BEGIN UNLOGGED BATCH). No batchlog. The coordinator simply fans the mutations out. If it dies halfway, half the mutations applied and nobody repairs that. Unlogged is only sane when every statement in the batch targets the same partition — in that case the mutations are a single write to a single replica set, genuinely cheaper than N round trips, and atomic and isolated as a side effect of being one partition.

Counter batch (BEGIN COUNTER BATCH). Counters are a separate write path with their own rules; see below.

Why the multi-partition batch hurts

Consider 100 inserts spread across 100 partitions, sent as one logged batch.

The client makes one round trip. Good for the client.
The coordinator now owns 100 mutations destined for up to 100 different replica sets, most of which it does not own. It writes the batchlog to 2 nodes, then sends ~300 inter-node messages (RF=3), waits for them under the consistency level, then deletes the batchlog.
Token-aware routing is defeated. Sent as 100 individual statements, a token-aware driver would have delivered each one directly to a replica, spreading coordination across the whole cluster. As a batch, one node coordinates everything.

The cluster did strictly more work, concentrated on one node, and the client's latency is now the latency of the slowest of 100 mutations. That is why batching "for speed" reliably makes throughput worse at scale: you have built a hotspot generator that moves with whichever coordinator the driver picked.

The failure mode is distinctive. Coordinator CPU and MutationStage/Native-Transport-Requests queues climb on a subset of nodes while disk utilization stays flat. tpstats shows dropped mutations. Write latency measured at the client is far worse than write latency measured per-node — a gap that almost always means coordination overhead, not storage.

Thresholds worth alerting on

Cassandra ships warnings for this and most teams never read them. In cassandra.yaml:

batch_size_warn_threshold — default 5 KiB. Logs Batch for [ks.table] is of size X, exceeding specified threshold.
batch_size_fail_threshold — default 50 KiB. Rejects the statement outright.
unlogged_batch_across_partitions_warn_threshold — default 10. Logs when an unlogged batch spans more than that many partitions.

(Older 3.x/4.0 builds spell these batch_size_warn_threshold_in_kb and friends; the semantics are the same.)

Three practical moves:

Grep the system log for BatchStatement warnings on day one of any audit. It is the cheapest diagnostic in Cassandra and it names the offending keyspace and table.
Alert on the warning rate, not just the fail threshold. By the time batches exceed 50 KiB someone has been degrading the cluster for months.
Do not raise the thresholds to silence the log. They are not conservative; they are a smoke alarm. Raising them is the Cassandra equivalent of taking the battery out.

The rewrites

Pattern: batch across partitions for throughput. Replace with concurrent single statements. Prepare the statement once, fire N async executes with a bounded concurrency limit (a semaphore around 128–512 in-flight per client is a reasonable starting point, tuned against max_requests_per_connection), and let the token-aware policy route each one to a replica. You get parallelism and spread coordination. In our experience this is the single highest-yield change in a write-heavy application: same code shape, often a 3–10x improvement in sustained write throughput with lower p99.

Pattern: batch across partitions because the writes must be atomic. This one is real, and a logged batch is the right tool — but keep it small (single-digit statements, well under the warn threshold) and accept that it is a correctness cost, not a performance feature. The usual legitimate case is denormalized duplicate tables: the same logical row written to events_by_user and events_by_device. Query-first modeling makes multiple write targets normal, and keeping them consistent is exactly what the batchlog is for.

Pattern: many rows, same partition. Keep the batch, make it UNLOGGED. Same partition means one replica set, one mutation, no batchlog. This is the one case where batching is unambiguously faster. Watch the size anyway — a 10 MB partition write will stall a memtable flush.

Pattern: bulk load. Batches are the wrong tool entirely. Use prepared statements with controlled concurrency, or build SSTables offline with CQLSSTableWriter and sstableloader for genuinely large one-shot loads. Astra DB users should check the per-request limits on their tier before assuming batch sizes from a self-managed cluster transfer over.

Counters: a different animal

Counter writes are not normal writes, and teams discover this during an incident.

They are read-before-write. A counter update must read the current shard value on the replica before applying the delta. That makes counter writes several times more expensive than regular writes and sensitive to read latency, compaction backlog, and page cache pressure.
They are not idempotent. A regular Cassandra write can be retried safely — the same cell value lands twice, same result. A counter increment applied twice increments twice. This interacts badly with driver retry policies and with speculative execution: an increment that times out may have applied. Retrying it may double-count; not retrying may lose a count.
Repair does not fix counter drift. Over-counts and under-counts caused by retries are indistinguishable from legitimate values. There is no reconciliation pass.

The operational guidance follows from that:

Turn off speculative execution and aggressive retries on the statements that touch counters. Idempotence is the precondition for both, and counters do not have it. Use a no-op or default retry policy on those statements specifically rather than globally.
Never mix counter and non-counter columns in a table; Cassandra will not let you, and that constraint is a hint about how separate the path is.
Counter batches are only a grouping convenience. BEGIN COUNTER BATCH is not the batchlog; it does not make the increments atomic across partitions. Keep them single-partition.
If you need exact counts, do not use counters. Write immutable event rows and aggregate — in a streaming job, a materialized rollup table, or at read time over a bounded partition. Counters are the right answer for approximate, high-volume, best-effort tallies (view counts, like counts). They are the wrong answer for billing, inventory, or anything auditable. If you need a strict decrement-if-available, that is a lightweight transaction on a regular table, with the Paxos cost that implies.

A short checklist

Before the next write-path incident:

Search system.log for Batch for and Unlogged batch covering warnings. Fix what they name.
Confirm the thresholds in cassandra.yaml are at defaults, not raised.
Audit the application for multi-partition batches and classify each one: throughput misuse (rewrite as concurrent statements) or genuine atomicity (keep, keep small).
Confirm token-aware, datacenter-local load balancing is actually active in the driver — a batch-heavy app often masks a misconfigured policy.
For any counter table, confirm retry and speculative-execution settings and document what a double-count would cost the business.

None of this is exotic. It is the difference between a cluster that absorbs a traffic spike and one that drops mutations on a Tuesday afternoon.

If your write latency is coordinator-bound and you want a second pair of eyes on the schema and the driver configuration, that is the first half of a cluster and data-model review. Tell us about the cluster — version, topology, and the shape of the writes is enough to start.