RapidMindLabs | The Client Side: Driver Timeouts, Retries, and Speculative Execution

A team pages us because p99 read latency tripled. We look at the cluster: compaction is current, repairs finished yesterday, GC pauses are unremarkable, server-side coordinator latency is flat. The cluster is fine. The application is not.

This happens often enough that driver configuration is a standing item in every cluster and data-model review we run. Cassandra's server-side behavior is well documented and widely discussed; the client-side defaults are neither, and they are where a surprising share of production latency actually lives.

What follows applies to the modern unified DataStax Java driver (4.x) and its Python, Go, Node, and C# siblings. Names differ; the decisions do not.

Start from the one number the server owns

Before touching the client, write down the server-side read and write timeouts from cassandra.yaml — typically read_request_timeout at 5 seconds and write_request_timeout at 2 seconds. These are the coordinator's own patience.

Your client request timeout should be at least as long as the server's, plus network. If the client gives up at 2 seconds on a read the coordinator is still willing to spend 5 seconds on, you have created a machine that abandons work the cluster is still doing, then — if retries are naive — asks for it again. That is how a slow table becomes an overload.

The opposite error is worse in aggregate: a 30-second client timeout means every stalled request occupies a connection slot and an application thread for half a minute. Under a partial failure you run out of threads before you run out of database.

Pick deliberately. A common shape: client timeout slightly above the server timeout for the request type, and a separate, much tighter budget at the service boundary above it so a caller is never waiting on a query the user has already given up on.

Idempotence is the switch that enables everything else

Driver 4.x will not retry or speculate on a statement unless it is marked idempotent. This is a safety default, and it is also the single setting most teams never set — which means their retry and speculative-execution configuration is quietly doing nothing.

A statement is idempotent when executing it twice produces the same result as executing it once. In Cassandra terms:

Idempotent: ordinary INSERT/UPDATE of fixed values, DELETE, SELECT.
Not idempotent: counter updates, lightweight transactions (IF NOT EXISTS, IF conditions), and any statement containing now(), uuid(), or a list append/prepend. Non-frozen list operations are read-modify-write internally; applying them twice appends twice.

Audit your statements, mark the safe ones — per statement, or at the profile level for a codebase you have actually checked — and generate uuid()/now() values in your application instead of the server. That last change also makes dual-write migrations correct, which is a good sign it is the right habit.

Retry policy: retry the transient, never the overload

The default retry policy is deliberately conservative. It retries a read timeout once if enough replicas responded but data was missing, retries a write timeout only for batch-log writes, and tries the next host on Unavailable or an aborted request. It does not retry on a general timeout, because a timeout usually means the cluster is busy, and retrying busy clusters is how you convert a latency blip into an outage.

If you write a custom policy, two rules:

Never blindly retry on WriteTimeoutException for SIMPLE or BATCH writes. You do not know whether the mutation applied. Retry only where the write is idempotent, and understand that you are choosing duplicate-work risk over lost-write risk on purpose.
Never retry on OverloadedException without a backoff. Overloaded and Unavailable are the cluster telling you to slow down. An immediate retry is the client arguing with it.

Bound total attempts, and instrument retry counts as a metric. A retry rate that used to be 0.01% and is now 3% is an incident forming, and it is usually visible days before latency moves.

Speculative execution: the right fix for the long tail

Cassandra stores data on several replicas, so a slow node need not mean a slow request. Speculative execution sends the same read to another replica when the first one has not answered within a threshold, and takes whichever comes back first.

This is the correct treatment for tail latency caused by one node doing garbage collection, one node compacting hard, or one node on a degraded disk. It is the wrong treatment for a cluster that is uniformly overloaded — there you are simply multiplying the load.

A constant policy with a delay set near your current p95 and a small maximum number of executions (two or three) is a reasonable starting point. Set the delay from measurement, not intuition, and re-derive it after any significant workload change. And watch the amplification: if speculative executions exceed a few percent of requests, the threshold is too tight or the cluster problem is not tail-shaped.

Speculative execution requires idempotence. See above.

Load balancing: stay local, stay token-aware

Two settings account for most avoidable coordinator hops:

Local datacenter. Set it explicitly. In driver 4.x it is required for the default policy, and in multi-DC topologies an application that treats a remote DC as usable will produce cross-region latency that looks, from the server side, like nothing at all is wrong.
Token awareness. On by default, and worth protecting. It routes each request to a replica that actually owns the partition, saving a hop. It only works if the driver can compute the routing key — which means prepared statements with bound values. String-concatenated CQL defeats it, along with the server-side prepared-statement cache.

Pair this with LOCAL_QUORUM rather than QUORUM for multi-DC clusters unless you have a specific, written reason to pay cross-DC latency on every request.

Paging and the queries that fetch too much

The default fetch size is 5,000 rows. Every driver pages transparently, so an unbounded SELECT looks fine in code and arrives as a slow scan in production. Two habits:

Set a fetch size that matches your row width. Wide rows at 5,000 per page can produce multi-megabyte responses that stress coordinator heap.
For anything iterating a large result set, use explicit paging state rather than materializing pages in a loop inside a request handler.

And if a query needs ALLOW FILTERING to run at all, no driver setting will save it. That is a data-model finding, and it belongs in the model, not the client.

Connection pooling and heartbeats

Driver 4.x defaults to a single connection per node with a high concurrent-request limit, which is right for most workloads — the protocol multiplexes. Raise the pool only when you can show in-flight request queuing at the connection level, not because more sounds faster.

Do confirm the heartbeat interval is shorter than the idle timeout of every firewall and load balancer between your application and the cluster. Silently dropped idle connections produce a distinctive symptom: a burst of errors after every quiet period, and nothing wrong on the server.

A short checklist

For each application that talks to your cluster, answer these in writing:

What is the client request timeout, and how does it compare to the server timeout for that request type?
Which statements are marked idempotent, and has someone verified the list?
What does the retry policy do on write timeout and on overload?
Is speculative execution on, at what threshold, derived from which measurement?
Is the local datacenter set explicitly, and are all hot-path statements prepared?
What is the fetch size, and which queries return unbounded result sets?
Is the heartbeat interval shorter than every idle timeout on the network path?

Seven questions. In our experience most teams can answer two of them, and the gap between two and seven is measured in p99 milliseconds.

The honest summary

None of this substitutes for a sound data model or a healthy cluster. A bad partition key will out-damage every setting on this page. But when the cluster metrics are clean and the application still feels slow, the driver is the next place to look — and it is the cheapest thing on the list to fix, because it ships with your application rather than with a maintenance window.

If your p99 does not match what the coordinator thinks it is doing, that difference is usually explainable in an hour or two of reading configuration next to metrics. Tell us the driver, the version, and the query shapes that hurt.