Add LWTRetryPolicy: retry CAS timeouts on same host with backoff by mykaul · Pull Request #783 · scylladb/python-driver

mykaul · 2026-04-01T09:13:02Z

Summary

LWT queries use Paxos consensus where the first replica (Paxos coordinator/leader) drives the consensus rounds. When a CAS write times out, retrying on a different host causes Paxos contention — the new coordinator must compete with the original, potentially causing cascading timeouts across the cluster.

Currently, no built-in retry policy retries CAS write timeouts at all — they are all RETHROWN immediately:

RetryPolicy.on_write_timeout: CAS → RETHROW
ExponentialBackoffRetryPolicy.on_write_timeout: CAS → RETHROW
DowngradingConsistencyRetryPolicy.on_write_timeout: CAS → RETHROW

This PR adds LWTRetryPolicy, a new retry policy that extends ExponentialBackoffRetryPolicy with LWT-aware behavior:

Scenario	Decision	Rationale
CAS write timeout	RETRY same host + backoff	Stay on Paxos coordinator to avoid contention
Serial read timeout	RETRY same host + backoff	CAS read at serial CL, same coordinator logic
Serial unavailable	RETRY next host + backoff	Paxos quorum lost on this node, try another
Non-CAS operations	Delegate to parent	Standard `ExponentialBackoffRetryPolicy` behavior

This is modeled after gocql's LWTRetryPolicy interface, which retries LWT queries on the same host to avoid Paxos contention. The key comment from gocql (line 188):

"Retrying on a different host is fine for normal (non-LWT) queries, but in case of LWTs it will cause Paxos contention and possibly even timeouts if other clients send statements touching the same partition to the same time."

Usage

fromcassandra.clusterimportClusterfromcassandra.policiesimportLWTRetryPolicy# Use as the default retry policycluster=Cluster(default_retry_policy=LWTRetryPolicy(max_num_retries=3))
# Or assign to a specific statementstatement.retry_policy=LWTRetryPolicy(max_num_retries=5)

Changes

cassandra/policies.py: Added LWTRetryPolicy class (extends ExponentialBackoffRetryPolicy)
tests/unit/test_policies.py: Added LWTRetryPolicyTest with 21 tests

Tests

21 new tests covering:

CAS write timeout retries on same host with backoff
Backoff delay increases with retry attempts
Max retries exceeded → RETHROW
Consistency level preserved across retries
Non-CAS writes delegate to parent (SIMPLE→RETHROW, BATCH_LOG→RETRY, COUNTER→RETHROW)
Serial read timeout retries on same host (SERIAL and LOCAL_SERIAL)
Serial unavailable retries on next host
Non-serial operations delegate to parent policy
Request errors inherit parent behavior
Constructor defaults and customization
All methods return proper 3-tuples

All 103 tests in tests/unit/test_policies.py pass.

@calebxyz - it's pointless to compare the different drivers' performance - they differ greatly. What is important is the correct and optimized behavior - and there we still have gaps. I think we are very far from testing the correct behavior - we need many more system level tests on one hand (and on the other hand, I'm against testing it in full setup - which is why I've created scylladb/scylla-ccm#731 (that is probably not ready yet , but that's a different issue)

calebxyz · 2026-04-07T07:50:56Z

CC @calebxyz It needs more review (for me first of all), but looks important to push for at some point.
If this behavior is something that we have on go drivers it should be good, do we know the performance for LWT on go vs java for example? Or vs python. Cc @temichus
@calebxyz - it's pointless to compare the different drivers' performance - they differ greatly. What is important is the correct and optimized behavior - and there we still have gaps.

This is sad, the amount of unpredictability is horrible

mykaul · 2026-04-07T07:53:29Z

CC @calebxyz It needs more review (for me first of all), but looks important to push for at some point.
If this behavior is something that we have on go drivers it should be good, do we know the performance for LWT on go vs java for example? Or vs python. Cc @temichus
@calebxyz - it's pointless to compare the different drivers' performance - they differ greatly. What is important is the correct and optimized behavior - and there we still have gaps.
This is sad, the amount of unpredictability is horrible

That's one of the major reasons to move some to be Rust based - Rust, CPP-over-Rust, NodeJS-over-Rust, Python-over-Rust. (and we'll stay with Java and Go, I reckon).
SAME situation with our Alternator clients!

LWT queries use Paxos consensus where the coordinator is the Paxos leader. Retrying on a different host causes Paxos contention — the new coordinator must compete with the original one, potentially causing cascading timeouts. LWTRetryPolicy (extends ExponentialBackoffRetryPolicy) handles this by: - CAS write timeouts: retry on SAME host with exponential backoff - Serial consistency read timeouts: retry on SAME host with backoff - Serial consistency unavailable: retry on NEXT host (paxos quorum lost) - Non-CAS operations: delegate to base ExponentialBackoffRetryPolicy Modeled after gocql's LWTRetryPolicy interface.

coderabbitai · 2026-07-29T20:26:20Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 97b7e56a-edc9-40f8-9ae5-513613f36154

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

_{Comment @coderabbitai help to get the list of available commands.}

mykaul · 2026-07-29T20:26:33Z

Rebased onto current master (was based on an older commit; conflicts were expected given how much LWT/retry-policy code has landed since, but the rebase applied cleanly — LWTRetryPolicy is purely additive at the end of policies.py and doesn't touch any of the lines master has changed).

Consistency check vs. related LWT/Paxos work:

Fix LWT routing: preserve Paxos leader order in TokenAwarePolicy #782 (fix/lwt-paxos-leader-routing, not yet merged): that PR fixes TokenAwarePolicy.make_query_plan so all clients converge on the same natural-order coordinator for a given key (avoiding dueling Paxos proposers). This PR's "retry CAS/serial timeouts on the same host" is complementary, not overlapping — routing decides which host is tried first, this policy decides whether a timeout on that host should escalate to a different host or keep hammering the same coordinator. The two changes reinforce the same principle (stick to one coordinator per key) and touch disjoint code (routing vs. retry), so there's no conflict. The value of "retry same host" is maximized once Fix LWT routing: preserve Paxos leader order in TokenAwarePolicy #782 lands (otherwise different clients' racks could already disagree on which host is "first"), so it may be worth noting that dependency in the PR description, but there's no code-level coupling.
ad8636e86/ca42a5478 (already on master):ad8636e86 makes TokenAwarePolicy skip shuffling for SERIAL/LOCAL_SERIAL too (routing side); ca42a5478 adds a guard in ResponseFuture._retry that refuses to let any retry policy downgrade SERIAL/LOCAL_SERIAL to non-serial consistency. LWTRetryPolicy never attempts a downgrade — its serial/CAS branches always return the original consistency unchanged — so the guard is a no-op safety net here, not a conflict.

Testing:

tests/unit/test_policies.py: 106 passed (21 new LWTRetryPolicyTest cases + all pre-existing, including the master-added test_serial_consistency_not_downgraded / test_no_shuffle_for_serial_consistency).
Full tests/unit/: 742 passed, 88 skipped (pre-existing skips, unrelated to this change), 0 failed.
CI on the PR is green (build + all integration test matrix legs + snyk).

No unresolved review threads found. Rebased and force-pushed (d2a853889 → 0fe5e65a6); still a single commit, still draft.

Copilot

Pull request overview

Note

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Adds an LWT-aware retry policy that prefers retrying CAS/serial operations on the same host with exponential backoff to reduce Paxos contention, plus unit tests validating the behavior.

Changes:

Introduced LWTRetryPolicy extending ExponentialBackoffRetryPolicy with LWT-specific handling for CAS write timeouts, serial read timeouts, and serial unavailable.
Added a new unit test suite (LWTRetryPolicyTest) covering same-host retries, backoff behavior, max-retry handling, and delegation to base policy for non-LWT cases.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
cassandra/policies.py	Adds `LWTRetryPolicy` with LWT-aware retry decisions and backoff.
tests/unit/test_policies.py	Adds unit tests for `LWTRetryPolicy` behavior and delegation paths.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+ def test_cas_write_timeout_retries_same_host(self):
+ """CAS write timeout on first attempt should retry on SAME host."""
+ policy = self._make_policy()
+ retry, consistency, delay = policy.on_write_timeout(
+ query=None, consistency=ConsistencyLevel.QUORUM,
+ write_type=WriteType.CAS,
+ required_responses=3, received_responses=1, retry_num=0)
+ assert retry == RetryPolicy.RETRY
+ assert consistency == ConsistencyLevel.QUORUM
+ assert delay is not None and delay > 0


+ def _make_policy(self, max_retries=3):
+ return LWTRetryPolicy(max_num_retries=max_retries)


+ LWT queries use Paxos consensus, where the first replica in the token ring
+ acts as the Paxos coordinator (leader). Retrying LWT queries on a *different*
+ host causes Paxos contention — the new coordinator must compete with the
+ original one, potentially causing cascading timeouts.


mykaul mentioned this pull request Apr 1, 2026
Fix SimpleStatement.is_lwt(): detect LWT from CQL query string #784
Closed

mykaul marked this pull request as draft April 1, 2026 12:46

mykaul force-pushed the feature/lwt-retry-policy branch from f1a865b to d2a8538Compare April 7, 2026 16:08

Lorak-mmk force-pushed the master branch from f2a9e87 to 763af09Compare June 15, 2026 10:57

CopilotAI review requested due to automatic review settings July 29, 2026 20:26

mykaul force-pushed the feature/lwt-retry-policy branch from d2a8538 to 0fe5e65Compare July 29, 2026 20:26

CopilotAI reviewed Jul 29, 2026
View reviewed changes

Copilotstarted reviewing on behalf of mykaul July 29, 2026 20:54 View session

Navigation Menu

Uh oh!

Add LWTRetryPolicy: retry CAS timeouts on same host with backoff - #783

Add LWTRetryPolicy: retry CAS timeouts on same host with backoff#783
mykaul wants to merge 1 commit into
scylladb:masterfrom
mykaul:feature/lwt-retry-policy

mykaul commented Apr 1, 2026

Uh oh!

mykaul commented Apr 7, 2026

Uh oh!

calebxyz commented Apr 7, 2026

Uh oh!

mykaul commented Apr 7, 2026

Uh oh!

calebxyz commented Apr 7, 2026

Uh oh!

mykaul commented Apr 7, 2026

Uh oh!

coderabbitaiBot commented Jul 29, 2026

Review skipped

Uh oh!

mykaul commented Jul 29, 2026

Uh oh!

CopilotAI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		def _make_policy(self, max_retries=3):
		return LWTRetryPolicy(max_num_retries=max_retries)

Uh oh!

Conversation

mykaul commented Apr 1, 2026

Summary

Usage

Changes

Tests

Related

Uh oh!

mykaul commented Apr 7, 2026

Uh oh!

calebxyz commented Apr 7, 2026

Uh oh!

mykaul commented Apr 7, 2026

Uh oh!

calebxyz commented Apr 7, 2026

Uh oh!

mykaul commented Apr 7, 2026

Uh oh!

coderabbitaiBot commented Jul 29, 2026

Review skipped

Uh oh!

mykaul commented Jul 29, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants