Add leader proxy circuit breaker

[bootjp]

Author: bootjp

Summary

• add a per-Raft-group circuit breaker keyed by leader ID, address, and term
• allow one half-open probe while concurrent requests fail fast through protocol-specific availability responses
• preserve transaction semantics by excluding circuit-open errors from coordinator redispatch
• stop resetting gRPC reconnect backoff on every transient failure

Safety and risk

• no Raft, protobuf, or persisted-data format changes
• caller cancellation does not count as leader failure and releases half-open ownership
• late results from old leader identities cannot overwrite current breaker state
• Redis returns NOTLEADER; DynamoDB, SQS, S3, and Admin return their existing 503 surfaces

Verification

• go test ./... -timeout=20m
• focused go test -race for leader proxy, connection cache, and retry classification
• golangci-lint run ./... --timeout=10m
• git diff --check

[coderabbitai]

Warning

Review limit reached

@bootjp, you've reached your PR review limit, so we couldn't start this review.

Next review available in:49 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: a109db6d-aa84-4661-934b-af640f1ffddc

📥 Commits

Reviewing files that changed from the base of the PR and between 915bc77 and 2266774.

📒 Files selected for processing (16)

• adapter/dynamodb_errors.go
• adapter/redis.go
• adapter/redis_error_prefix_test.go
• adapter/s3.go
• adapter/sqs_catalog.go
• adapter/sqs_query_protocol.go
• adapter/startup_gate_error_test.go
• docs/design/2026_07_19_implemented_leader_proxy_circuit_breaker.md
• kv/coordinator_retry_test.go
• kv/grpc_conn_cache.go
• kv/leader_proxy.go
• kv/leader_proxy_breaker.go
• kv/leader_proxy_breaker_test.go
• kv/leader_proxy_test.go
• main_admin.go
• main_admin_test.go

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[bootjp]

Implemented the leader-proxy circuit breaker design with caller-cancellation handling, stale-identity result rejection, adapter error mappings, and coordinator caller audit.

Verification:

• go test ./... -timeout=20m
• focused go test -race for leader proxy and retry classification
• golangci-lint run ./... --timeout=10m
• signed commit 66bab630d5c9e17845d574c34243e938678f6ef2

@codex review

[gemini-code-assist]

Code Review

This pull request implements a leader proxy circuit breaker to mitigate retry storms during leader elections or network partitions, introducing the transient ErrLeaderProxyCircuitOpen error and mapping it to appropriate 503 or NOTLEADER responses across DynamoDB, SQS, S3, Redis, and Admin adapters. Feedback on the changes suggests optimizing the circuit breaker's fast-fail path by caching the pre-formatted UTC RFC3339Nano string of the openUntil timestamp, thereby avoiding expensive time.Format allocations on every rejected request when the circuit is open.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:66bab630d5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[bootjp]

Implemented the current-head review fixes in signed commit d6b1a71.

Correctness evidence:

• Separated caller cancellation/deadline from the derived proxy retry budget. Proxy-owned DeadlineExceeded failures are now recorded by the breaker; caller cancellation still releases a half-open probe immediately.
• A canceled recovery owner stops retrying and returns the caller context error.
• The request that opens the breaker on repeated Unavailable responses retains recovery ownership and reaches the half-open probe.
• S3 multipart/chunk upload errors now use the same circuit-open/Unavailable classifier and return ServiceUnavailable (503).
• The open-until RFC3339Nano text is cached coherently across open, close, reset, and leader-identity changes.

Caller/classifier/mapping audit:

• Audited every forwardWithRetry/runForwardCycle/forward call and every breaker allow/record/release/reset path.
• Audited all ErrLeaderProxyCircuitOpen, gRPC status, coordinator dispatch, and S3 error-mapping call sites.
• Commit and Abort share the corrected forwarding path; S3 chunk upload, upload-part, PUT, and multipart preparation/load paths now reach the shared retryable mapping.

Verification:

• Focused kv breaker/proxy tests: pass
• Focused kv tests with -race: pass
• Full kv package: pass
• Focused adapter 503 mapping tests: pass
• kv + adapter compile: pass
• golangci-lint ./...: pass (0 issues)
• git diff --check: pass
• Signed commit identity: bootjp contact@bootjp.me

The repository-wide go test ./... was also attempted. It encountered broad pre-existing Redis/raft election timing failures under severe host contention (including adapter and internal/raftengine/etcd), while the changed kv package passed in that run and the focused adapter paths pass independently.

All nine actionable current-head threads have been replied to and resolved.

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:d6b1a71c19

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[bootjp]

Addressed the latest-head P2 findings in signed commit 2266774.

Correctness evidence:

• Half-open admission now preserves the request that opened the breaker as recovery owner. Later requests cannot overwrite that owner after backoff.
• Request-lifetime cleanup releases owner reservation when the caller cancels or its proxy budget ends, so recovery cannot remain permanently reserved.
• CreateMultipartUpload routes coordinator dispatch failures through the S3 mutation mapper and returns ServiceUnavailable (503) for circuit-open and wrapped gRPC Unavailable.
• SQS JSON and Query protocols share one service-unavailable classifier; Query returns HTTP 503 with the ServiceUnavailable XML code.

Audit:

• Re-audited every forwardWithRetry/runForwardCycle/forward path and all breaker allow/record/release/ownership/reset call sites.
• Re-audited all synchronous S3 coordinator Dispatch response paths; each mutation reaches writeS3MutationError or writeS3ResponseOrInternalError.
• Re-audited every SQS Query handler; all errors converge on writeSQSQueryError and the shared classifier.

Verification:

• Focused kv breaker/proxy tests: pass
• Focused kv tests with -race: pass
• Full kv package: pass
• Focused adapter mapping plus existing SQS Query error tests: pass
• golangci-lint ./...: pass (0 issues)
• git diff --check: pass
• Signed author/committer: bootjp contact@bootjp.me

All three current-head P2 threads have been replied to and resolved.

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit:22667744d2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".