Add asynchronous S3 chunk blob backfill by bootjp · Pull Request #1126 · bootjp/elastickv

bootjp · 2026-07-19T09:46:40Z

Summary

enqueue applied S3 chunk references without blocking the Raft apply goroutine
restore missing local chunk blobs with a bounded worker pool, per-peer token buckets, SHA/size verification, and alternate-peer fallback
recover snapshot-installed or dropped notifications through startup and periodic scans of every local shard store
expose queue depth and outcome metrics and fail startup on invalid resource-bound configuration

Safety

background writes use the same startup-rotation gate as peer push and proxy-on-miss repair
shutdown stops backfill workers before closing peer connections
local scans never use leader routing and queue pressure cannot stall Raft apply

Verification

go test ./... -count=1
go test -race ./adapter -run 'TestS3BlobBackfill|TestS3BlobTokenBucket' -count=3\n- golangci-lint run ./... --timeout=5m --allow-parallel-runners -j 2\n- focused backfill, shard-store, and metrics tests x10\n- git diff --check\n\nStacked on s3: add blob fetch rpc #1063.

coderabbitai · 2026-07-19T09:46:45Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: ee939daa-1370-4298-a9e1-3811f7bc2bfa

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

bootjp · 2026-07-19T09:46:49Z

Signed head a1d6a3ac implements M2 follower apply/backfill. Full Go tests, focused race tests, full lint, repeated worker/scan/rate-limit tests, and diff checks pass.

@codex review

gemini-code-assist

Code Review

This pull request introduces an asynchronous S3 blob backfiller (S3BlobBackfiller) to handle follower catch-up work independently from the Raft apply loop, complete with background workers, periodic scanning, token-bucket rate limiting, and Prometheus metrics. The feedback highlights three key areas for improvement: ensuring that stopping the backfiller properly drains the queue and clears pending keys to prevent permanent blocks, optimizing the local blob existence check to avoid loading entire payloads into memory, and addressing a potential thundering herd issue in the custom token bucket implementation by adopting a stateful reservation mechanism.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:a1d6a3aca6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

bootjp · 2026-07-19T10:53:04Z

Signed head 100acd3 addresses every current review finding.

Changes:

drain queued and pending work on Stop and safely support restart
use ExistsAt for local presence checks
reserve token-bucket slots across concurrent workers and restore canceled reservations
write peer-local repairs at the replicated chunkref commit timestamp without a follower TSO allocation
atomically publish and clear the backfill server pointer
route metadata, local blobs, and peer discovery by the chunkref placement key
resolve lint findings by extracting helpers and parsing runtime configuration outside run

Caller audit:

all storeFetchedS3ChunkBlob callers now pass the chunkref route key and replicated commit timestamp
proxy-on-miss repair and asynchronous backfill are the only production callers
OnApply, worker, scanner, and observer paths load the server through the atomic pointer

Verification:

focused adapter, monitoring, and main tests repeated 20 times: pass
focused adapter race tests repeated 5 times: pass
full golangci-lint: 0 issues
git diff --check: pass
both commits verified with a valid bootjp contact@bootjp.me signature

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:100acd365f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

bootjp · 2026-07-19T11:13:53Z

Rebased the signed M2 commits onto #1063 head 0a7eed9. Current signed head is 07b963f; the M2 diff is unchanged and now includes the base multipart isolation fix.

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:07b963fedc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

bootjp · 2026-07-19T11:27:35Z

History rebuilt without content changes and restacked on the sanitized #1063 head.

Previous head: 07b963fedcb35d121296e871931296c2e1630922
Current head: b374525c5279d529356e2083dbcdb322e6e2b367
Tree before/after: ad0dc6b36b88e7478cd05421c10981a11757984f
git diff --exit-code <previous> <current>: clean
Parent: 02e3a6d5b4fe559dbfc0144313995332736e1b30
PR-visible commits: one
Author and committer: bootjp <contact@bootjp.me>
GitHub signature verification: valid

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:b374525c52

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

bootjp · 2026-07-19T12:09:18Z

Current signed head eaaa08bb45fd0fb73f46015024b2457cdc3065ef is restacked on corrected #1063 and addresses both current P2 findings.

Changes:

self-heal corrupt same/newer local chunkblob versions with a peer-verified local repair timestamp while preserving the store-wide MVCC watermark
validate local blob content during backfill scans instead of treating key existence as healthy
start a listenless backfill lifecycle on peer-token nodes with no public S3 address

Caller audit:

storeFetchedS3ChunkBlob has two production callers: GET proxy-on-miss and async backfill
normal gRPC push and local replication retain the existing fail-closed storeChunkBlob semantics
peer repair remains follower-safe and does not allocate through TSO

Verification:

focused repair/backfill/main lifecycle tests
focused tests repeated 20 times
focused race tests
all-package compile
full golangci-lint: 0 issues
proto breaking and regenerated diff checks
git diff --check
PR-visible author/committer and signature verification: valid

@codex review

chatgpt-codex-connector · 2026-07-19T12:19:16Z

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit:eaaa08bb45

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

This document declares itself the canonical ownership index, so rows that call finished work unimplemented and unowned send contributors off to write a competing design. Seven rows named work that is already an ancestor of this commit: - route catalog delta watch (#1117, 6c7a66e) -> implemented owner - SST ingest snapshot transfer (#1130, 910a97e) -> implemented owner - leader-proxy circuit breaker (#1132, 56e36e9) -> implemented owner - physical snapshot offload (#1131, 764db2d) -> partial owner, which records M0/M1 implemented and still owns M2/M3 - TSO ceiling state machine (#1150, 0e85c82) -> merged, separated from the group-0 issuance work that remains open - autosplit committed-window reader (#1152, afec059) -> M3 is partially implemented, not absent - S3 offload transport and backfill (77ea547, #1126/5eaaa05d) -> the transport path is merged; the focused design names reference counting, GC readiness, and legacy migration as the remaining blockers Each cited commit was confirmed to be an ancestor of this revision, and every referenced design document was confirmed to exist. The completion rule also could not be satisfied by every row: it demanded each implemented row's focused owner be promoted to `implemented`, but the shared Pebble cache row names only PR #1082 as its canonical owner and a pull request has no status to promote. That made the roadmap permanently ineligible for promotion regardless of what shipped. Merged implementation evidence now closes such a row; an open pull request still does not. Claude-Session: https://claude.ai/code/session_013rNHooj7NF3giihWVba8QE

gemini-code-assistBot reviewed Jul 19, 2026
View reviewed changes

Comment threadadapter/s3_blob_backfill.go
Comment threadadapter/s3_blob_backfill.go Outdated
Comment threadadapter/s3_blob_backfill.go

chatgpt-codex-connectorBot reviewed Jul 19, 2026
View reviewed changes

Comment threadadapter/s3_blob_backfill.go Outdated
Comment threadadapter/s3_blob_backfill.go Outdated

bootjp force-pushed the feature/s3-blob-offload-m2 branch from a1d6a3a to 100acd3Compare July 19, 2026 10:48

chatgpt-codex-connectorBot reviewed Jul 19, 2026
View reviewed changes

Comment threadadapter/s3_blob_read.go Outdated
Comment threadmain_s3.go
Comment threadadapter/s3_blob_backfill.go

bootjp force-pushed the feature/s3-blob-offload-m2 branch from 100acd3 to 07b963fCompare July 19, 2026 11:13

bootjp force-pushed the design/s3-blob-offload-rpc branch from 0a7eed9 to 02e3a6dCompare July 19, 2026 11:25

chatgpt-codex-connectorBot reviewed Jul 19, 2026
View reviewed changes

Comment threadadapter/s3_blob_backfill.go
Comment threadadapter/s3_blob_read.go Outdated
Comment threadmain_s3.go

bootjp force-pushed the feature/s3-blob-offload-m2 branch from 07b963f to b374525Compare July 19, 2026 11:26

chatgpt-codex-connectorBot reviewed Jul 19, 2026
View reviewed changes

Comment threadadapter/s3_blob_read.go Outdated
Comment threadmain_s3.go

bootjp force-pushed the design/s3-blob-offload-rpc branch from 02e3a6d to 77ea547Compare July 19, 2026 11:52

Add asynchronous S3 blob backfill
eaaa08b

bootjp force-pushed the feature/s3-blob-offload-m2 branch from b374525 to eaaa08bCompare July 19, 2026 12:06

Base automatically changed from design/s3-blob-offload-rpc to mainJuly 19, 2026 14:17

bootjp merged commit 5eaaa05 into mainJul 20, 2026
11 checks passed

bootjp deleted the feature/s3-blob-offload-m2 branch July 20, 2026 04:39

bootjp mentioned this pull request Aug 23, 2026
docs: audit June 23 scaling roadmap ownership #1112
Open

Conversation

bootjp commented Jul 19, 2026

Summary

Safety

Verification

Uh oh!

coderabbitaiBot commented Jul 19, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

bootjp commented Jul 19, 2026

Uh oh!

gemini-code-assistBot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connectorBot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

bootjp commented Jul 19, 2026

Uh oh!

chatgpt-codex-connectorBot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bootjp commented Jul 19, 2026

Uh oh!

chatgpt-codex-connectorBot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bootjp commented Jul 19, 2026

Uh oh!

chatgpt-codex-connectorBot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

bootjp commented Jul 19, 2026

Uh oh!

chatgpt-codex-connectorBot commented Jul 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

coderabbitaiBot commented Jul 19, 2026•
edited
Loading