Share Pebble block cache across stores by bootjp · Pull Request #1082 · bootjp/elastickv

bootjp · 2026-07-13T05:40:06Z

Author: bootjp

Summary:

Reuse one process-wide Pebble block cache across all stores opened in the node process.
Size the cache to 25% of the smallest discovered GOMEMLIMIT, cgroup limit, or physical RAM size by default; support percentage and absolute-MiB overrides.
Keep explicit per-open cache references across NewPebbleStore, restore reopen paths, and temporary restore DBs, releasing each borrowed reference on close or error.
Emit shared block-cache capacity, size, hit, and miss metrics once per node while keeping LSM, memtable, and compaction metrics group-scoped.
Update the scaling roadmaps to mark shared block-cache M1 complete; shared memtable/compaction budgets and cache fairness remain separate follow-up milestones.

Validation:

go test ./store ./monitoring -count=1 -timeout=10m
go test -race ./store ./monitoring -run "Test(PebbleCacheEnvOverride|MemoryBudgetHelpers|DefaultPebbleOptionsSharesProcessCacheConcurrently|NewPebbleStoreSharesProcessCache|PebbleStoreRestoreKeepsProcessCache|PebbleCollectorEmitsSharedBlockCacheOnce|PebbleCollectorHandlesSourceReset)$" -count=1 -timeout=10m
ELASTICKV_PEBBLE_CACHE_MB=64 go test . ./kv -count=1 -timeout=15m
go test ./... -run "^$" -count=1 -timeout=15m
GOOS=linux GOARCH=amd64 go test -c ./store -o /tmp/elastickv-store-linux.test
golangci-lint --config=.golangci.yaml run ./... --timeout=10m
git diff --check

bootjp · 2026-07-13T05:40:17Z

@codex review

coderabbitai · 2026-07-13T05:40:18Z

Warning

Review limit reached

@bootjp, you've reached your PR review limit, so we couldn't start this review.

Next review available in:18 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: e18d6ff2-3455-40df-af75-dcf9d191b3f1

📥 Commits

Reviewing files that changed from the base of the PR and between 910a97e and b4ab1a6.

📒 Files selected for processing (10)

docs/design/2026_06_12_proposed_scaling_roadmap.md
docs/design/2026_06_23_proposed_scaling_roadmap.md
monitoring/pebble.go
monitoring/pebble_test.go
store/lsm_store.go
store/lsm_store_env_test.go
store/pebble_memory_budget.go
store/pebble_memory_darwin.go
store/pebble_memory_linux.go
store/pebble_memory_other.go

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

gemini-code-assist

Code Review

This pull request implements a process-wide shared Pebble block cache across all stores in a node, replacing the previous behavior where each store allocated its own private 256 MiB cache. This change addresses the high memory overhead when running multiple Raft groups on a single node. The design roadmaps have been updated to reflect that this milestone (M1) has landed, and new tests have been added to verify that multiple stores correctly share the same cache instance. I have no feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:9eec7e7f77

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

bootjp · 2026-07-13T08:23:00Z

Addressed the shared block-cache metrics finding.

Changes:

Emit elastickv_pebble_block_cache_{size,capacity}_bytes and {hits,misses}_total once per process/node instead of once per raft group.
Keep L0, compaction, and memtable metrics group-scoped.
Split block-cache counter state from per-group compaction counter state so shared cache counters are not overcounted.
Added a regression test proving two group sources produce a single block-cache series.

Validation:

GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./monitoring -run 'TestPebbleCollector|TestSetFSMApplySyncMode' -count=1 -timeout=120s
GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./store -run 'Test(DefaultPebbleOptions|NewPebbleStoreSharesProcessCache|PebbleCacheEnvOverride|SetSmallPebbleCacheForTestRestores)' -count=1 -timeout=180s
GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./monitoring ./store -count=1 -timeout=300s
GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./... -run '^$' -count=1 -timeout=300s
GOCACHE=$(pwd)/.cache GOLANGCI_LINT_CACHE=$(pwd)/.golangci-cache golangci-lint run ./monitoring ./store --timeout=5m
git diff --check

@codex review

chatgpt-codex-connector · 2026-07-13T08:28:52Z

Codex Review: Didn't find any major issues. Swish!

Reviewed commit:d8ae776d61

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

bootjp · 2026-07-18T11:02:56Z

Design requirement audit completed on latest origin/main.

HEAD: bf5f0f6b67baa4e0eb0f2ff37fcf7d2113d766ce (verified signed commit by bootjp <contact@bootjp.me>)

Implemented and verified:

One process-wide pebble.Cache is shared by all production store opens.
The default capacity is 25% of the smallest discovered GOMEMLIMIT, cgroup limit, or physical RAM size.
ELASTICKV_PEBBLE_CACHE_PERCENT configures the fraction; the existing ELASTICKV_PEBBLE_CACHE_MB absolute override takes precedence.
NewPebbleStore, empty restore reopen, native restore temp/open, streaming restore temp/open, swap reopen, open errors, metadata errors, and Close each borrow and release exactly one explicit cache reference.
Closing one store does not invalidate another store using the shared cache.
Block-cache size/capacity/hit/miss metrics are emitted once per node; LSM, memtable, and compaction signals remain group-scoped.

Caller audit:

resolvePebbleCacheBytes has one production caller: package init; all other callers are tests.
Every production defaultPebbleOptionsWithCache caller and every pebble.Open in store/lsm_store.go was audited for matching Close/Unref behavior.
Direct pebble.Open calls outside that factory are test-only inspection fixtures.

Validation:

go test ./store ./monitoring -count=1 -timeout=10m
targeted go test -race ./store ./monitoring ... -count=1 -timeout=10m
ELASTICKV_PEBBLE_CACHE_MB=64 go test . ./kv -count=1 -timeout=15m
go test ./... -run "^$" -count=1 -timeout=15m
Linux store test binary cross-compile
golangci-lint --config=.golangci.yaml run ./... --timeout=10m (0 issues)
git diff --check

@codex review

bootjp · 2026-07-18T11:08:41Z

Addressed the latest-head Linux lint finding.

HEAD: adb2d0a73f6c94778be6bbe8f6589c9e12b9c1b4

Removed the redundant uint64 conversion on unix.Sysinfo_t.Totalram.
GOOS=linux GOARCH=amd64 go test -c ./store -o /tmp/elastickv-store-linux.test: pass.
GOOS=linux GOARCH=amd64 golangci-lint --config=.golangci.yaml run ./store --timeout=5m --allow-parallel-runners: 0 issues.
Commit is signed and authored/committed by bootjp <contact@bootjp.me>.

@codex review

bootjp · 2026-07-18T11:24:57Z

@codex review

bootjp · 2026-07-18T11:28:04Z

@coderabbitai review

coderabbitai · 2026-07-18T11:28:12Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:adb2d0a73f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

bootjp · 2026-07-19T09:59:40Z

Addressed both latest-head memory-budget findings in signed commit 467dd08.

walk cgroup v2 memory.max from the process leaf through every ancestor
size the off-heap shared cache only from hard cgroup or physical capacity, not GOMEMLIMIT
add v1/v2 ancestor-path and boundary regression tests
audit the sole production memory-budget caller and update its contract

Verification: targeted store tests, Linux cross-compile, race tests, Shell-independent diff checks, and the signed commit lint hook passed. Current-head CI is running.

@codex review

chatgpt-codex-connector · 2026-07-19T10:11:12Z

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit:467dd08b9a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

bootjp · 2026-07-19T14:01:45Z

Rebased onto current main (including #1130 SST-ingest snapshot transfer) and resolved the restore/cache lifecycle overlap.

Current signed head: b4ab1a666c890ef2294b38eee34d074ad057be43

Resolution:

preserve Add Pebble SST ingest snapshot transfer #1130's centralized reopenStoreDB / restoreSwapBackup cleanup path, which already closes the replacement DB and unreferences its shared cache on metadata failure
retain process-wide shared-cache sizing and reference ownership from this PR
restore the generic test-only cache-capacity helper required by the merged SST snapshot tests

Verification:

go test ./store ./monitoring -count=1 -timeout=20m
focused go test -race ./store ./monitoring
go test ./internal/raftengine/etcd -count=1 -timeout=20m (isolated rerun passed)
full go test ./... -count=1 -timeout=20m: adapter passed; one etcd package run failed only under concurrent multi-worktree load, then passed in isolation
target and full commit-hook golangci-lint: 0 issues
all six PR-visible commits are signed by bootjp <contact@bootjp.me>
git diff --check

@codex review

chatgpt-codex-connector · 2026-07-19T14:11:09Z

Codex Review: Didn't find any major issues. 👍

Reviewed commit:b4ab1a666c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Review findings on the current HEAD: - The hotspot split M2 and S3 blob offload rows cited the pre-rename *_proposed_* filenames. Both docs were promoted to *_partial_* and the pointers were left behind. - The streaming transport row and the §5 gap table still called the multi-group soak evidence outstanding. It landed with the design's §8: cmd/elastickv-raft-stream-soak, the Jepsen soak script, and a fail-closed verifier over docs/evidence/raft_streaming_multigroup_soak.json. The reviewer also flagged that the cited PR states were a 2026-07-19 snapshot and needed a spot-check. Re-audited every cited PR against GitHub: #1096, #1097, #1108, #1082, #1057, #1063, and #1059 have merged since. Rows updated and the audit date moved to 2026-08-22. Claude-Session: https://claude.ai/code/session_013rNHooj7NF3giihWVba8QE

This document declares itself the canonical ownership index, so rows that call finished work unimplemented and unowned send contributors off to write a competing design. Seven rows named work that is already an ancestor of this commit: - route catalog delta watch (#1117, 6c7a66e) -> implemented owner - SST ingest snapshot transfer (#1130, 910a97e) -> implemented owner - leader-proxy circuit breaker (#1132, 56e36e9) -> implemented owner - physical snapshot offload (#1131, 764db2d) -> partial owner, which records M0/M1 implemented and still owns M2/M3 - TSO ceiling state machine (#1150, 0e85c82) -> merged, separated from the group-0 issuance work that remains open - autosplit committed-window reader (#1152, afec059) -> M3 is partially implemented, not absent - S3 offload transport and backfill (77ea547, #1126/5eaaa05d) -> the transport path is merged; the focused design names reference counting, GC readiness, and legacy migration as the remaining blockers Each cited commit was confirmed to be an ancestor of this revision, and every referenced design document was confirmed to exist. The completion rule also could not be satisfied by every row: it demanded each implemented row's focused owner be promoted to `implemented`, but the shared Pebble cache row names only PR #1082 as its canonical owner and a pull request has no status to promote. That made the roadmap permanently ineligible for promotion regardless of what shipped. Merged implementation evidence now closes such a row; an open pull request still does not. Claude-Session: https://claude.ai/code/session_013rNHooj7NF3giihWVba8QE

gemini-code-assistBot reviewed Jul 13, 2026
View reviewed changes

chatgpt-codex-connectorBot reviewed Jul 13, 2026
View reviewed changes

Comment threadstore/lsm_store.go

bootjp force-pushed the design/shared-pebble-cache branch from d8ae776 to bf5f0f6Compare July 18, 2026 11:01

github-actionsBot reviewed Jul 18, 2026
View reviewed changes

Comment threadstore/pebble_memory_linux.go Outdated

chatgpt-codex-connectorBot reviewed Jul 18, 2026
View reviewed changes

Comment threadstore/pebble_memory_budget.go Outdated
Comment threadstore/pebble_memory_budget.go Outdated

bootjp added 6 commits July 19, 2026 22:30

Share Pebble block cache across stores
00a16a0

monitoring: emit shared cache metrics once
0500937

store: size shared cache from node memory
604354d

store: remove redundant memory conversion
e10bfa5

Harden shared cache memory budgeting
f84ecba

store: preserve cache test override
b4ab1a6

bootjp force-pushed the design/shared-pebble-cache branch from 467dd08 to b4ab1a6Compare July 19, 2026 14:01

bootjp merged commit acf802e into mainJul 19, 2026
11 checks passed

bootjp deleted the design/shared-pebble-cache branch July 19, 2026 14:15

claudeBot mentioned this pull request Aug 21, 2026
docs: audit June 23 scaling roadmap ownership #1112
Open

Conversation

bootjp commented Jul 13, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bootjp commented Jul 13, 2026

Uh oh!

coderabbitaiBot commented Jul 13, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Uh oh!

gemini-code-assistBot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

chatgpt-codex-connectorBot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

bootjp commented Jul 13, 2026

Uh oh!

chatgpt-codex-connectorBot commented Jul 13, 2026

Uh oh!

bootjp commented Jul 18, 2026

Uh oh!

Uh oh!

bootjp commented Jul 18, 2026

Uh oh!

bootjp commented Jul 18, 2026

Uh oh!

bootjp commented Jul 18, 2026

Uh oh!

coderabbitaiBot commented Jul 18, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connectorBot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

bootjp commented Jul 19, 2026

Uh oh!

chatgpt-codex-connectorBot commented Jul 19, 2026

Uh oh!

bootjp commented Jul 19, 2026

Uh oh!

chatgpt-codex-connectorBot commented Jul 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

bootjp commented Jul 13, 2026•
edited
Loading

coderabbitaiBot commented Jul 13, 2026•
edited
Loading

coderabbitaiBot commented Jul 18, 2026•
edited
Loading