Uh oh!
There was an error while loading. Please reload this page.
[fix](filecache) Limit LRU recorder shadow queue size - #64798
Merged
Conversation
### What problem does this PR solve? Issue Number: None Related PR: https://github.com/selectdb/selectdb-core/pull/9858 Problem Summary: The file cache LRU recorder can replay more records into the shadow queue than the configured dump-tail limit, allowing the shadow queue to keep growing after long-running replay. This change trims replayed shadow queues to file_cache_background_lru_dump_tail_record_num, exposes shadow queue element counts in file cache stats, and adds BE UT coverage for tail retention, remove-event trimming, metric refresh, and replay queue cap behavior. ### Release note None ### Check List (For Author) - Test: Build and unit test - Build: DORIS_TOOLCHAIN=clang DISABLE_BE_JAVA_EXTENSIONS=ON ENABLE_INJECTION_POINT=ON ENABLE_CACHE_LOCK_DEBUG=0 ENABLE_PCH=0 ./build.sh --be - Unit Test: DORIS_TOOLCHAIN=clang DISABLE_BE_JAVA_EXTENSIONS=ON ENABLE_INJECTION_POINT=ON ENABLE_CACHE_LOCK_DEBUG=0 ENABLE_PCH=0 sh run-be-ut.sh --run --filter=CacheLRUDumperTest.* - Behavior changed: Yes. The LRU recorder shadow queue is capped by file_cache_background_lru_dump_tail_record_num during replay. - Does this need documentation: No
freemandealer
requested review from
Gabriel39, gavinchou, liaoxin01 and morningman
as code ownersJune 24, 2026 14:16
hello-stephen
commented
Jun 24, 2026
Contributor
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
freemandealer
commented
Jun 24, 2026
MemberAuthor
run buildall |
freemandealer
commented
Jun 24, 2026
MemberAuthor
/review |
Contributor
There was a problem hiding this comment.
I reviewed the full PR diff and the surrounding file-cache LRU recorder/dumper paths. I did not find a substantiated blocking issue to raise inline.
Critical checkpoint conclusions:
- Goal/test coverage: The PR caps replayed shadow queue growth and exposes queue-size stats; added BE UTs cover tail retention, remove-event trimming, metrics refresh, and replay queue cap behavior.
- Scope: The change is localized to BE file-cache LRU bookkeeping and tests.
- Concurrency/lifecycle: The new shadow-queue metric refresh uses the existing
_mutex_lru_log; no new cache-lock/lru-log nested lock order was introduced. - Configuration/compatibility: Existing mutable BE configs are reused; no FE/session propagation, storage format, or RPC compatibility path changes.
- Parallel paths: All four file-cache queue types are handled consistently through the existing replay/dump type lists.
- Observability: The new stats entries expose current shadow queue element counts.
- User focus: No additional user-provided review focus was supplied.
- Tests: I verified
git diff --checkfor the changed files. I did not rerun the BE UT binary locally.
Subagent conclusions:
optimizer-rewrite:OR-NONE; no optimizer/rewrite, semantic-equivalence, or parallel join/aggregate issue because the PR only touches BE file-cache bookkeeping.tests-session-config:TSC-NONE; no regression/session/config/compatibility/style issue found; subagent also rangit diff --check.- Convergence: Round 1 ended with both live subagents replying
NO_NEW_VALUABLE_FINDINGSfor the same current ledger/comment set after the final main status update.
hello-stephen
commented
Jun 24, 2026
Contributor
TPC-H: Total hot run time: 29199 ms |
hello-stephen
commented
Jun 24, 2026
Contributor
TPC-DS: Total hot run time: 172489 ms |
hello-stephen
commented
Jun 24, 2026
Contributor
ClickBench: Total hot run time: 25.33 s |
### What problem does this PR solve?\n\nIssue Number: None\n\nRelated PR: apache#64798\n\nProblem Summary: The lru queue recorder now trims shadow replay queues to the configured tail size during restart replay. BlockFileCacheTest.version3_add_remove_restart still asserted the pre-change queue length of 5 after replaying 5 records, but the new behavior keeps only the last 2 records when file_cache_background_lru_dump_tail_record_num is set to 2. Update the test to match the new replay semantics.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test: Regression test\n - run-be-ut.sh --run --filter=BlockFileCacheTest.version3_add_remove_restart\n- Behavior changed: No\n- Does this need documentation: No
freemandealer
commented
Jun 25, 2026
MemberAuthor
run buildall |
hello-stephen
commented
Jun 25, 2026
Contributor
TPC-H: Total hot run time: 29206 ms |
hello-stephen
commented
Jun 25, 2026
Contributor
TPC-DS: Total hot run time: 172678 ms |
hello-stephen
commented
Jun 25, 2026
Contributor
ClickBench: Total hot run time: 25.17 s |
freemandealer
commented
Jun 25, 2026
MemberAuthor
/review |
Contributor
There was a problem hiding this comment.
Review found blocking correctness issues in the capped shadow LRU replay/restore behavior.
Subagent conclusions:
- optimizer-rewrite: OPT-1, OPT-2, and OPT-3 were independently verified and accepted as MAIN-1, MAIN-2, and MAIN-3.
- tests-session-config: TEST-1 was merged into MAIN-1 because the same non-ADD replay fix covers it; convergence round 3 ended with both subagents replying NO_NEW_VALUABLE_FINDINGS for the same three-comment set.
Critical checkpoints:
- Goal/test proof: the PR aims to cap replayed shadow queues, but the current implementation does not preserve the intended LRU order under touch/remove/restart scenarios, and tests mostly assert counts rather than final order.
- Scope: the code change is small, but it changes replay semantics for ADD/REMOVE/MOVETOBACK/RESIZE and restart reconstruction.
- Concurrency/lifecycle: replay uses the recorder mutex; no new lock-order issue found. A lifecycle/order issue was found because LRU dump restore runs before storage metadata loading.
- Configuration: the mutable tail limit is not consistently honored on non-ADD replay paths; covered by the MOVETOBACK comment.
- Parallel paths: ADD, REMOVE, MOVETOBACK, RESIZE, dump, restore, and metadata-load paths were checked. The accepted comments cover the uncovered parallel paths.
- Tests: missing coverage for missing-entry MOVETOBACK, remove backfill after prior cap trimming, and restart final LRU order. I did not run BE UTs because this checkout lacks .worktree_initialized and thirdparty/installed.
- Storage/transaction/FE-BE/security/documentation: no additional issue found in the reviewed scope.
User focus: no additional user-provided focus was supplied.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
hello-stephen
commented
Jun 25, 2026
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
gavinchou
approved these changes
Jun 25, 2026
Contributor
PR approved by at least one committer and no changes requested. |
Uh oh!
There was an error while loading. Please reload this page.
This was referenced Jul 27, 2026
yiguolei pushed a commit
that referenced
this pull request
Jul 29, 2026
…#64798) (#66132) ### What problem does this PR solve? Issue Number: N/A Related PR: #64798 Problem Summary: Pick #64798 into branch-4.1. SelectDB reference: selectdb/selectdb-core#10567. ### Release note None ### Check List (For Author) - Test: - Format: git diff --check passed; modified C++ files checked with build-support/run_clang_format.py using clang-format v16. Full build-support/check-format.sh reports pre-existing unrelated be/src/format_v2/json/json_reader.cpp, not included in this PR. - BE: JAVA_HOME=/mnt/disk1/zhangzhengyu/build-dep/jdk-17.0.2/ DORIS_TOOLCHAIN=clang DISABLE_BE_JAVA_EXTENSIONS=ON ENABLE_INJECTION_POINT=ON ENABLE_CACHE_LOCK_DEBUG=0 ENABLE_PCH=0 ./build.sh --be - BE UT: same environment, sh run-be-ut.sh --run --filter=CacheLRUDumperTest.* - Behavior changed: No - Does this need documentation: No
16 tasks
deardeng added a commit
that referenced
this pull request
Sep 3, 2026
…67315) run_background_monitor() ended its loop with update_shadow_queue_element_count_metrics(), which takes _mutex_lru_log. The LRU log replay thread holds that lock for as long as it takes to drain the log queue, so a slow consumer froze the monitor with it: check_disk_resource_limit(), check_need_evict_cache_in_advance() and every gauge stopped running, and the disk resource limit mode stayed at whatever value it happened to hold. Gauges were observed frozen for up to 40 minutes in production. The call was redundant from the start. #64798 added the shadow queue element count gauge and published it in two places: inside replay_queue_event(), under the same lock that mutates the shadow queue, and again from the monitor every file_cache_background_monitor_interval_ms as a periodic refresh. Nothing outside replay_queue_event() mutates a shadow queue, so that refresh could only rewrite a value that had just been published and could not have changed since. What it did add was a dependency from the disk protection loop onto a lock owned by a background consumer. Drop the call, and update_shadow_queue_element_count_metrics() with it: it existed only for that refresh, and leaving a public method that takes _mutex_lru_log invites the next background loop to reintroduce the coupling. The gauge is still published by replay, now on the replay interval instead of the monitor interval. Its test is replaced by one asserting that replay publishes the gauge on its own. How long replay holds the lock is a separate problem, addressed separately.
github-actionsBot
pushed a commit
that referenced
this pull request
Sep 3, 2026
…67315) run_background_monitor() ended its loop with update_shadow_queue_element_count_metrics(), which takes _mutex_lru_log. The LRU log replay thread holds that lock for as long as it takes to drain the log queue, so a slow consumer froze the monitor with it: check_disk_resource_limit(), check_need_evict_cache_in_advance() and every gauge stopped running, and the disk resource limit mode stayed at whatever value it happened to hold. Gauges were observed frozen for up to 40 minutes in production. The call was redundant from the start. #64798 added the shadow queue element count gauge and published it in two places: inside replay_queue_event(), under the same lock that mutates the shadow queue, and again from the monitor every file_cache_background_monitor_interval_ms as a periodic refresh. Nothing outside replay_queue_event() mutates a shadow queue, so that refresh could only rewrite a value that had just been published and could not have changed since. What it did add was a dependency from the disk protection loop onto a lock owned by a background consumer. Drop the call, and update_shadow_queue_element_count_metrics() with it: it existed only for that refresh, and leaving a public method that takes _mutex_lru_log invites the next background loop to reintroduce the coupling. The gauge is still published by replay, now on the replay interval instead of the monitor interval. Its test is replaced by one asserting that replay publishes the gauge on its own. How long replay holds the lock is a separate problem, addressed separately.
yiguolei pushed a commit
that referenced
this pull request
Sep 5, 2026
…67315) run_background_monitor() ended its loop with update_shadow_queue_element_count_metrics(), which takes _mutex_lru_log. The LRU log replay thread holds that lock for as long as it takes to drain the log queue, so a slow consumer froze the monitor with it: check_disk_resource_limit(), check_need_evict_cache_in_advance() and every gauge stopped running, and the disk resource limit mode stayed at whatever value it happened to hold. Gauges were observed frozen for up to 40 minutes in production. The call was redundant from the start. #64798 added the shadow queue element count gauge and published it in two places: inside replay_queue_event(), under the same lock that mutates the shadow queue, and again from the monitor every file_cache_background_monitor_interval_ms as a periodic refresh. Nothing outside replay_queue_event() mutates a shadow queue, so that refresh could only rewrite a value that had just been published and could not have changed since. What it did add was a dependency from the disk protection loop onto a lock owned by a background consumer. Drop the call, and update_shadow_queue_element_count_metrics() with it: it existed only for that refresh, and leaving a public method that takes _mutex_lru_log invites the next background loop to reintroduce the coupling. The gauge is still published by replay, now on the replay interval instead of the monitor interval. Its test is replaced by one asserting that replay publishes the gauge on its own. How long replay holds the lock is a separate problem, addressed separately.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Problem Summary: The file cache LRU recorder can replay more records into the shadow queue than the configured dump-tail limit, allowing the shadow queue to keep growing after long-running replay. This change trims replayed shadow queues to
file_cache_background_lru_dump_tail_record_num, exposes shadow queue element counts in file cache stats, and adds BE UT coverage for tail retention, remove-event trimming, metric refresh, and replay queue cap behavior.Release note
None
Check List (For Author)
DORIS_TOOLCHAIN=clang DISABLE_BE_JAVA_EXTENSIONS=ON ENABLE_INJECTION_POINT=ON ENABLE_CACHE_LOCK_DEBUG=0 ENABLE_PCH=0 ./build.sh --beDORIS_TOOLCHAIN=clang DISABLE_BE_JAVA_EXTENSIONS=ON ENABLE_INJECTION_POINT=ON ENABLE_CACHE_LOCK_DEBUG=0 ENABLE_PCH=0 sh run-be-ut.sh --run --filter=CacheLRUDumperTest.*file_cache_background_lru_dump_tail_record_numduring replay.