Uh oh!
There was an error while loading. Please reload this page.
branch-4.1: [fix](file cache) keep the cache monitor off the LRU recorder lock #67315 - #67495
Merged
Merged
Conversation
hello-stephen
commented
Sep 3, 2026
Contributor
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
hello-stephen
commented
Sep 3, 2026
Contributor
run buildall |
…67315) run_background_monitor() ended its loop with update_shadow_queue_element_count_metrics(), which takes _mutex_lru_log. The LRU log replay thread holds that lock for as long as it takes to drain the log queue, so a slow consumer froze the monitor with it: check_disk_resource_limit(), check_need_evict_cache_in_advance() and every gauge stopped running, and the disk resource limit mode stayed at whatever value it happened to hold. Gauges were observed frozen for up to 40 minutes in production. The call was redundant from the start. #64798 added the shadow queue element count gauge and published it in two places: inside replay_queue_event(), under the same lock that mutates the shadow queue, and again from the monitor every file_cache_background_monitor_interval_ms as a periodic refresh. Nothing outside replay_queue_event() mutates a shadow queue, so that refresh could only rewrite a value that had just been published and could not have changed since. What it did add was a dependency from the disk protection loop onto a lock owned by a background consumer. Drop the call, and update_shadow_queue_element_count_metrics() with it: it existed only for that refresh, and leaving a public method that takes _mutex_lru_log invites the next background loop to reintroduce the coupling. The gauge is still published by replay, now on the replay interval instead of the monitor interval. Its test is replaced by one asserting that replay publishes the gauge on its own. How long replay holds the lock is a separate problem, addressed separately.
yiguoleiforce-pushed
the
auto-pick-67315-branch-4.1
branch
from
September 5, 2026 22:07
2e844d4 to
c088c4cCompareyiguolei
commented
Sep 5, 2026
Contributor
run buildall |
hello-stephen
commented
Sep 6, 2026
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
commented
Sep 6, 2026
Contributor
skip buildall |
yiguolei
approved these changes
Sep 6, 2026
ContributorAuthor
PR approved by anyone and no changes requested. |
ContributorAuthor
PR approved by at least one committer and no changes requested. |
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picked from #67315