Uh oh!
There was an error while loading. Please reload this page.
[enhancement](cloud) refine block file cache evict policy - #42451
Merged
Conversation
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
doris-robot
commented
Oct 24, 2024
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
freemandealer
commented
Oct 24, 2024
MemberAuthor
run buildall |
Contributor
clang-tidy review says "All clean, LGTM! 👍" |
doris-robot
commented
Oct 24, 2024
TPC-H: Total hot run time: 41625 ms |
doris-robot
commented
Oct 24, 2024
TPC-DS: Total hot run time: 192350 ms |
doris-robot
commented
Oct 24, 2024
ClickBench: Total hot run time: 32.8 s |
freemandealer
commented
Oct 25, 2024
MemberAuthor
run p0 |
1 similar comment
freemandealer
commented
Oct 25, 2024
MemberAuthor
run p0 |
Contributor
clang-tidy review says "All clean, LGTM! 👍" |
Uh oh!
There was an error while loading. Please reload this page.
gavinchou
commented
Oct 27, 2024
Contributor
This PR introduce some behavior change, pls describe what it is |
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
Contributor
clang-tidy review says "All clean, LGTM! 👍" |
| context.cache_type = io::FileCacheType::TTL; | ||
| context.expiration_time = UnixSeconds() + 3600; | ||
| auto sp = SyncPoint::get_instance(); |
Contributor
There was a problem hiding this comment.
warning: 'auto sp' can be declared as 'auto *sp' [readability-qualified-auto]
Suggested change
| auto sp = SyncPoint::get_instance(); | |
| s);auto * |
Comment on lines
+4901
to
+4902
| ; | ||
| EXPECT_EQ(cache._cur_cache_size, 30); |
Contributor
There was a problem hiding this comment.
warning: statement should be inside braces [readability-braces-around-statements]
Suggested change
| ; | |
| EXPECT_EQ(cache._cur_cache_size, 30); | |
| he) { | |
| ; | |
| } |
| fs::remove_all(cache_base_path); | ||
| } | ||
| fs::create_directories(cache_base_path); | ||
| auto sp = SyncPoint::get_instance(); |
Contributor
There was a problem hiding this comment.
warning: 'auto sp' can be declared as 'auto *sp' [readability-qualified-auto]
Suggested change
| auto sp = SyncPoint::get_instance(); | |
| h);auto * |
| } | ||
| } | ||
| TEST_F(BlockFileCacheTest, populate_empty_cache_with_disposable) { |
Contributor
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5285: 103 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
| TEST_F(BlockFileCacheTest, disposable_seize_after_normal) { |
Contributor
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5390: 115 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
| TEST_F(BlockFileCacheTest, evict_privilege_order) { |
Contributor
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5507: 172 lines including whitespace and comments (threshold 80)
^freemandealerforce-pushed
the
evict-policy
branch
from
October 31, 2024 11:54
c10721b to
a91968aCompare| } | ||
| } | ||
| TEST_F(BlockFileCacheTest, populate_empty_cache_with_normal) { |
Contributor
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5390: 102 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
| TEST_F(BlockFileCacheTest, populate_empty_cache_with_index) { |
Contributor
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5494: 102 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
| TEST_F(BlockFileCacheTest, populate_empty_cache_with_ttl) { |
Contributor
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5598: 102 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
| TEST_F(BlockFileCacheTest, disposable_seize_after_normal) { |
Contributor
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5702: 115 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
| TEST_F(BlockFileCacheTest, evict_privilege_order_for_disposable) { |
Contributor
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5819: 172 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
| TEST_F(BlockFileCacheTest, evict_privilege_order_for_normal) { |
Contributor
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5993: 171 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
| TEST_F(BlockFileCacheTest, evict_privilege_order_for_index) { |
Contributor
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:6166: 171 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
| TEST_F(BlockFileCacheTest, evict_privilege_order_for_ttl) { |
Contributor
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:6339: 171 lines including whitespace and comments (threshold 80)
^Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
freemandealerforce-pushed
the
evict-policy
branch
from
October 31, 2024 12:40
a91968a to
8e005c2Compare| } | ||
| } | ||
| TEST_F(BlockFileCacheTest, seize_after_full) { |
Contributor
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5819: 155 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
| TEST_F(BlockFileCacheTest, evict_privilege_order_for_disposable) { |
Contributor
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5976: 172 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
| TEST_F(BlockFileCacheTest, evict_privilege_order_for_normal) { |
Contributor
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:6150: 171 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
| TEST_F(BlockFileCacheTest, evict_privilege_order_for_index) { |
Contributor
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:6323: 171 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
| TEST_F(BlockFileCacheTest, evict_privilege_order_for_ttl) { |
Contributor
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:6496: 171 lines including whitespace and comments (threshold 80)
^Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
doris-robot
commented
Nov 1, 2024
TeamCity be ut coverage result: |
doris-robot
commented
Nov 1, 2024
TPC-H: Total hot run time: 41575 ms |
doris-robot
commented
Nov 1, 2024
TPC-DS: Total hot run time: 196015 ms |
doris-robot
commented
Nov 1, 2024
ClickBench: Total hot run time: 32.62 s |
freemandealer
commented
Nov 1, 2024
MemberAuthor
run cloud_p0 |
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
freemandealer
commented
Nov 1, 2024
MemberAuthor
run buildall |
doris-robot
commented
Nov 1, 2024
TPC-H: Total hot run time: 41512 ms |
doris-robot
commented
Nov 1, 2024
TPC-DS: Total hot run time: 190963 ms |
doris-robot
commented
Nov 1, 2024
ClickBench: Total hot run time: 31.68 s |
doris-robot
commented
Nov 1, 2024
TeamCity be ut coverage result: |
| _evict_by_heat_metrics_matrix[FileCacheType::DISPOSABLE][FileCacheType::NORMAL] = | ||
| std::make_shared<bvar::Adder<size_t>>(_cache_base_path.c_str(), | ||
| "file_cache_evict_by_heat_disposable_to_normal"); |
Contributor
There was a problem hiding this comment.
"file_cache_evict_by_heat" -> "file_cache_evict_by_time"
| _hit_ratio_1h = std::make_shared<bvar::Status<double>>(_cache_base_path.c_str(), | ||
| "file_cache_hit_ratio_1h", 0.0); | ||
| _disk_limit_mode_metrics = | ||
| std::make_shared<bvar::Status<size_t>>(_cache_base_path.c_str(), "disk_limit_mode", 0); |
Contributor
There was a problem hiding this comment.
"disk_limit_mode" -> "file_cache_disk_limit_mode"
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
gavinchou
approved these changes
Nov 4, 2024
Contributor
PR approved by at least one committer and no changes requested. |
Contributor
PR approved by anyone and no changes requested. |
pingchunzhang
approved these changes
Nov 4, 2024
freemandealer added a commit
to freemandealer/doris
that referenced
this pull request
Nov 4, 2024
- Enhance cache performance and reduce cache misses by: - Prevent starvation of any particular cache. - Improve disk space utilization. - Maintain strict and systematic eviction priorities. - Enhance the observability of caching strategies, allowing cache behavior to be intuitively self-explanatory through monitoring. - Improve code comprehensibility: unify the cache framework to avoid ad-hoc handling of TTL, increasing the modularity of the code. - Reduce operational complexity by eliminating and standardizing configuration items. --------- Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
dataroaring pushed a commit
that referenced
this pull request
Nov 4, 2024
…43201) - Enhance cache performance and reduce cache misses by: - Prevent starvation of any particular cache. - Improve disk space utilization. - Maintain strict and systematic eviction priorities. - Enhance the observability of caching strategies, allowing cache behavior to be intuitively self-explanatory through monitoring. - Improve code comprehensibility: unify the cache framework to avoid ad-hoc handling of TTL, increasing the modularity of the code. - Reduce operational complexity by eliminating and standardizing configuration items. Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
Closed
HappenLee pushed a commit
to HappenLee/incubator-doris
that referenced
this pull request
Apr 24, 2026
- Enhance cache performance and reduce cache misses by: - Prevent starvation of any particular cache. - Improve disk space utilization. - Maintain strict and systematic eviction priorities. - Enhance the observability of caching strategies, allowing cache behavior to be intuitively self-explanatory through monitoring. - Improve code comprehensibility: unify the cache framework to avoid ad-hoc handling of TTL, increasing the modularity of the code. - Reduce operational complexity by eliminating and standardizing configuration items. --------- Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.