[enhancement](cloud) optimize block cache lock by freemandealer · Pull Request #41818 · apache/doris

freemandealer · 2024-10-14T13:57:27Z

async deletion when do stale rowsets reclycle
minimize lock critical size
add cache lock held & wait time info for debug

1. async deletion when do stale rowsets reclycle 2. minimize lock critical size 3. add cache lock held & wait time info for debug Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>

doris-robot · 2024-10-14T13:57:32Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions

clang-tidy made some suggestions

github-actions · 2024-10-14T14:02:27Z

@@ -19,6 +19,7 @@

 #include <bvar/bvar.h>


warning: 'bvar/bvar.h' file not found [clang-diagnostic-error]
#include<bvar/bvar.h> ^

github-actions · 2024-10-14T14:02:27Z


+class LockScopedTimer {
+public:
+ LockScopedTimer() : start_(std::chrono::high_resolution_clock::now()) {}


warning: use '= default' to define a trivial default constructor [modernize-use-equals-default]
Suggested change
LockScopedTimer() : start_(std::chrono::high_resolution_clock::now()) {}
LockScopedTimer() : start_(std::chrono::high_resolution_clock::now()) =default;

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>

freemandealer · 2024-10-14T14:12:23Z

run buildall

freemandealer · 2024-10-15T07:36:12Z

run buildall

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>

freemandealer · 2024-10-15T08:09:57Z

run buildall

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>

github-actions

clang-tidy made some suggestions

github-actions · 2024-10-15T09:10:57Z


+class LockScopedTimer {
+public:
+ LockScopedTimer() : start_(std::chrono::steady_clock::now()) {}


warning: use '= default' to define a trivial default constructor [modernize-use-equals-default]
Suggested change
LockScopedTimer() : start_(std::chrono::steady_clock::now()) {}
LockScopedTimer() : start_(std::chrono::steady_clock::now()) =default;

freemandealer · 2024-10-15T09:11:10Z

run buildall

doris-robot · 2024-10-15T09:42:00Z

TeamCity be ut coverage result:
Function Coverage: 37.45% (9709/25922)
Line Coverage: 28.73% (80607/280562)
Region Coverage: 28.19% (41725/148027)
Branch Coverage: 24.76% (21212/85660)
Coverage Report: http://coverage.selectdb-in.cc/coverage/35aac9ef8aa51fa6981f8d4d2c6c9bd1c82b396e_35aac9ef8aa51fa6981f8d4d2c6c9bd1c82b396e/report/index.html

dataroaring · 2024-10-17T02:52:44Z

+ std::chrono::duration_cast<std::chrono::milliseconds>(acq_time - start_time).count(); \
+ if (duration > config::cache_lock_long_tail_threshold) \
+ LOG(WARNING) << "Lock wait time " << std::to_string(duration) << "ms. " \
+ << get_stack_trace_by_boost() << std::endl; \


We should limit frequency of the log.

@dataroaring It is limited already: only > config::cache_lock_long_tail_threshold (default:1s) will print the log. If not enough, enlarge the threshold.

freemandealer · 2024-10-17T13:04:41Z

run cloud_p0

freemandealer · 2024-10-18T02:35:06Z

run buildall

doris-robot · 2024-10-18T04:21:21Z

TeamCity be ut coverage result:
Function Coverage: 37.46% (9712/25925)
Line Coverage: 28.72% (80628/280694)
Region Coverage: 28.18% (41753/148155)
Branch Coverage: 24.74% (21211/85726)
Coverage Report: http://coverage.selectdb-in.cc/coverage/00ae5bd2f3bd4c4cfb87018fc0a23d5a733af054_00ae5bd2f3bd4c4cfb87018fc0a23d5a733af054/report/index.html

gavinchou · 2024-10-24T06:31:27Z

+ while (_recycle_keys->pop(key)) {
+ Status st = _storage->remove(key);
+ if (!st.ok()) {
+ LOG_WARNING("").error(st);


put some key word in the log e.g.
"failed to remove when recycle_stale_rowset_async_bottom_half, key = {}" ...

gavinchou · 2024-10-24T06:31:46Z

+ LOG_WARNING("Failed to push recycle key to queue, do it synchronously");
+ Status st = _storage->remove(key);
+ if (!st.ok()) {
+ LOG_WARNING("").error(st);


put some key word in the log e.g.
"failed to remove async, key = {}" ...

github-actions · 2024-10-30T09:17:35Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-10-30T09:17:37Z

PR approved by anyone and no changes requested.

Hastyshell

LGTM

1. async deletion when do stale rowsets reclycle 2. minimize lock critical size 3. add cache lock held & wait time info for debug

pick apache#41818 from master 1. async deletion when do stale rowsets reclycle 2. minimize lock critical size 3. add cache lock held & wait time info for debug

pick #41818 from master 1. async deletion when do stale rowsets reclycle 2. minimize lock critical size 3. add cache lock held & wait time info for debug

…#43401) pick apache#41818 from master 1. async deletion when do stale rowsets reclycle 2. minimize lock critical size 3. add cache lock held & wait time info for debug

1. async deletion when do stale rowsets reclycle 2. minimize lock critical size 3. add cache lock held & wait time info for debug

[enhancement](cloud) optimize block cache lock
6abefcd
1. async deletion when do stale rowsets reclycle 2. minimize lock critical size 3. add cache lock held & wait time info for debug Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>

github-actionsBot reviewed Oct 14, 2024
View reviewed changes

minor fix
b02dd70
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>

freemandealer force-pushed the opt-cache-lock branch from 9e44e1e to 1981d93Compare October 15, 2024 07:49

minor fix
384b29d
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>

freemandealer force-pushed the opt-cache-lock branch from 1981d93 to 384b29dCompare October 15, 2024 07:56

response to the reviewer
35aac9e
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>

github-actionsBot reviewed Oct 15, 2024
View reviewed changes

dataroaring reviewed Oct 17, 2024
View reviewed changes

Merge branch 'master' into opt-cache-lock
00ae5bd

dataroaring added the dev/3.0.x label Oct 24, 2024

gavinchou reviewed Oct 24, 2024
View reviewed changes

Comment threadbe/src/io/cache/block_file_cache.h

gavinchou reviewed Oct 24, 2024
View reviewed changes

Comment threadbe/src/common/config.h

gavinchou reviewed Oct 24, 2024
View reviewed changes

gavinchou approved these changes Oct 30, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Oct 30, 2024

github-actionsBot added the reviewed label Oct 30, 2024

Hastyshell approved these changes Oct 30, 2024
View reviewed changes

gavinchou merged commit cf1714c into apache:masterOct 30, 2024

github-actionsBot pushed a commit that referenced this pull request Oct 30, 2024
[enhancement](cloud) optimize block cache lock (#41818)
b84673b
1. async deletion when do stale rowsets reclycle 2. minimize lock critical size 3. add cache lock held & wait time info for debug

github-actionsBot mentioned this pull request Oct 30, 2024
branch-3.0: [enhancement](cloud) optimize block cache lock #42920
Closed

freemandealer mentioned this pull request Nov 7, 2024
[enhancement](cloud) optimize block cache lock (#41818) #43401
Merged
16 tasks

dataroaring added dev/3.0.3-merged and removed dev/3.0.x labels Nov 7, 2024

gavinchou mentioned this pull request Nov 26, 2024
Release Note 3.0.3 #44522
Closed

	LockScopedTimer() : start_(std::chrono::high_resolution_clock::now()) {}
	LockScopedTimer() : start_(std::chrono::high_resolution_clock::now()) =default;

Uh oh!

Conversation

freemandealer commented Oct 14, 2024• edited by gavinchou Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doris-robot commented Oct 14, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBotOct 14, 2024

Choose a reason for hiding this comment

Uh oh!

github-actionsBotOct 14, 2024

Choose a reason for hiding this comment

Uh oh!

freemandealer commented Oct 14, 2024

Uh oh!

freemandealer commented Oct 15, 2024

Uh oh!

freemandealer commented Oct 15, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBotOct 15, 2024

Choose a reason for hiding this comment

Uh oh!

freemandealer commented Oct 15, 2024

Uh oh!

doris-robot commented Oct 15, 2024

Uh oh!

dataroaringOct 17, 2024

Choose a reason for hiding this comment

Uh oh!

freemandealerOct 17, 2024

Choose a reason for hiding this comment

Uh oh!

freemandealer commented Oct 17, 2024

Uh oh!

freemandealer commented Oct 18, 2024

Uh oh!

doris-robot commented Oct 18, 2024

Uh oh!

Uh oh!

Uh oh!

gavinchouOct 24, 2024

Choose a reason for hiding this comment

Uh oh!

gavinchouOct 24, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Oct 30, 2024

Uh oh!

github-actionsBot commented Oct 30, 2024

Uh oh!

Hastyshell left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

freemandealer commented Oct 14, 2024•
edited by gavinchou
Loading

gavinchouOct 24, 2024•
edited
Loading