[improvement](be) Add queue time for delete bitmap tasks by bobhan1 · Pull Request #65248 · apache/doris

bobhan1 · 2026-07-06T07:16:55Z

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: MOW memtable flush calculates delete bitmap asynchronously, but its log did not show how long the async task waited in the executor queue. The load delete-bitmap executor also used the same thread-pool metric name as the normal delete-bitmap executor, making their metrics hard to distinguish. This change records queue time for flush, segment-level, between-segment, and cloud tablet-level async delete-bitmap tasks, and gives the load delete-bitmap executor a distinct thread-pool name.

Release note

None

Check List (For Author)

Test:
- Build: ./build.sh --be -j 100
- Code style: build-support/check-format.sh
Behavior changed: No
Does this need documentation: No

### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: MOW memtable flush calculates delete bitmap asynchronously, but its log did not show how long the async task waited in the executor queue. The load delete-bitmap executor also used the same thread-pool metric name as the normal delete-bitmap executor, making their metrics hard to distinguish. This change records queue time for flush, segment-level, between-segment, and cloud tablet-level async delete-bitmap tasks, and gives the load delete-bitmap executor a distinct thread-pool name. ### Release note None ### Check List (For Author) - Test: - Build: `./build.sh --be -j 100` - Code style: `build-support/check-format.sh` - Behavior changed: No - Does this need documentation: No

hello-stephen · 2026-07-06T07:16:59Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

bobhan1 · 2026-07-06T07:30:41Z

/review

bobhan1 · 2026-07-06T07:30:47Z

run buildall

github-actions

Reviewed PR 65248 against the Doris code-review checkpoints and the BE cloud/storage guides. I did not find a PR-scoped correctness issue that needs an inline comment.

Critical checkpoint conclusions:

Goal/test proof: the patch achieves the stated observability goal by adding queue-time logging to delete-bitmap async work and separating the load delete-bitmap thread-pool name. The author reports ./build.sh --be -j 100 and build-support/check-format.sh; I could not rerun build/tests in this runner because .worktree_initialized, thirdparty/installed, and thirdparty/installed/bin/protoc are absent.
Scope/focus: the change is small and limited to nine BE storage/cloud files. No additional user-provided review focus points were supplied.
Concurrency/lifecycle: async task behavior is unchanged apart from capturing MonotonicMicros() before enqueue and reading the elapsed value inside the worker. Existing token/thread-pool and lock behavior is preserved.
Compatibility/config/session: no storage format, thrift/protocol, config, session variable, or persistence compatibility change was introduced.
Data correctness and parallel paths: cloud and non-cloud engines both pass distinct executor names; token-based segment and between-segment delete-bitmap paths pass measured queue time, while synchronous/default callers keep queue_time_us = 0. Delete-bitmap version, sentinel, and update logic is unchanged.
Observability/performance: the added work is a small monotonic timestamp read per submitted task and extra logging fields/log-threshold checks; I did not find a hot-path correctness or lifecycle issue.
Tests/style: scoped clang-format 16 wrapper check passed for all nine changed files, and scoped git diff --check against the fetched base/head was clean.

Subagent conclusions:

optimizer-rewrite: no optimizer/Nereids or semantic-equivalence candidate found; final convergence round ended with NO_NEW_VALUABLE_FINDINGS.
tests-session-config: no regression-output, session/config, compatibility, or basic CI/style candidate found; final convergence round ended with NO_NEW_VALUABLE_FINDINGS.

hello-stephen · 2026-07-06T08:46:33Z

TPC-H: Total hot run time: 29951 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 897ff9670fc8f64ec6024e84c1614503ce9ea633, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17676	4050	4048	4048
q2	2013	320	203	203
q3	10339	1474	845	845
q4	4676	472	343	343
q5	7526	872	573	573
q6	179	173	135	135
q7	783	849	626	626
q8	9347	1590	1568	1568
q9	5592	4400	4411	4400
q10	6758	1794	1548	1548
q11	533	345	322	322
q12	717	558	429	429
q13	18132	3454	2721	2721
q14	269	257	246	246
q15	q16	802	779	711	711
q17	1072	985	1087	985
q18	7211	5786	5559	5559
q19	1297	1215	1140	1140
q20	776	684	569	569
q21	6079	2702	2677	2677
q22	428	356	303	303
Total cold run time: 102205 ms
Total hot run time: 29951 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4359	4270	4280	4270
q2	287	318	210	210
q3	4681	5005	4426	4426
q4	2084	2158	1379	1379
q5	4429	4318	4311	4311
q6	230	179	130	130
q7	1827	2073	1672	1672
q8	2542	2176	2207	2176
q9	8076	8028	7943	7943
q10	4708	4726	4272	4272
q11	595	409	398	398
q12	778	762	552	552
q13	3306	3644	2965	2965
q14	303	319	277	277
q15	q16	716	754	666	666
q17	1311	1322	1324	1322
q18	8124	7426	7168	7168
q19	1165	1171	1085	1085
q20	2225	2200	1921	1921
q21	5245	4549	4407	4407
q22	519	458	393	393
Total cold run time: 57510 ms
Total hot run time: 51943 ms

hello-stephen · 2026-07-06T08:57:34Z

TPC-DS: Total hot run time: 173746 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 897ff9670fc8f64ec6024e84c1614503ce9ea633, data reload: false
query5	4311	649	490	490
query6	460	233	207	207
query7	4860	609	362	362
query8	333	188	177	177
query9	8766	4009	4026	4009
query10	490	353	300	300
query11	5949	2371	2162	2162
query12	157	105	101	101
query13	1260	591	437	437
query14	6267	5282	4946	4946
query14_1	4298	4292	4308	4292
query15	212	202	180	180
query16	1000	471	448	448
query17	950	714	592	592
query18	2428	472	353	353
query19	212	190	176	176
query20	111	110	106	106
query21	236	156	139	139
query22	13694	13530	13433	13433
query23	17395	16488	16129	16129
query23_1	16192	16270	16229	16229
query24	7541	1771	1288	1288
query24_1	1357	1299	1301	1299
query25	598	466	401	401
query26	1369	369	203	203
query27	2568	625	396	396
query28	4556	2043	2063	2043
query29	1097	646	517	517
query30	343	258	258	258
query31	1119	1094	986	986
query32	113	67	64	64
query33	539	332	266	266
query34	1183	1139	655	655
query35	783	791	677	677
query36	1388	1381	1229	1229
query37	152	108	91	91
query38	1883	1723	1641	1641
query39	962	916	898	898
query39_1	875	874	867	867
query40	249	157	136	136
query41	64	63	67	63
query42	96	98	93	93
query43	320	320	277	277
query44	1432	771	796	771
query45	205	194	175	175
query46	1056	1246	753	753
query47	2346	2324	2208	2208
query48	379	400	298	298
query49	584	425	334	334
query50	1016	430	333	333
query51	4470	4401	4324	4324
query52	85	86	74	74
query53	262	275	203	203
query54	284	237	224	224
query55	74	70	67	67
query56	295	285	263	263
query57	1416	1402	1328	1328
query58	281	251	262	251
query59	1542	1595	1430	1430
query60	337	270	248	248
query61	156	152	149	149
query62	711	652	579	579
query63	253	205	207	205
query64	2525	783	653	653
query65	4861	4771	4787	4771
query66	1837	512	387	387
query67	29453	29492	29327	29327
query68	3166	1585	998	998
query69	442	316	273	273
query70	1059	974	944	944
query71	350	325	304	304
query72	3055	2811	2334	2334
query73	833	806	419	419
query74	5133	4935	4784	4784
query75	2619	2601	2224	2224
query76	2333	1202	784	784
query77	351	372	286	286
query78	12614	12534	11882	11882
query79	1417	1198	781	781
query80	1336	546	467	467
query81	531	323	282	282
query82	586	156	126	126
query83	399	315	294	294
query84	322	161	128	128
query85	957	610	520	520
query86	432	292	281	281
query87	1840	1817	1754	1754
query88	3730	2814	2780	2780
query89	444	402	361	361
query90	1929	203	199	199
query91	204	187	165	165
query92	67	60	57	57
query93	1712	1557	936	936
query94	723	350	321	321
query95	779	605	467	467
query96	1099	845	377	377
query97	2709	2704	2582	2582
query98	219	223	199	199
query99	1184	1150	986	986
Total cold run time: 259557 ms
Total hot run time: 173746 ms

hello-stephen · 2026-07-06T09:02:30Z

ClickBench: Total hot run time: 25.15 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 897ff9670fc8f64ec6024e84c1614503ce9ea633, data reload: false
query1	0.00	0.00	0.00
query2	0.10	0.04	0.04
query3	0.26	0.13	0.12
query4	1.61	0.14	0.13
query5	0.24	0.22	0.22
query6	1.28	1.13	1.20
query7	0.03	0.01	0.00
query8	0.06	0.04	0.04
query9	0.37	0.31	0.31
query10	0.55	0.54	0.54
query11	0.20	0.14	0.14
query12	0.19	0.15	0.14
query13	0.49	0.48	0.48
query14	1.02	1.01	0.98
query15	0.64	0.59	0.59
query16	0.34	0.32	0.31
query17	1.09	1.07	1.06
query18	0.23	0.21	0.21
query19	2.05	1.87	1.88
query20	0.02	0.01	0.02
query21	15.43	0.22	0.13
query22	4.90	0.05	0.05
query23	16.12	0.31	0.12
query24	2.96	0.44	0.32
query25	0.11	0.06	0.04
query26	0.74	0.23	0.16
query27	0.05	0.04	0.04
query28	3.49	0.94	0.53
query29	12.49	4.32	3.44
query30	0.28	0.15	0.15
query31	2.77	0.61	0.31
query32	3.22	0.60	0.49
query33	3.18	3.21	3.25
query34	15.74	4.22	3.50
query35	3.62	3.52	3.52
query36	0.55	0.46	0.43
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.18	0.16	0.16
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 96.94 s
Total hot run time: 25.15 s

hello-stephen · 2026-07-06T11:14:21Z

BE Regression && UT Coverage Report

Increment line coverage 85.42% (82/96) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.74% (29830/39913)
Line Coverage	58.72% (326696/556320)
Region Coverage	55.58% (274369/493661)
Branch Coverage	56.81% (120806/212635)

hello-stephen · 2026-07-06T16:02:44Z

BE Regression && UT Coverage Report

Increment line coverage 85.42% (82/96) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.05% (29158/39913)
Line Coverage	56.65% (315153/556320)
Region Coverage	53.29% (263051/493661)
Branch Coverage	54.21% (115259/212635)

liaoxin01

LGTM

github-actions · 2026-07-07T03:33:52Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-07-07T03:33:54Z

PR approved by anyone and no changes requested.

Problem Summary: MOW memtable flush calculates delete bitmap asynchronously, but its log did not show how long the async task waited in the executor queue. The load delete-bitmap executor also used the same thread-pool metric name as the normal delete-bitmap executor, making their metrics hard to distinguish. This change records queue time for flush, segment-level, between-segment, and cloud tablet-level async delete-bitmap tasks, and gives the load delete-bitmap executor a distinct thread-pool name.

…65523) ### What problem does this PR solve? Issue Number: None Related PR: #65248 Problem Summary: Backport #65248 to branch-4.1. MOW memtable flush calculates delete bitmap asynchronously, but its log did not show how long the task waited in the executor queue. The load delete-bitmap executor also used the same thread-pool metric name as the normal delete-bitmap executor. This change records queue time for flush, segment-level, between-segment, and cloud tablet-level async delete-bitmap tasks, and gives the load delete-bitmap executor a distinct thread-pool name. The branch-4.1 conflict in `beta_rowset_writer.cpp` was resolved while preserving the branch-specific debug point. ### Release note None ### Check List (For Author) - Test - [x] Manual test (add detailed scripts or steps below) - `./build.sh --be -j100` - `build-support/check-format.sh` - Behavior changed: - [x] No. - Does this need documentation? - [x] No. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

…65248 (#65509) Cherry-picked from #65248 Co-authored-by: bobhan1 <baohan@selectdb.com>

bobhan1 marked this pull request as ready for review July 6, 2026 07:30

bobhan1 requested review from gavinchou and luwei16 as code owners July 6, 2026 07:30

github-actionsBot reviewed Jul 6, 2026
View reviewed changes

liaoxin01 approved these changes Jul 7, 2026
View reviewed changes

liaoxin01 added dev/3.1.x dev/4.0.x dev/4.1.x labels Jul 7, 2026

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 7, 2026

github-actionsBot added the reviewed label Jul 7, 2026

gavinchou approved these changes Jul 13, 2026
View reviewed changes

gavinchou merged commit 64e8cf2 into apache:masterJul 13, 2026
38 of 39 checks passed

github-actionsBot mentioned this pull request Jul 13, 2026
branch-4.0: [improvement](be) Add queue time for delete bitmap tasks #65248 #65509
Merged

github-actionsBot added the dev/4.1.x-conflict label Jul 13, 2026

bobhan1 mentioned this pull request Jul 13, 2026
branch-4.1: [improvement](be) Add queue time for delete bitmap tasks #65523
Merged
7 tasks

yiguolei removed dev/4.1.x dev/4.1.x-conflict labels Jul 15, 2026

yiguolei added the dev/4.1.4-merged label Jul 15, 2026

morningman pushed a commit that referenced this pull request Jul 27, 2026
branch-4.0: [improvement](be) Add queue time for delete bitmap tasks#…
a0c45d1
…65248 (#65509) Cherry-picked from #65248 Co-authored-by: bobhan1 <baohan@selectdb.com>

morningman added dev/4.0.8-merged and removed dev/4.0.x labels Jul 27, 2026

morningman mentioned this pull request Aug 11, 2026
4.0.8 Release Notes #66642
Open

bobhan1 added dev/3.1.x int/3.1.9-merged and removed dev/3.1.x labels Aug 19, 2026

Navigation Menu

Uh oh!

[improvement](be) Add queue time for delete bitmap tasks - #65248

[improvement](be) Add queue time for delete bitmap tasks#65248
gavinchou merged 1 commit into
apache:masterfrom
bobhan1:fix/mow-delete-bitmap-queue-metrics

bobhan1 commented Jul 6, 2026

Uh oh!

hello-stephen commented Jul 6, 2026

Uh oh!

bobhan1 commented Jul 6, 2026

Uh oh!

bobhan1 commented Jul 6, 2026

Uh oh!

github-actionsBot left a comment

Uh oh!

hello-stephen commented Jul 6, 2026

Uh oh!

hello-stephen commented Jul 6, 2026

Uh oh!

hello-stephen commented Jul 6, 2026

Uh oh!

hello-stephen commented Jul 6, 2026

Uh oh!

hello-stephen commented Jul 6, 2026

Uh oh!

liaoxin01 left a comment

Uh oh!

github-actionsBot commented Jul 7, 2026

Uh oh!

github-actionsBot commented Jul 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

bobhan1 commented Jul 6, 2026

What problem does this PR solve?

Release note

Check List (For Author)

Uh oh!

hello-stephen commented Jul 6, 2026

Uh oh!

bobhan1 commented Jul 6, 2026

Uh oh!

bobhan1 commented Jul 6, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

hello-stephen commented Jul 6, 2026

Uh oh!

hello-stephen commented Jul 6, 2026

Uh oh!

hello-stephen commented Jul 6, 2026

Uh oh!

hello-stephen commented Jul 6, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Jul 6, 2026

BE Regression && UT Coverage Report

Uh oh!

liaoxin01 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jul 7, 2026

Uh oh!

github-actionsBot commented Jul 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants