[Opt](cloud-mow) Add more delete bitmap verbose log by bobhan1 · Pull Request #51751 · apache/doris

bobhan1 · 2025-06-16T06:45:17Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-06-16T06:45:25Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

bobhan1 · 2025-06-16T07:50:03Z

run buildall

zhannngchen

LGTM

github-actions · 2025-06-16T08:11:31Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-06-16T08:11:33Z

PR approved by anyone and no changes requested.

doris-robot · 2025-06-16T09:25:06Z

TPC-H: Total hot run time: 34821 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3194ce79737b16708f3b7cd68e95b76ae25e08bd, data reload: false
------ Round 1 ----------------------------------
q1	17670	5216	5019	5019
q2	1936	288	174	174
q3	10408	1271	758	758
q4	10256	1012	534	534
q5	7767	2389	2337	2337
q6	183	170	130	130
q7	893	738	614	614
q8	9306	1293	1111	1111
q9	6817	5153	5189	5153
q10	6850	2320	1913	1913
q11	500	316	300	300
q12	356	356	226	226
q13	17785	3663	3154	3154
q14	227	245	207	207
q15	552	482	502	482
q16	436	443	382	382
q17	653	867	403	403
q18	7532	7191	7108	7108
q19	1675	946	560	560
q20	360	361	243	243
q21	4157	3646	3045	3045
q22	1065	1035	968	968
Total cold run time: 107384 ms
Total hot run time: 34821 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5181	5403	5066	5066
q2	241	319	227	227
q3	2198	2678	2308	2308
q4	1354	1763	1384	1384
q5	4283	4270	4380	4270
q6	223	173	127	127
q7	2012	1926	1752	1752
q8	2609	2695	2583	2583
q9	7162	7207	7265	7207
q10	2930	3145	2837	2837
q11	591	508	495	495
q12	694	795	609	609
q13	3473	3871	3367	3367
q14	285	315	271	271
q15	518	476	477	476
q16	444	504	455	455
q17	1168	1570	1364	1364
q18	7756	7599	7518	7518
q19	837	818	949	818
q20	2024	2014	1926	1926
q21	4963	4502	4507	4502
q22	1145	1073	1062	1062
Total cold run time: 52091 ms
Total hot run time: 50624 ms

doris-robot · 2025-06-16T09:37:27Z

TPC-DS: Total hot run time: 196101 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3194ce79737b16708f3b7cd68e95b76ae25e08bd, data reload: false
query1	1406	1037	1029	1029
query2	6267	1920	1898	1898
query3	11171	4637	4672	4637
query4	54440	25280	23082	23082
query5	5274	536	470	470
query6	349	221	201	201
query7	4901	522	302	302
query8	294	240	231	231
query9	5820	2822	2813	2813
query10	471	341	280	280
query11	15100	15000	15043	15000
query12	164	109	105	105
query13	1050	553	435	435
query14	10215	6367	6479	6367
query15	210	200	172	172
query16	7064	673	512	512
query17	1104	747	598	598
query18	1542	419	334	334
query19	211	210	184	184
query20	138	134	133	133
query21	216	129	116	116
query22	4431	4620	4330	4330
query23	34334	33617	33622	33617
query24	6593	2436	2468	2436
query25	459	503	426	426
query26	722	280	160	160
query27	2456	529	355	355
query28	3097	2326	2318	2318
query29	609	578	435	435
query30	283	224	199	199
query31	886	840	763	763
query32	71	64	60	60
query33	476	373	311	311
query34	821	937	570	570
query35	782	854	735	735
query36	985	1062	933	933
query37	130	114	75	75
query38	4148	4429	4368	4368
query39	1558	1518	1560	1518
query40	228	127	124	124
query41	66	63	65	63
query42	145	135	123	123
query43	589	560	585	560
query44	1490	954	948	948
query45	183	182	176	176
query46	950	1082	727	727
query47	1928	1901	1865	1865
query48	453	480	388	388
query49	704	551	429	429
query50	748	738	444	444
query51	4271	4455	4300	4300
query52	140	130	113	113
query53	265	286	208	208
query54	677	687	586	586
query55	98	96	98	96
query56	376	348	331	331
query57	1266	1351	1208	1208
query58	297	297	291	291
query59	2982	3046	2929	2929
query60	371	352	375	352
query61	155	177	132	132
query62	783	745	682	682
query63	240	203	213	203
query64	1874	1106	712	712
query65	4262	4172	4148	4148
query66	741	411	320	320
query67	16165	15975	15684	15684
query68	6840	942	570	570
query69	563	309	283	283
query70	1268	1160	1146	1146
query71	522	350	329	329
query72	5732	4879	4945	4879
query73	1403	734	388	388
query74	8917	9172	8921	8921
query75	3930	3226	2732	2732
query76	4285	1197	757	757
query77	645	450	299	299
query78	10078	10513	9418	9418
query79	1436	870	625	625
query80	611	548	465	465
query81	488	257	226	226
query82	415	132	109	109
query83	268	270	259	259
query84	295	110	97	97
query85	802	361	325	325
query86	367	299	296	296
query87	4391	4557	4398	4398
query88	3189	2518	2551	2518
query89	428	324	285	285
query90	1948	212	218	212
query91	157	151	124	124
query92	78	64	62	62
query93	1164	992	645	645
query94	696	401	336	336
query95	392	308	321	308
query96	536	587	302	302
query97	2757	2795	2657	2657
query98	239	214	211	211
query99	1447	1407	1269	1269
Total cold run time: 298982 ms
Total hot run time: 196101 ms

doris-robot · 2025-06-16T09:42:46Z

ClickBench: Total hot run time: 29.19 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3194ce79737b16708f3b7cd68e95b76ae25e08bd, data reload: false
query1	0.04	0.04	0.03
query2	0.07	0.04	0.05
query3	0.24	0.06	0.07
query4	1.61	0.11	0.11
query5	0.45	0.42	0.42
query6	1.17	0.66	0.68
query7	0.03	0.02	0.02
query8	0.05	0.04	0.03
query9	0.59	0.54	0.53
query10	0.58	0.59	0.58
query11	0.17	0.11	0.12
query12	0.15	0.12	0.12
query13	0.61	0.60	0.60
query14	0.79	0.82	0.81
query15	0.90	0.86	0.86
query16	0.39	0.39	0.41
query17	1.08	1.09	1.04
query18	0.23	0.21	0.21
query19	1.90	1.90	1.87
query20	0.01	0.01	0.01
query21	15.40	0.98	0.57
query22	0.75	1.16	0.81
query23	14.79	1.41	0.63
query24	7.42	1.29	0.60
query25	0.47	0.26	0.13
query26	0.56	0.17	0.15
query27	0.06	0.06	0.06
query28	8.87	0.93	0.46
query29	12.53	4.09	3.45
query30	0.26	0.10	0.06
query31	2.82	0.60	0.39
query32	3.24	0.56	0.47
query33	3.06	3.06	3.06
query34	15.89	5.24	4.50
query35	4.57	4.57	4.51
query36	0.69	0.51	0.49
query37	0.08	0.07	0.07
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.14	0.14
query41	0.09	0.03	0.03
query42	0.04	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 102.94 s
Total hot run time: 29.19 s

bobhan1 · 2025-06-16T11:46:42Z

run beut

hello-stephen · 2025-06-16T13:51:52Z

BE UT Coverage Report

Increment line coverage 0.00% (0/30) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	56.42% (15068/26709)
Line Coverage	45.17% (134727/298252)
Region Coverage	44.29% (67763/152993)
Branch Coverage	38.86% (34759/89450)

hello-stephen · 2025-06-16T13:55:13Z

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/30) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.81% (20981/26288)
Line Coverage	72.75% (216887/298120)
Region Coverage	71.03% (127840/179984)
Branch Coverage	64.76% (66228/102260)

dataroaring

LGTM

github-actions · 2025-06-17T06:37:19Z

PR approved by at least one committer and no changes requested.

tmp fix add old rowsets dbm log when sync fix Revert "tmp fix" This reverts commit 6fdbb76.

bobhan1 · 2025-06-18T08:44:24Z

run buildall

doris-robot · 2025-06-18T09:19:59Z

TPC-H: Total hot run time: 33532 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2f0082d7e5907738cc0c4968dcc26d3c881fa03c, data reload: false
------ Round 1 ----------------------------------
q1	17583	5084	4975	4975
q2	1952	282	168	168
q3	10297	1248	759	759
q4	10204	1031	514	514
q5	7513	2317	2315	2315
q6	180	159	138	138
q7	919	749	615	615
q8	9328	1268	980	980
q9	6703	4982	5101	4982
q10	6928	2395	1949	1949
q11	480	286	266	266
q12	347	346	214	214
q13	17777	3679	3123	3123
q14	235	222	208	208
q15	544	488	484	484
q16	426	427	384	384
q17	607	853	357	357
q18	7588	7232	7042	7042
q19	1460	971	553	553
q20	346	346	213	213
q21	3699	3122	2313	2313
q22	1048	1024	980	980
Total cold run time: 106164 ms
Total hot run time: 33532 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5073	5052	5038	5038
q2	235	315	218	218
q3	2140	2648	2271	2271
q4	1336	1777	1394	1394
q5	4269	4104	4290	4104
q6	214	170	133	133
q7	2002	1931	1810	1810
q8	2579	2645	2512	2512
q9	7371	7311	7347	7311
q10	3116	3265	2772	2772
q11	613	529	514	514
q12	739	811	648	648
q13	3636	3961	3485	3485
q14	300	316	308	308
q15	522	485	480	480
q16	432	510	456	456
q17	1160	1544	1445	1445
q18	7962	7552	7451	7451
q19	790	861	1031	861
q20	1981	2042	1903	1903
q21	4815	4410	4527	4410
q22	1059	1070	1032	1032
Total cold run time: 52344 ms
Total hot run time: 50556 ms

doris-robot · 2025-06-18T09:32:14Z

TPC-DS: Total hot run time: 193287 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2f0082d7e5907738cc0c4968dcc26d3c881fa03c, data reload: false
query1	1424	1040	1036	1036
query2	6148	1838	1825	1825
query3	11021	4578	4541	4541
query4	52770	25598	23440	23440
query5	5297	547	489	489
query6	361	208	197	197
query7	4953	525	294	294
query8	303	228	217	217
query9	5892	2705	2711	2705
query10	467	334	276	276
query11	15201	15127	14783	14783
query12	166	111	109	109
query13	1075	528	425	425
query14	10184	6444	6477	6444
query15	207	189	197	189
query16	7041	644	509	509
query17	1096	757	595	595
query18	1579	427	328	328
query19	199	205	178	178
query20	127	131	120	120
query21	215	128	107	107
query22	4547	4396	4455	4396
query23	34507	33375	33684	33375
query24	6706	2408	2432	2408
query25	472	481	405	405
query26	768	287	152	152
query27	2422	536	357	357
query28	3074	2208	2199	2199
query29	612	607	441	441
query30	285	232	199	199
query31	881	892	769	769
query32	75	63	70	63
query33	451	358	299	299
query34	822	879	558	558
query35	788	863	774	774
query36	950	987	911	911
query37	118	105	78	78
query38	4231	4311	4312	4311
query39	1526	1447	1448	1447
query40	222	124	114	114
query41	61	56	60	56
query42	127	118	114	114
query43	528	511	491	491
query44	1393	851	833	833
query45	177	176	167	167
query46	872	1045	674	674
query47	1800	1882	1790	1790
query48	404	442	320	320
query49	686	497	400	400
query50	645	702	422	422
query51	4277	4327	4214	4214
query52	110	119	106	106
query53	242	259	186	186
query54	593	581	524	524
query55	88	81	86	81
query56	304	346	292	292
query57	1228	1249	1191	1191
query58	272	271	262	262
query59	2772	2879	2719	2719
query60	342	319	312	312
query61	127	125	127	125
query62	711	787	686	686
query63	231	191	197	191
query64	2332	1080	689	689
query65	4253	4185	4179	4179
query66	770	412	311	311
query67	15871	15701	15381	15381
query68	8212	897	534	534
query69	562	326	269	269
query70	1295	1099	1088	1088
query71	495	317	326	317
query72	5831	4827	4890	4827
query73	1325	685	356	356
query74	8989	9120	8849	8849
query75	3898	3210	2713	2713
query76	4369	1195	756	756
query77	637	370	285	285
query78	10119	10262	9354	9354
query79	2310	820	596	596
query80	623	513	453	453
query81	503	260	234	234
query82	418	129	96	96
query83	327	261	239	239
query84	288	150	92	92
query85	782	358	315	315
query86	388	306	290	290
query87	4542	4488	4298	4298
query88	3131	2295	2262	2262
query89	404	316	298	298
query90	1928	209	211	209
query91	139	146	113	113
query92	78	61	60	60
query93	1159	929	595	595
query94	687	396	302	302
query95	372	296	291	291
query96	490	575	286	286
query97	2783	2734	2652	2652
query98	235	210	205	205
query99	1479	1419	1274	1274
Total cold run time: 299197 ms
Total hot run time: 193287 ms

doris-robot · 2025-06-18T09:37:28Z

ClickBench: Total hot run time: 29.67 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2f0082d7e5907738cc0c4968dcc26d3c881fa03c, data reload: false
query1	0.03	0.03	0.03
query2	0.08	0.03	0.04
query3	0.24	0.06	0.07
query4	1.61	0.10	0.11
query5	0.43	0.40	0.42
query6	1.19	0.66	0.68
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.57	0.51	0.51
query10	0.58	0.57	0.57
query11	0.16	0.11	0.12
query12	0.15	0.12	0.11
query13	0.63	0.61	0.60
query14	0.80	0.82	0.82
query15	0.88	0.87	0.88
query16	0.38	0.38	0.37
query17	1.07	1.08	1.07
query18	0.24	0.21	0.21
query19	2.02	1.83	1.93
query20	0.01	0.01	0.02
query21	15.42	0.91	0.53
query22	0.75	1.23	0.64
query23	14.96	1.40	0.62
query24	6.85	1.31	0.94
query25	0.48	0.29	0.07
query26	0.60	0.17	0.15
query27	0.07	0.06	0.05
query28	9.72	0.90	0.45
query29	12.57	3.99	3.38
query30	0.25	0.09	0.07
query31	2.84	0.62	0.38
query32	3.23	0.56	0.46
query33	3.10	3.09	3.10
query34	16.00	5.42	4.78
query35	4.88	4.85	4.81
query36	0.68	0.51	0.51
query37	0.09	0.07	0.07
query38	0.05	0.04	0.03
query39	0.04	0.02	0.03
query40	0.18	0.16	0.14
query41	0.08	0.03	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.04
Total cold run time: 104.04 s
Total hot run time: 29.67 s

hello-stephen · 2025-06-18T10:41:24Z

BE UT Coverage Report

Increment line coverage 0.00% (0/68) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	56.33% (15048/26713)
Line Coverage	45.09% (134579/298458)
Region Coverage	44.23% (67679/153027)
Branch Coverage	38.81% (34727/89476)

hello-stephen · 2025-06-18T11:17:45Z

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/68) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	61.21% (16092/26290)
Line Coverage	50.76% (151412/298299)
Region Coverage	48.09% (86564/180001)
Branch Coverage	41.61% (42559/102282)

bobhan1 · 2025-06-18T11:41:27Z

run p0

bobhan1 · 2025-06-18T11:41:32Z

run cloud_p0

hello-stephen · 2025-06-18T13:14:23Z

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/68) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	61.21% (16091/26290)
Line Coverage	50.74% (151348/298299)
Region Coverage	48.10% (86585/180001)
Branch Coverage	41.58% (42524/102282)

dataroaring

LGTM

github-actions · 2025-06-18T13:25:37Z

PR approved by at least one committer and no changes requested.

zhannngchen

LGTM

update

#52122) Cherry-pick from #51751

…) (#52121) pick #51751

bobhan1 requested review from dataroaring and gavinchou as code owners June 16, 2025 06:45

zhannngchen previously approved these changes Jun 16, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jun 16, 2025

github-actionsBot added the reviewed label Jun 16, 2025

bobhan1 dismissed zhannngchen’s stale review via 6fdbb76June 17, 2025 06:16

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Jun 17, 2025

dataroaring previously approved these changes Jun 17, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jun 17, 2025

bobhan1 dismissed dataroaring’s stale review via 59b07c7June 17, 2025 09:16

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Jun 17, 2025

add log
2f0082d
tmp fix add old rowsets dbm log when sync fix Revert "tmp fix" This reverts commit 6fdbb76.

bobhan1 force-pushed the more-verbose-dbm-log branch from 4b7e2be to 2f0082dCompare June 18, 2025 08:43

dataroaring approved these changes Jun 18, 2025
View reviewed changes

dataroaring added the dev/3.0.x label Jun 18, 2025

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jun 18, 2025

zhannngchen approved these changes Jun 20, 2025
View reviewed changes

zhannngchen merged commit 177c65d into apache:masterJun 20, 2025

zhannngchen added the dev/3.1.x label Jun 20, 2025

github-actionsBot pushed a commit that referenced this pull request Jun 20, 2025
[Opt](cloud-mow) Add more delete bitmap verbose log (#51751)
a8dca25

github-actionsBot mentioned this pull request Jun 20, 2025
branch-3.1: [Opt](cloud-mow) Add more delete bitmap verbose log #51751 #52082
Closed

github-actionsBot pushed a commit that referenced this pull request Jun 20, 2025
[Opt](cloud-mow) Add more delete bitmap verbose log (#51751)
c7ffd3e

github-actionsBot mentioned this pull request Jun 20, 2025
branch-3.0: [Opt](cloud-mow) Add more delete bitmap verbose log #51751 #52083
Closed

bobhan1 added a commit to bobhan1/doris that referenced this pull request Jun 23, 2025
[Opt](cloud-mow) Add more delete bitmap verbose log (apache#51751)
75f7594
update

bobhan1 added a commit to bobhan1/doris that referenced this pull request Jun 23, 2025
[Opt](cloud-mow) Add more delete bitmap verbose log (apache#51751)
5298dac
update

This was referenced Jun 23, 2025
branch-3.0: [Opt](cloud-mow) Add more delete bitmap verbose log (#51751) #52121
Merged
branch-3.1: [Opt](cloud-mow) Add more delete bitmap verbose log (#51751) #52122
Merged

morrySnow pushed a commit that referenced this pull request Jun 23, 2025
branch-3.1: [Opt](cloud-mow) Add more delete bitmap verbose log#51751(
38ada3e
#52122) Cherry-pick from #51751

morrySnow added dev/3.1.0-merged and removed dev/3.1.x labels Jun 23, 2025

dataroaring pushed a commit that referenced this pull request Jun 23, 2025
branch-3.0: [Opt](cloud-mow) Add more delete bitmap verbose log (#51751…
dc679ae
…) (#52121) pick #51751

dataroaring added dev/3.0.7-merged and removed dev/3.0.x labels Jun 23, 2025

Uh oh!

Conversation

bobhan1 commented Jun 16, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jun 16, 2025

Uh oh!

bobhan1 commented Jun 16, 2025

Uh oh!

zhannngchen left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jun 16, 2025

Uh oh!

github-actionsBot commented Jun 16, 2025

Uh oh!

doris-robot commented Jun 16, 2025

Uh oh!

doris-robot commented Jun 16, 2025

Uh oh!

doris-robot commented Jun 16, 2025

Uh oh!

bobhan1 commented Jun 16, 2025

Uh oh!

hello-stephen commented Jun 16, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 16, 2025

BE Regression && UT Coverage Report

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jun 17, 2025

Uh oh!

bobhan1 commented Jun 18, 2025

Uh oh!

doris-robot commented Jun 18, 2025

Uh oh!

doris-robot commented Jun 18, 2025

Uh oh!

doris-robot commented Jun 18, 2025

Uh oh!

hello-stephen commented Jun 18, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 18, 2025

BE Regression && UT Coverage Report

Uh oh!

bobhan1 commented Jun 18, 2025

Uh oh!

bobhan1 commented Jun 18, 2025

Uh oh!

hello-stephen commented Jun 18, 2025

BE Regression && UT Coverage Report

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jun 18, 2025

Uh oh!

zhannngchen left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants