[fix](mow) reduce mow delete bitmap count: agg and remove delete bitmap of pre rowsets when cu compaction by mymeiyi · Pull Request #49383 · apache/doris

mymeiyi · 2025-03-24T04:18:35Z

What problem does this PR solve?

This PR reduce delete bitmap count of mow tables

Basic idea

when do cumulative compaction [start_rowset, end_rowset], which version is [start_version, end_version]
agg pre_rowsets'(version less than start_version) delete bitmap between [start_version, end_version] to end_version,
when all of the rowsets of [start_rowset, end_rowset] are deleted, delete pre rowsets' delete bitmap between [start_version, end_version).

One example

suppose the rowset and delete bitmap layout is:

[rowset_id]: [delete bitmap]
r1-3: v6, v7, v8, v9
r4-6: v7, v8, v9
r7: v8, v9 r8: v9
r9:

if do cumulative compaction for {r7, r8, r9}

before this pr, the delete bitmap layout is (there is 7 delete bitmap left):

[rowset_id]: [delete bitmap]
r1-3: v6, v7, v8, v9
r4-6: v7, v8, v9
r7-9:

after this pr (there is 3 delete bitmap left):

[rowset_id]: [delete bitmap]
r1-3: v6, new_v9 // new_v9 is agged by the previous v7, v8, v9 of r1-3
r4-6: new_v9 // new_v9 is agged by the previous v7, v8, v9 of r4-6
r7-9:

The implementation of local mode

delete bitmap in BE memory

agged when delete stale rowset path
removed when delete unused rowsets

delete bitmap in BE local storage

save_meta to store delete bitmap after remove unused rowsets related delete bitmap

The implementation of cloud mode

delete bitmap in BE memory

agged when delete stale rowset path
removed when delete unused rowsets

delete bitmap in Meta Service

agged when finish compaction
send update_delete_bitmap to ms to modify and remove pre rowsets delete bitmap

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-03-24T04:18:41Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

mymeiyi · 2025-03-24T06:10:55Z

run buildall

doris-robot · 2025-03-24T07:03:14Z

TPC-H: Total hot run time: 34480 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6aab3881bd0f55239a11f546c3ac99c91f9834f9, data reload: false
------ Round 1 ----------------------------------
q1	26233	5112	5257	5112
q2	2077	283	170	170
q3	10425	1277	719	719
q4	10247	1038	558	558
q5	7546	2359	2421	2359
q6	189	170	143	143
q7	926	761	610	610
q8	9318	1346	1168	1168
q9	6858	5172	5178	5172
q10	6874	2322	1897	1897
q11	481	273	250	250
q12	354	362	218	218
q13	17774	3667	3089	3089
q14	233	229	211	211
q15	546	488	492	488
q16	654	622	585	585
q17	594	853	348	348
q18	7586	7263	7172	7172
q19	1824	984	582	582
q20	324	331	201	201
q21	4038	2775	2457	2457
q22	1093	1026	971	971
Total cold run time: 116194 ms
Total hot run time: 34480 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5260	5197	5202	5197
q2	253	338	230	230
q3	2171	2669	2269	2269
q4	1458	1846	1490	1490
q5	4490	4477	4390	4390
q6	225	176	129	129
q7	2024	1939	1753	1753
q8	2661	2567	2511	2511
q9	7356	7138	7316	7138
q10	3041	3224	2770	2770
q11	589	518	520	518
q12	677	761	647	647
q13	3558	3974	3330	3330
q14	286	299	291	291
q15	537	483	477	477
q16	655	678	665	665
q17	1200	1527	1461	1461
q18	7895	7609	7449	7449
q19	906	843	906	843
q20	1944	1967	1846	1846
q21	5568	4998	5001	4998
q22	1122	1096	1051	1051
Total cold run time: 53876 ms
Total hot run time: 51453 ms

doris-robot · 2025-03-24T07:15:19Z

TPC-DS: Total hot run time: 194000 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6aab3881bd0f55239a11f546c3ac99c91f9834f9, data reload: false
query1	1403	1066	1071	1066
query2	6382	1949	1962	1949
query3	11184	4816	4569	4569
query4	25702	24285	23015	23015
query5	4768	682	507	507
query6	327	215	193	193
query7	3991	501	268	268
query8	298	257	259	257
query9	8556	2569	2580	2569
query10	480	312	253	253
query11	15535	15224	14843	14843
query12	159	112	102	102
query13	1563	515	371	371
query14	8413	6410	6201	6201
query15	217	186	168	168
query16	7101	657	471	471
query17	1153	745	600	600
query18	1999	437	295	295
query19	192	191	158	158
query20	123	126	118	118
query21	208	134	114	114
query22	4402	4738	4336	4336
query23	34440	33661	33665	33661
query24	8549	2463	2424	2424
query25	524	478	405	405
query26	1177	272	148	148
query27	2905	512	331	331
query28	4803	2484	2461	2461
query29	760	565	452	452
query30	277	224	194	194
query31	931	880	789	789
query32	74	67	67	67
query33	565	387	326	326
query34	803	898	516	516
query35	823	865	781	781
query36	1016	1007	926	926
query37	126	111	82	82
query38	4178	4103	4277	4103
query39	1519	1439	1445	1439
query40	214	123	116	116
query41	60	57	58	57
query42	121	109	108	108
query43	514	529	501	501
query44	1356	826	835	826
query45	187	174	172	172
query46	877	1055	665	665
query47	1830	1932	1818	1818
query48	381	428	307	307
query49	818	540	467	467
query50	699	830	418	418
query51	4347	4319	4333	4319
query52	110	108	98	98
query53	233	260	189	189
query54	496	515	430	430
query55	86	80	81	80
query56	300	271	304	271
query57	1227	1202	1113	1113
query58	249	242	236	236
query59	2782	3042	2957	2957
query60	290	277	262	262
query61	150	129	144	129
query62	784	728	662	662
query63	221	186	186	186
query64	4088	1036	694	694
query65	4564	4479	4617	4479
query66	1044	399	305	305
query67	16724	15884	15701	15701
query68	9686	907	499	499
query69	485	305	272	272
query70	1242	1153	1106	1106
query71	437	291	257	257
query72	5330	4983	4923	4923
query73	696	562	349	349
query74	9274	8933	8910	8910
query75	4353	3236	2690	2690
query76	4947	1204	782	782
query77	1002	387	307	307
query78	9907	10129	9259	9259
query79	4831	823	547	547
query80	595	528	433	433
query81	469	260	220	220
query82	398	127	97	97
query83	172	174	153	153
query84	291	95	73	73
query85	754	358	307	307
query86	331	307	277	277
query87	4430	4598	4321	4321
query88	2874	2244	2234	2234
query89	427	313	279	279
query90	2141	218	210	210
query91	147	154	111	111
query92	76	61	56	56
query93	3185	1029	588	588
query94	681	427	313	313
query95	424	276	256	256
query96	498	567	275	275
query97	3323	3407	3281	3281
query98	235	205	199	199
query99	1386	1430	1322	1322
Total cold run time: 287405 ms
Total hot run time: 194000 ms

doris-robot · 2025-03-24T07:20:38Z

ClickBench: Total hot run time: 31.14 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6aab3881bd0f55239a11f546c3ac99c91f9834f9, data reload: false
query1	0.04	0.04	0.03
query2	0.12	0.11	0.10
query3	0.24	0.19	0.19
query4	1.60	0.18	0.19
query5	0.59	0.59	0.62
query6	1.19	0.72	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.58	0.52	0.53
query10	0.56	0.59	0.58
query11	0.16	0.10	0.10
query12	0.14	0.11	0.11
query13	0.62	0.61	0.60
query14	2.81	2.70	2.81
query15	0.93	0.85	0.85
query16	0.40	0.39	0.40
query17	1.02	1.01	1.06
query18	0.21	0.20	0.19
query19	1.92	1.95	1.86
query20	0.02	0.01	0.01
query21	15.36	0.90	0.54
query22	0.76	1.17	0.64
query23	14.99	1.40	0.60
query24	7.25	1.75	0.48
query25	0.52	0.22	0.10
query26	0.59	0.16	0.14
query27	0.06	0.04	0.04
query28	9.06	0.86	0.43
query29	12.56	4.17	3.48
query30	0.25	0.09	0.06
query31	2.83	0.59	0.39
query32	3.22	0.56	0.47
query33	3.08	3.13	3.12
query34	15.76	5.16	4.55
query35	4.55	4.53	4.60
query36	0.66	0.49	0.48
query37	0.09	0.06	0.07
query38	0.04	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 105.17 s
Total hot run time: 31.14 s

hello-stephen · 2025-03-24T08:38:10Z

BE UT Coverage Report

Increment line coverage 17.68% (29/164) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	49.96% (13372/26765)
Line Coverage	39.40% (115706/293634)
Region Coverage	38.11% (58754/154152)
Branch Coverage	33.26% (29664/89198)

mymeiyi · 2025-03-28T04:05:31Z

run buildall

doris-robot · 2025-03-28T04:29:25Z

TeamCity cloud ut coverage result:
Function Coverage: 83.05% (1088/1310)
Line Coverage: 66.00% (18107/27435)
Region Coverage: 65.35% (8915/13642)
Branch Coverage: 55.31% (4803/8684)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2c5da111ef13571863d005a59877ed469ac3a330_2c5da111ef13571863d005a59877ed469ac3a330_cloud/report/index.html

doris-robot · 2025-03-28T05:24:19Z

TPC-H: Total hot run time: 34206 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2c5da111ef13571863d005a59877ed469ac3a330, data reload: false
------ Round 1 ----------------------------------
q1	25805	5109	5051	5051
q2	2093	286	171	171
q3	10462	1303	706	706
q4	10238	1028	549	549
q5	7546	2321	2419	2321
q6	186	160	131	131
q7	917	764	622	622
q8	9335	1318	1202	1202
q9	6813	5143	5066	5066
q10	6885	2305	1900	1900
q11	481	272	281	272
q12	351	359	221	221
q13	17795	3665	3104	3104
q14	240	237	206	206
q15	543	500	487	487
q16	626	616	585	585
q17	582	935	336	336
q18	7556	7317	7160	7160
q19	2261	972	548	548
q20	313	328	189	189
q21	3857	3361	2400	2400
q22	1050	1025	979	979
Total cold run time: 115935 ms
Total hot run time: 34206 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5239	5136	5069	5069
q2	249	321	232	232
q3	2169	2613	2298	2298
q4	1425	1812	1495	1495
q5	4517	4400	4406	4400
q6	219	164	135	135
q7	2005	1870	1798	1798
q8	2647	2697	2611	2611
q9	7292	7077	7233	7077
q10	2999	3181	2708	2708
q11	567	502	495	495
q12	705	768	588	588
q13	3455	3889	3367	3367
q14	280	300	282	282
q15	538	497	488	488
q16	658	687	684	684
q17	1161	1553	1427	1427
q18	7966	7487	7489	7487
q19	791	788	853	788
q20	1926	1960	1943	1943
q21	5370	4843	4782	4782
q22	1089	1082	1033	1033
Total cold run time: 53267 ms
Total hot run time: 51187 ms

doris-robot · 2025-03-28T05:36:16Z

TPC-DS: Total hot run time: 192722 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2c5da111ef13571863d005a59877ed469ac3a330, data reload: false
query1	1387	1046	1031	1031
query2	6256	2007	2020	2007
query3	11141	4737	4887	4737
query4	25400	23603	23350	23350
query5	4244	647	475	475
query6	303	202	206	202
query7	3983	483	274	274
query8	317	244	257	244
query9	8512	2543	2545	2543
query10	471	322	261	261
query11	15297	15114	14737	14737
query12	170	116	105	105
query13	1578	507	404	404
query14	8905	6199	6297	6199
query15	206	188	166	166
query16	7353	631	486	486
query17	1205	754	603	603
query18	2101	428	320	320
query19	206	196	175	175
query20	132	119	133	119
query21	212	127	109	109
query22	4427	4573	4334	4334
query23	34218	33437	33636	33437
query24	8528	2411	2484	2411
query25	535	476	395	395
query26	1313	261	146	146
query27	2751	503	327	327
query28	4565	2455	2384	2384
query29	703	593	442	442
query30	277	222	192	192
query31	907	865	792	792
query32	74	66	65	65
query33	540	414	295	295
query34	797	876	506	506
query35	825	859	794	794
query36	1014	1026	934	934
query37	117	99	85	85
query38	4109	4109	4189	4109
query39	1469	1429	1444	1429
query40	215	114	105	105
query41	53	54	52	52
query42	129	107	106	106
query43	527	520	498	498
query44	1302	826	811	811
query45	177	177	167	167
query46	844	1021	642	642
query47	1874	1901	1784	1784
query48	381	417	299	299
query49	751	515	425	425
query50	699	751	436	436
query51	4286	4299	4302	4299
query52	106	105	99	99
query53	224	256	180	180
query54	488	505	432	432
query55	84	83	89	83
query56	292	272	274	272
query57	1157	1202	1103	1103
query58	255	238	274	238
query59	2877	2969	2681	2681
query60	293	316	257	257
query61	133	145	132	132
query62	782	753	703	703
query63	214	188	191	188
query64	4081	1042	715	715
query65	4497	4352	4351	4351
query66	966	396	294	294
query67	16076	15682	15479	15479
query68	8737	870	504	504
query69	481	295	249	249
query70	1204	1129	1054	1054
query71	481	333	262	262
query72	5773	4653	4586	4586
query73	688	550	334	334
query74	9435	8826	8678	8678
query75	3882	3197	2714	2714
query76	3734	1168	736	736
query77	787	375	283	283
query78	10043	10131	9396	9396
query79	4169	793	540	540
query80	668	517	433	433
query81	481	261	220	220
query82	551	119	97	97
query83	212	172	151	151
query84	295	99	75	75
query85	788	355	312	312
query86	330	298	276	276
query87	4404	4640	4393	4393
query88	3201	2241	2274	2241
query89	443	306	288	288
query90	2011	207	215	207
query91	140	141	110	110
query92	78	63	55	55
query93	2532	1015	581	581
query94	675	414	309	309
query95	358	287	275	275
query96	499	555	269	269
query97	3201	3250	3102	3102
query98	233	202	202	202
query99	1447	1373	1335	1335
Total cold run time: 282272 ms
Total hot run time: 192722 ms

doris-robot · 2025-03-28T05:41:39Z

ClickBench: Total hot run time: 31.79 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2c5da111ef13571863d005a59877ed469ac3a330, data reload: false
query1	0.04	0.04	0.02
query2	0.15	0.11	0.10
query3	0.34	0.20	0.20
query4	1.59	0.21	0.20
query5	0.67	0.61	0.60
query6	1.19	0.73	0.72
query7	0.03	0.02	0.01
query8	0.06	0.05	0.05
query9	0.63	0.52	0.53
query10	0.57	0.58	0.58
query11	0.26	0.12	0.12
query12	0.24	0.13	0.13
query13	0.64	0.63	0.62
query14	2.80	2.67	2.70
query15	1.00	0.87	0.87
query16	0.36	0.38	0.37
query17	1.08	1.07	1.01
query18	0.19	0.19	0.19
query19	1.97	2.01	1.81
query20	0.01	0.02	0.01
query21	15.36	0.97	0.69
query22	0.92	1.06	0.78
query23	14.71	1.52	0.77
query24	5.56	0.57	0.27
query25	0.17	0.09	0.08
query26	0.55	0.22	0.18
query27	0.09	0.08	0.08
query28	11.02	1.22	0.57
query29	12.53	4.10	3.38
query30	0.28	0.08	0.06
query31	2.81	0.61	0.43
query32	3.23	0.58	0.52
query33	3.05	3.07	3.15
query34	16.51	5.10	4.44
query35	4.49	4.50	4.56
query36	0.64	0.51	0.52
query37	0.18	0.18	0.17
query38	0.16	0.16	0.15
query39	0.05	0.04	0.04
query40	0.21	0.15	0.15
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.05	0.05
Total cold run time: 106.55 s
Total hot run time: 31.79 s

doris-robot · 2025-03-28T07:11:12Z

BE UT Coverage Report

Increment line coverage 11.93% (29/243) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	50.89% (13625/26776)
Line Coverage	40.25% (118299/293909)
Region Coverage	38.92% (60087/154382)
Branch Coverage	33.83% (30223/89330)

mymeiyi · 2025-04-03T03:07:47Z

run buildall

doris-robot · 2025-04-03T03:22:18Z

TeamCity cloud ut coverage result:
Function Coverage: 83.05% (1088/1310)
Line Coverage: 66.20% (18155/27426)
Region Coverage: 65.51% (8923/13621)
Branch Coverage: 55.33% (4807/8688)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9a434ec7921f07164110ab28bc7058c4bccbc029_9a434ec7921f07164110ab28bc7058c4bccbc029_cloud/report/index.html

doris-robot · 2025-04-03T04:13:22Z

TPC-H: Total hot run time: 34384 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9a434ec7921f07164110ab28bc7058c4bccbc029, data reload: false
------ Round 1 ----------------------------------
q1	26030	5071	5021	5021
q2	2072	286	180	180
q3	10578	1253	712	712
q4	10241	1034	552	552
q5	7805	2460	2346	2346
q6	195	160	134	134
q7	935	760	612	612
q8	9343	1317	1088	1088
q9	6793	5142	5156	5142
q10	6809	2326	1897	1897
q11	468	287	269	269
q12	343	351	218	218
q13	17778	3650	3080	3080
q14	235	234	213	213
q15	544	502	508	502
q16	642	611	567	567
q17	604	871	385	385
q18	7692	7247	7241	7241
q19	1541	959	574	574
q20	332	343	228	228
q21	3922	2689	2454	2454
q22	1090	1053	969	969
Total cold run time: 115992 ms
Total hot run time: 34384 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5159	5083	5127	5083
q2	252	340	227	227
q3	2140	2686	2255	2255
q4	1489	1801	1468	1468
q5	4591	4501	4391	4391
q6	210	165	127	127
q7	2011	1939	1770	1770
q8	2612	2447	2555	2447
q9	7155	7172	7119	7119
q10	2981	3199	2725	2725
q11	587	514	475	475
q12	683	787	581	581
q13	3475	3969	3262	3262
q14	295	301	264	264
q15	541	494	493	493
q16	636	692	653	653
q17	1164	1523	1402	1402
q18	7600	7531	7458	7458
q19	798	810	929	810
q20	2009	1980	1836	1836
q21	5202	4781	4620	4620
q22	1091	1044	975	975
Total cold run time: 52681 ms
Total hot run time: 50441 ms

doris-robot · 2025-04-03T04:24:57Z

TPC-DS: Total hot run time: 186397 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9a434ec7921f07164110ab28bc7058c4bccbc029, data reload: false
query1	1004	485	466	466
query2	6557	1988	1976	1976
query3	6754	224	216	216
query4	26417	23241	23437	23241
query5	4373	608	464	464
query6	309	211	199	199
query7	4638	493	286	286
query8	307	254	239	239
query9	8625	2608	2605	2605
query10	453	329	280	280
query11	15405	14939	14940	14939
query12	160	112	115	112
query13	1671	524	402	402
query14	8830	6253	6279	6253
query15	203	189	173	173
query16	7288	650	460	460
query17	1211	732	568	568
query18	1982	416	321	321
query19	197	187	158	158
query20	121	119	121	119
query21	214	128	106	106
query22	4217	4232	4064	4064
query23	33968	32818	32792	32792
query24	8477	2404	2397	2397
query25	560	454	397	397
query26	1230	273	146	146
query27	2753	503	325	325
query28	4357	2442	2425	2425
query29	780	543	427	427
query30	290	222	187	187
query31	925	900	771	771
query32	82	63	65	63
query33	555	357	306	306
query34	797	887	514	514
query35	774	830	736	736
query36	963	995	910	910
query37	117	99	77	77
query38	4224	4272	3968	3968
query39	1491	1408	1394	1394
query40	217	123	109	109
query41	58	55	51	51
query42	124	104	107	104
query43	511	485	492	485
query44	1330	795	799	795
query45	175	170	168	168
query46	841	1025	619	619
query47	1748	1798	1704	1704
query48	391	415	302	302
query49	781	511	451	451
query50	655	687	397	397
query51	4147	4139	4134	4134
query52	110	101	95	95
query53	237	260	185	185
query54	564	567	500	500
query55	86	81	81	81
query56	303	300	286	286
query57	1128	1165	1087	1087
query58	277	293	255	255
query59	2683	2718	2762	2718
query60	323	326	303	303
query61	133	129	130	129
query62	796	731	656	656
query63	227	189	188	188
query64	4430	1028	698	698
query65	4365	4236	4233	4233
query66	1168	403	304	304
query67	15883	15658	15327	15327
query68	8367	874	521	521
query69	474	297	259	259
query70	1138	1122	1113	1113
query71	502	318	301	301
query72	5672	4739	4776	4739
query73	718	623	347	347
query74	8863	9060	8641	8641
query75	3920	3169	2694	2694
query76	3719	1181	756	756
query77	789	373	284	284
query78	9991	10235	9071	9071
query79	1996	866	573	573
query80	590	502	437	437
query81	492	254	226	226
query82	462	129	97	97
query83	256	250	234	234
query84	250	108	87	87
query85	773	355	407	355
query86	390	285	276	276
query87	4411	4433	4521	4433
query88	3530	2244	2270	2244
query89	395	317	284	284
query90	1897	212	214	212
query91	138	144	111	111
query92	75	61	58	58
query93	1537	971	589	589
query94	658	422	287	287
query95	465	295	290	290
query96	491	555	286	286
query97	3175	3203	3198	3198
query98	226	222	202	202
query99	1455	1354	1346	1346
Total cold run time: 274785 ms
Total hot run time: 186397 ms

doris-robot · 2025-04-03T04:30:19Z

ClickBench: Total hot run time: 31.18 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9a434ec7921f07164110ab28bc7058c4bccbc029, data reload: false
query1	0.04	0.03	0.03
query2	0.12	0.10	0.11
query3	0.25	0.20	0.20
query4	1.60	0.19	0.19
query5	0.60	0.60	0.54
query6	1.18	0.71	0.72
query7	0.02	0.01	0.02
query8	0.05	0.03	0.04
query9	0.57	0.51	0.54
query10	0.57	0.57	0.58
query11	0.16	0.11	0.11
query12	0.15	0.11	0.12
query13	0.62	0.60	0.60
query14	2.70	2.82	2.68
query15	0.92	0.86	0.86
query16	0.37	0.39	0.39
query17	1.02	1.00	0.99
query18	0.21	0.19	0.19
query19	1.87	1.92	1.81
query20	0.01	0.01	0.01
query21	15.48	0.93	0.53
query22	0.77	1.21	0.73
query23	14.82	1.39	0.64
query24	7.52	1.57	0.78
query25	0.53	0.21	0.16
query26	0.57	0.15	0.14
query27	0.05	0.05	0.05
query28	9.14	0.82	0.43
query29	12.54	3.93	3.26
query30	0.25	0.08	0.07
query31	2.84	0.59	0.39
query32	3.22	0.54	0.46
query33	3.06	3.06	3.13
query34	15.87	5.12	4.49
query35	4.59	4.56	4.54
query36	0.66	0.50	0.49
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 105.45 s
Total hot run time: 31.18 s

hello-stephen · 2025-04-03T06:10:59Z

BE UT Coverage Report

Increment line coverage 3.23% (10/310) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.13% (13970/26800)
Line Coverage	40.78% (120207/294742)
Region Coverage	39.55% (61284/154970)
Branch Coverage	34.19% (30640/89610)

mymeiyi · 2025-04-03T06:18:15Z

run buildall

doris-robot · 2025-04-03T06:34:53Z

TeamCity cloud ut coverage result:
Function Coverage: 83.05% (1088/1310)
Line Coverage: 66.15% (18140/27424)
Region Coverage: 65.45% (8913/13617)
Branch Coverage: 55.33% (4805/8684)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d5304969dc0d57f308576b025c43467fbcdd63e0_d5304969dc0d57f308576b025c43467fbcdd63e0_cloud/report/index.html

doris-robot · 2025-04-03T07:38:30Z

TPC-H: Total hot run time: 34483 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d5304969dc0d57f308576b025c43467fbcdd63e0, data reload: false
------ Round 1 ----------------------------------
q1	26127	5135	5007	5007
q2	2072	280	181	181
q3	10543	1282	711	711
q4	10234	1042	584	584
q5	7767	2432	2354	2354
q6	183	167	137	137
q7	946	764	630	630
q8	9329	1361	1145	1145
q9	6751	5173	5245	5173
q10	6865	2308	1899	1899
q11	482	283	272	272
q12	349	351	225	225
q13	17796	3734	3068	3068
q14	234	235	210	210
q15	537	484	496	484
q16	644	605	575	575
q17	616	865	367	367
q18	7674	7195	7131	7131
q19	1752	1001	590	590
q20	331	348	227	227
q21	4473	3490	2549	2549
q22	1054	1012	964	964
Total cold run time: 116759 ms
Total hot run time: 34483 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5223	5125	5145	5125
q2	240	328	235	235
q3	2120	2678	2297	2297
q4	1424	1949	1450	1450
q5	4478	4490	4408	4408
q6	218	176	128	128
q7	2024	1916	1739	1739
q8	2620	2623	2580	2580
q9	7237	7080	7165	7080
q10	3012	3177	2734	2734
q11	583	520	485	485
q12	702	780	630	630
q13	3501	3921	3325	3325
q14	291	308	267	267
q15	510	471	470	470
q16	661	689	668	668
q17	1199	1593	1384	1384
q18	7975	7653	7427	7427
q19	866	803	882	803
q20	1919	1961	1881	1881
q21	5426	4931	4777	4777
q22	1121	1067	994	994
Total cold run time: 53350 ms
Total hot run time: 50887 ms

doris-robot · 2025-04-03T07:50:37Z

TPC-DS: Total hot run time: 194753 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d5304969dc0d57f308576b025c43467fbcdd63e0, data reload: false
query1	1389	1095	1029	1029
query2	6367	1943	1966	1943
query3	11057	4561	4651	4561
query4	25708	23740	23246	23246
query5	4259	675	481	481
query6	303	223	214	214
query7	3992	525	293	293
query8	323	258	241	241
query9	8475	2594	2590	2590
query10	476	333	279	279
query11	15693	15290	14895	14895
query12	157	113	111	111
query13	1574	539	386	386
query14	9166	6683	6633	6633
query15	207	196	182	182
query16	7279	670	513	513
query17	1210	802	626	626
query18	2007	449	341	341
query19	208	206	184	184
query20	135	130	127	127
query21	207	133	117	117
query22	4493	4720	4338	4338
query23	34507	33747	33424	33424
query24	8752	2532	2487	2487
query25	521	473	438	438
query26	1190	299	162	162
query27	2726	532	348	348
query28	4786	2477	2428	2428
query29	764	604	455	455
query30	296	241	205	205
query31	981	911	816	816
query32	76	70	68	68
query33	544	384	334	334
query34	847	979	538	538
query35	860	906	783	783
query36	1012	1090	946	946
query37	120	113	86	86
query38	4211	4240	4115	4115
query39	1541	1451	1463	1451
query40	223	125	115	115
query41	67	54	51	51
query42	120	106	108	106
query43	512	524	478	478
query44	1379	830	814	814
query45	190	182	173	173
query46	933	1114	696	696
query47	1896	1902	1803	1803
query48	409	438	315	315
query49	785	555	448	448
query50	740	749	437	437
query51	4437	4368	4274	4274
query52	121	114	106	106
query53	243	278	194	194
query54	606	603	530	530
query55	86	84	83	83
query56	309	345	313	313
query57	1238	1226	1111	1111
query58	274	268	265	265
query59	2821	2923	2781	2781
query60	360	373	319	319
query61	128	126	140	126
query62	806	808	725	725
query63	236	195	197	195
query64	4125	1077	723	723
query65	4596	4401	4395	4395
query66	1117	456	348	348
query67	16163	15583	15527	15527
query68	9546	919	527	527
query69	491	317	278	278
query70	1234	1161	1132	1132
query71	460	323	300	300
query72	5183	4722	4702	4702
query73	689	606	363	363
query74	9145	9143	8734	8734
query75	4448	3401	2773	2773
query76	4388	1314	811	811
query77	985	376	295	295
query78	10064	10310	9291	9291
query79	4826	868	579	579
query80	649	533	458	458
query81	475	276	227	227
query82	427	136	96	96
query83	284	263	244	244
query84	295	112	94	94
query85	774	366	342	342
query86	338	305	280	280
query87	4546	4527	4306	4306
query88	2962	2266	2250	2250
query89	437	336	295	295
query90	2168	221	222	221
query91	144	224	113	113
query92	82	61	55	55
query93	2973	962	588	588
query94	676	435	313	313
query95	383	312	287	287
query96	508	614	281	281
query97	3198	3273	3105	3105
query98	236	226	203	203
query99	1498	1460	1345	1345
Total cold run time: 287851 ms
Total hot run time: 194753 ms

doris-robot · 2025-04-03T07:56:02Z

ClickBench: Total hot run time: 31.74 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d5304969dc0d57f308576b025c43467fbcdd63e0, data reload: false
query1	0.04	0.04	0.02
query2	0.16	0.11	0.10
query3	0.34	0.20	0.20
query4	1.59	0.20	0.20
query5	0.62	0.60	0.62
query6	1.19	0.73	0.73
query7	0.03	0.01	0.02
query8	0.06	0.04	0.05
query9	0.63	0.52	0.52
query10	0.57	0.58	0.58
query11	0.26	0.12	0.12
query12	0.25	0.14	0.13
query13	0.64	0.61	0.61
query14	2.67	2.70	2.68
query15	1.00	0.89	0.88
query16	0.36	0.39	0.37
query17	1.06	1.04	1.02
query18	0.19	0.18	0.19
query19	1.95	2.03	1.81
query20	0.02	0.01	0.01
query21	15.36	0.97	0.67
query22	0.93	1.04	0.82
query23	14.68	1.54	0.77
query24	5.32	0.62	0.29
query25	0.18	0.09	0.08
query26	0.55	0.23	0.18
query27	0.08	0.09	0.08
query28	11.04	1.20	0.56
query29	12.57	4.14	3.39
query30	0.27	0.08	0.07
query31	2.85	0.62	0.44
query32	3.24	0.60	0.50
query33	3.05	3.08	3.12
query34	16.61	5.11	4.40
query35	4.43	4.55	4.46
query36	0.63	0.50	0.49
query37	0.18	0.16	0.17
query38	0.17	0.16	0.16
query39	0.05	0.05	0.04
query40	0.19	0.16	0.16
query41	0.11	0.05	0.05
query42	0.06	0.04	0.04
query43	0.05	0.04	0.05
Total cold run time: 106.23 s
Total hot run time: 31.74 s

doris-robot · 2025-04-03T10:13:50Z

BE UT Coverage Report

Increment line coverage 3.23% (10/310) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.21% (13984/26785)
Line Coverage	40.93% (120477/294345)
Region Coverage	39.65% (61296/154574)
Branch Coverage	34.28% (30608/89294)

zhannngchen · 2025-05-26T07:25:49Z

run buildall

github-actions · 2025-05-26T07:27:46Z

PR approved by at least one committer and no changes requested.

hello-stephen · 2025-05-26T07:43:56Z

Cloud UT Coverage Report

Increment line coverage 83.44% (267/320) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	83.30% (1117/1341)
Line Coverage	66.42% (18997/28600)
Region Coverage	65.99% (9399/14242)
Branch Coverage	55.85% (5080/9096)

doris-robot · 2025-05-26T09:46:27Z

TPC-H: Total hot run time: 33972 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d8abf2c49f7b650636d80d0184390235c717b042, data reload: false
------ Round 1 ----------------------------------
q1	26789	5012	4971	4971
q2	2091	282	199	199
q3	10495	1274	718	718
q4	10233	997	527	527
q5	7703	2399	2321	2321
q6	190	163	133	133
q7	930	723	593	593
q8	9299	1207	1072	1072
q9	6744	5105	5147	5105
q10	6807	2317	1893	1893
q11	518	292	268	268
q12	339	354	229	229
q13	17817	3683	3105	3105
q14	227	230	228	228
q15	528	508	466	466
q16	422	440	379	379
q17	601	856	370	370
q18	7671	7178	7222	7178
q19	1694	976	567	567
q20	331	334	230	230
q21	3943	3110	2430	2430
q22	1018	1046	990	990
Total cold run time: 116390 ms
Total hot run time: 33972 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5090	5069	5051	5051
q2	244	326	238	238
q3	2169	2650	2306	2306
q4	1323	1739	1380	1380
q5	4427	4476	4424	4424
q6	235	170	129	129
q7	2007	1932	1784	1784
q8	2597	2575	2501	2501
q9	7251	7098	7042	7042
q10	3092	3181	2745	2745
q11	591	510	502	502
q12	697	782	621	621
q13	3468	3846	3288	3288
q14	314	323	290	290
q15	515	473	471	471
q16	432	492	454	454
q17	1150	1556	1365	1365
q18	8005	7617	7515	7515
q19	836	845	898	845
q20	1950	2006	1863	1863
q21	4815	4585	4461	4461
q22	1128	1068	1031	1031
Total cold run time: 52336 ms
Total hot run time: 50306 ms

doris-robot · 2025-05-26T09:58:42Z

TPC-DS: Total hot run time: 193320 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d8abf2c49f7b650636d80d0184390235c717b042, data reload: false
query1	1425	1087	1069	1069
query2	6191	1857	1742	1742
query3	11041	4542	4534	4534
query4	54377	24982	22956	22956
query5	5226	548	477	477
query6	342	206	199	199
query7	4886	528	289	289
query8	325	280	260	260
query9	5786	2685	2708	2685
query10	429	335	274	274
query11	15016	15312	14844	14844
query12	161	112	106	106
query13	1045	546	412	412
query14	10217	6399	6290	6290
query15	209	199	185	185
query16	7064	639	475	475
query17	1075	726	595	595
query18	1505	412	307	307
query19	203	184	164	164
query20	131	129	120	120
query21	210	137	108	108
query22	4654	4700	4476	4476
query23	34364	33572	33584	33572
query24	6807	2483	2467	2467
query25	457	463	411	411
query26	675	275	157	157
query27	2315	512	351	351
query28	3119	2182	2192	2182
query29	581	579	436	436
query30	273	224	190	190
query31	878	864	787	787
query32	88	74	62	62
query33	456	376	342	342
query34	788	857	573	573
query35	795	856	730	730
query36	952	998	894	894
query37	119	105	80	80
query38	4253	4315	4346	4315
query39	1559	1446	1490	1446
query40	219	121	107	107
query41	54	53	54	53
query42	133	110	115	110
query43	517	512	475	475
query44	1387	864	854	854
query45	185	180	174	174
query46	866	1043	670	670
query47	1817	1878	1835	1835
query48	407	448	330	330
query49	668	511	448	448
query50	682	706	416	416
query51	4279	4298	4233	4233
query52	112	107	106	106
query53	236	257	184	184
query54	583	620	520	520
query55	91	80	80	80
query56	306	310	302	302
query57	1171	1240	1107	1107
query58	271	260	257	257
query59	2668	2843	2638	2638
query60	344	340	306	306
query61	131	180	122	122
query62	744	696	672	672
query63	232	188	189	188
query64	1417	1015	660	660
query65	4337	4229	4237	4229
query66	752	398	307	307
query67	16022	15722	15539	15539
query68	6571	931	520	520
query69	538	307	277	277
query70	1192	1094	1084	1084
query71	518	320	286	286
query72	5748	4912	5066	4912
query73	1480	726	359	359
query74	8967	8992	9047	8992
query75	3875	3205	2724	2724
query76	4275	1198	771	771
query77	657	373	288	288
query78	10155	10326	9378	9378
query79	2401	833	595	595
query80	604	507	433	433
query81	490	261	219	219
query82	438	131	98	98
query83	334	248	231	231
query84	300	105	86	86
query85	761	346	310	310
query86	374	313	284	284
query87	4525	4444	4405	4405
query88	3610	2275	2277	2275
query89	406	313	280	280
query90	1844	217	211	211
query91	145	141	113	113
query92	74	66	56	56
query93	1970	934	581	581
query94	655	422	319	319
query95	375	301	291	291
query96	504	576	284	284
query97	2723	2840	2701	2701
query98	223	206	205	205
query99	1419	1392	1268	1268
Total cold run time: 298779 ms
Total hot run time: 193320 ms

doris-robot · 2025-05-26T10:03:59Z

ClickBench: Total hot run time: 29.8 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d8abf2c49f7b650636d80d0184390235c717b042, data reload: false
query1	0.04	0.03	0.03
query2	0.16	0.10	0.10
query3	0.37	0.20	0.20
query4	1.59	0.21	0.07
query5	0.42	0.42	0.42
query6	1.16	0.67	0.68
query7	0.03	0.02	0.02
query8	0.06	0.05	0.05
query9	0.62	0.53	0.53
query10	0.59	0.60	0.58
query11	0.26	0.13	0.14
query12	0.26	0.14	0.14
query13	0.64	0.62	0.62
query14	0.81	0.82	0.83
query15	0.97	0.87	0.88
query16	0.37	0.37	0.37
query17	1.05	1.04	1.07
query18	0.19	0.18	0.17
query19	2.02	1.91	1.86
query20	0.02	0.02	0.02
query21	15.44	0.97	0.67
query22	0.93	1.10	0.81
query23	14.73	1.55	0.76
query24	5.35	0.58	0.28
query25	0.17	0.09	0.08
query26	0.54	0.22	0.19
query27	0.09	0.09	0.09
query28	11.05	1.24	0.59
query29	12.62	4.12	3.41
query30	0.28	0.08	0.07
query31	2.83	0.63	0.43
query32	3.24	0.59	0.51
query33	3.03	3.17	3.19
query34	16.80	5.16	4.42
query35	4.51	4.43	4.54
query36	0.62	0.51	0.50
query37	0.20	0.16	0.16
query38	0.18	0.16	0.17
query39	0.06	0.05	0.05
query40	0.19	0.16	0.16
query41	0.10	0.06	0.06
query42	0.06	0.05	0.04
query43	0.06	0.05	0.05
Total cold run time: 104.71 s
Total hot run time: 29.8 s

dataroaring · 2025-05-29T11:39:39Z

run buildall

hello-stephen · 2025-05-29T11:54:59Z

Cloud UT Coverage Report

Increment line coverage 83.44% (267/320) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	83.31% (1118/1342)
Line Coverage	66.80% (19146/28663)
Region Coverage	66.43% (9474/14261)
Branch Coverage	56.33% (5129/9106)

dataroaring

LGTM

doris-robot · 2025-05-29T13:29:34Z

TPC-H: Total hot run time: 33570 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9685355ce5d95e5fb7e492c7ba539df9a1ba2ed5, data reload: false
------ Round 1 ----------------------------------
q1	26078	5686	5052	5052
q2	1963	274	183	183
q3	10439	1251	684	684
q4	10242	993	533	533
q5	7673	2359	2324	2324
q6	178	162	145	145
q7	875	723	614	614
q8	9319	1219	1038	1038
q9	6789	5051	5064	5051
q10	6856	2317	1883	1883
q11	504	287	283	283
q12	348	348	212	212
q13	17808	3702	3031	3031
q14	233	237	212	212
q15	541	507	516	507
q16	420	435	368	368
q17	586	844	347	347
q18	7754	7167	7028	7028
q19	1992	938	571	571
q20	337	330	223	223
q21	3679	2581	2291	2291
q22	1064	1021	990	990
Total cold run time: 115678 ms
Total hot run time: 33570 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5217	5044	5017	5017
q2	236	321	228	228
q3	2176	2663	2279	2279
q4	1390	1809	1335	1335
q5	4547	4414	4359	4359
q6	214	168	127	127
q7	1970	1882	1687	1687
q8	2547	2485	2469	2469
q9	7306	7236	7210	7210
q10	3005	3172	2751	2751
q11	569	493	495	493
q12	658	741	607	607
q13	3492	3937	3317	3317
q14	283	309	274	274
q15	545	497	501	497
q16	458	477	446	446
q17	1146	1507	1406	1406
q18	7762	7518	7361	7361
q19	755	785	886	785
q20	1943	2030	1910	1910
q21	4933	4456	4387	4387
q22	1077	1064	1022	1022
Total cold run time: 52229 ms
Total hot run time: 49967 ms

doris-robot · 2025-05-29T13:41:08Z

TPC-DS: Total hot run time: 186221 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9685355ce5d95e5fb7e492c7ba539df9a1ba2ed5, data reload: false
query1	1022	492	490	490
query2	6585	1793	1815	1793
query3	6750	223	225	223
query4	26820	23939	23286	23286
query5	4329	613	461	461
query6	308	202	184	184
query7	4633	486	291	291
query8	262	213	231	213
query9	8624	2634	2651	2634
query10	449	341	269	269
query11	16079	15248	15021	15021
query12	159	110	109	109
query13	1649	532	411	411
query14	8870	6161	6233	6161
query15	243	198	173	173
query16	6997	636	492	492
query17	1179	677	569	569
query18	1933	376	290	290
query19	192	184	171	171
query20	122	115	114	114
query21	216	134	110	110
query22	4017	3974	4112	3974
query23	34093	33361	33239	33239
query24	8435	2408	2377	2377
query25	547	446	393	393
query26	1244	269	150	150
query27	2749	496	338	338
query28	4334	2132	2124	2124
query29	735	565	427	427
query30	280	214	193	193
query31	942	819	762	762
query32	69	67	66	66
query33	559	357	331	331
query34	801	849	539	539
query35	787	814	710	710
query36	952	953	880	880
query37	116	99	82	82
query38	4192	4117	4145	4117
query39	1479	1399	1424	1399
query40	225	126	112	112
query41	71	66	68	66
query42	130	115	115	115
query43	505	503	462	462
query44	1286	839	826	826
query45	198	181	172	172
query46	846	1021	639	639
query47	1753	1777	1722	1722
query48	393	439	320	320
query49	775	482	431	431
query50	647	673	404	404
query51	4076	4277	4071	4071
query52	105	103	96	96
query53	218	248	191	191
query54	584	587	508	508
query55	86	82	83	82
query56	316	308	287	287
query57	1119	1144	1068	1068
query58	262	256	259	256
query59	2628	2595	2529	2529
query60	318	318	314	314
query61	128	121	141	121
query62	821	718	668	668
query63	233	193	193	193
query64	4443	988	678	678
query65	4272	4185	4164	4164
query66	1141	419	305	305
query67	15816	15753	15306	15306
query68	8002	884	536	536
query69	481	308	281	281
query70	1180	1118	1109	1109
query71	451	324	301	301
query72	5598	4751	4880	4751
query73	704	597	354	354
query74	9212	9156	8975	8975
query75	3803	3205	2684	2684
query76	3632	1172	752	752
query77	795	378	299	299
query78	10072	10115	9290	9290
query79	1968	819	585	585
query80	577	516	445	445
query81	494	254	219	219
query82	434	122	94	94
query83	280	253	241	241
query84	293	103	91	91
query85	771	340	316	316
query86	343	324	277	277
query87	4463	4405	4371	4371
query88	3575	2276	2249	2249
query89	386	320	285	285
query90	1938	211	205	205
query91	145	139	111	111
query92	73	60	58	58
query93	1489	955	586	586
query94	679	409	310	310
query95	380	284	285	284
query96	492	567	317	317
query97	2717	2712	2668	2668
query98	225	205	205	205
query99	1411	1431	1300	1300
Total cold run time: 274210 ms
Total hot run time: 186221 ms

doris-robot · 2025-05-29T13:46:22Z

ClickBench: Total hot run time: 29.34 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9685355ce5d95e5fb7e492c7ba539df9a1ba2ed5, data reload: false
query1	0.04	0.03	0.04
query2	0.11	0.10	0.10
query3	0.25	0.19	0.20
query4	1.59	0.19	0.20
query5	0.45	0.41	0.43
query6	1.14	0.67	0.66
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.58	0.51	0.53
query10	0.56	0.57	0.57
query11	0.15	0.10	0.11
query12	0.14	0.12	0.12
query13	0.61	0.61	0.59
query14	0.78	0.80	0.80
query15	0.86	0.84	0.84
query16	0.38	0.38	0.39
query17	1.05	1.08	1.02
query18	0.23	0.22	0.21
query19	1.93	1.81	1.85
query20	0.02	0.01	0.01
query21	15.42	0.89	0.53
query22	0.78	1.44	0.81
query23	14.69	1.36	0.66
query24	6.98	1.38	0.92
query25	0.47	0.15	0.09
query26	0.65	0.16	0.14
query27	0.06	0.04	0.05
query28	9.94	0.85	0.44
query29	12.54	4.03	3.31
query30	0.26	0.09	0.06
query31	2.83	0.60	0.38
query32	3.23	0.57	0.47
query33	2.97	3.09	3.21
query34	15.74	5.07	4.51
query35	4.56	4.57	4.54
query36	0.67	0.49	0.47
query37	0.08	0.06	0.06
query38	0.06	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.12	0.12
query41	0.07	0.02	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 103.19 s
Total hot run time: 29.34 s

…ap of pre rowsets when cu compaction (apache#49383) ### What problem does this PR solve? This PR reduce delete bitmap count of mow tables #### Basic idea 1. when do cumulative compaction [start_rowset, end_rowset], which version is [start_version, end_version] 2. agg pre_rowsets'(version less than start_version) delete bitmap between [start_version, end_version] to end_version, 3. when all of the rowsets of [start_rowset, end_rowset] are deleted, delete pre rowsets' delete bitmap between [start_version, end_version). #### One example suppose the rowset and delete bitmap layout is: ``` [rowset_id]: [delete bitmap] r1-3: v6, v7, v8, v9 r4-6: v7, v8, v9 r7: v8, v9 r8: v9 r9: ``` if do cumulative compaction for {r7, r8, r9} before this pr, the delete bitmap layout is (there is 7 delete bitmap left): ``` [rowset_id]: [delete bitmap] r1-3: v6, v7, v8, v9 r4-6: v7, v8, v9 r7-9: ``` after this pr (there is 3 delete bitmap left): ``` [rowset_id]: [delete bitmap] r1-3: v6, new_v9 // new_v9 is agged by the previous v7, v8, v9 of r1-3 r4-6: new_v9 // new_v9 is agged by the previous v7, v8, v9 of r4-6 r7-9: ``` #### The implementation of local mode ##### delete bitmap in BE memory 1. agged when delete stale rowset path 2. removed when delete unused rowsets ##### delete bitmap in BE local storage `save_meta` to store delete bitmap after remove unused rowsets related delete bitmap #### The implementation of cloud mode ##### delete bitmap in BE memory 1. agged when delete stale rowset path 2. removed when delete unused rowsets ##### delete bitmap in Meta Service 1. agged when finish compaction 2. send `update_delete_bitmap` to ms to modify and remove pre rowsets delete bitmap ### Release note None

### What problem does this PR solve? now, the delete bitmap is removed when move stale rowsets to unused rowsets, but if some queries is still read the stale rowsets, the query result may get duplicated keys. this pr change the remove of delete bitmap when remove unused rowsets. this pr is based on #49383

…ap of pre rowsets when cu compaction (apache#49383) This PR reduce delete bitmap count of mow tables 1. when do cumulative compaction [start_rowset, end_rowset], which version is [start_version, end_version] 2. agg pre_rowsets'(version less than start_version) delete bitmap between [start_version, end_version] to end_version, 3. when all of the rowsets of [start_rowset, end_rowset] are deleted, delete pre rowsets' delete bitmap between [start_version, end_version). suppose the rowset and delete bitmap layout is: ``` [rowset_id]: [delete bitmap] r1-3: v6, v7, v8, v9 r4-6: v7, v8, v9 r7: v8, v9 r8: v9 r9: ``` if do cumulative compaction for {r7, r8, r9} before this pr, the delete bitmap layout is (there is 7 delete bitmap left): ``` [rowset_id]: [delete bitmap] r1-3: v6, v7, v8, v9 r4-6: v7, v8, v9 r7-9: ``` after this pr (there is 3 delete bitmap left): ``` [rowset_id]: [delete bitmap] r1-3: v6, new_v9 // new_v9 is agged by the previous v7, v8, v9 of r1-3 r4-6: new_v9 // new_v9 is agged by the previous v7, v8, v9 of r4-6 r7-9: ``` 1. agged when delete stale rowset path 2. removed when delete unused rowsets `save_meta` to store delete bitmap after remove unused rowsets related delete bitmap 1. agged when delete stale rowset path 2. removed when delete unused rowsets 1. agged when finish compaction 2. send `update_delete_bitmap` to ms to modify and remove pre rowsets delete bitmap None

… delete bitmap of pre rowsets when cu compaction #49383 (#52204) Cherry-pick from #49383

### What problem does this PR solve? Issue Number: None Related PR: apache#49383 Problem Summary: The MOW version-count promotion protects against excessive delete bitmap metadata, but its default threshold of 1000 can promote tiny cumulative output rowsets and trigger frequent base compactions. Retain the local and cloud promotion logic as an operational fallback, while changing the default threshold to the maximum int64 value so version count alone does not promote rowsets by default. Operators can lower compaction_promotion_version_count if the fallback is needed. ### Release note MOW cumulative compaction no longer promotes output rowsets by version count under the default configuration. The existing configuration remains available as a fallback. ### Check List (For Author) - Test: Unit Test - ./run-be-ut.sh --run --filter=TestSizeBasedCumulativeCompactionPolicy.*:TestCloudSizeBasedCumulativeCompactionPolicy.* -j80 - build-support/check-format.sh - build-support/check-build-hygiene.sh - Behavior changed: Yes. Version-count promotion is disabled by default but remains configurable. - Does this need documentation: No

mymeiyi force-pushed the delete-dm branch from 15317c5 to 6aab388Compare March 24, 2025 06:09

mymeiyi force-pushed the delete-dm branch 2 times, most recently from 2643fe4 to 2c5da11Compare March 28, 2025 04:02

mymeiyi force-pushed the delete-dm branch 2 times, most recently from 6a68198 to 9a434ecCompare April 3, 2025 03:01

mymeiyi force-pushed the delete-dm branch from d530496 to 87b777dCompare April 7, 2025 08:28

zhannngchen approved these changes May 26, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label May 26, 2025

dataroaring added the dev/3.1.x label May 29, 2025

mymeiyi added 2 commits May 29, 2025 19:39

[fix](mow) remove delete bitmap
32ff60a

fix checker
9685355

dataroaring force-pushed the delete-dm branch from d8abf2c to 9685355Compare May 29, 2025 11:39

dataroaring approved these changes May 29, 2025
View reviewed changes

dataroaring merged commit 383f45c into apache:masterMay 30, 2025

mymeiyi mentioned this pull request Jun 24, 2025
branch-3.1: [fix](mow) reduce mow delete bitmap count: agg and remove delete bitmap of pre rowsets when cu compaction (#49383) #52204
Merged

morrySnow pushed a commit that referenced this pull request Jun 24, 2025
branch-3.1: [fix](mow) reduce mow delete bitmap count: agg and remove…
88f3fe1
… delete bitmap of pre rowsets when cu compaction #49383 (#52204) Cherry-pick from #49383

morrySnow added dev/3.1.0-merged and removed dev/3.1.x labels Jun 24, 2025

Yukang-Lian mentioned this pull request Aug 26, 2026
[improvement](compaction) Disable MOW version-count promotion by default #67178
Open
16 tasks

Uh oh!

Conversation

mymeiyi commented Mar 24, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Basic idea

One example

The implementation of local mode

delete bitmap in BE memory

delete bitmap in BE local storage

The implementation of cloud mode

delete bitmap in BE memory

delete bitmap in Meta Service

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Mar 24, 2025

Uh oh!

mymeiyi commented Mar 24, 2025

Uh oh!

doris-robot commented Mar 24, 2025

Uh oh!

doris-robot commented Mar 24, 2025

Uh oh!

doris-robot commented Mar 24, 2025

Uh oh!

hello-stephen commented Mar 24, 2025

BE UT Coverage Report

Uh oh!

mymeiyi commented Mar 28, 2025

Uh oh!

doris-robot commented Mar 28, 2025

Uh oh!

doris-robot commented Mar 28, 2025

Uh oh!

doris-robot commented Mar 28, 2025

Uh oh!

doris-robot commented Mar 28, 2025

Uh oh!

doris-robot commented Mar 28, 2025

BE UT Coverage Report

Uh oh!

mymeiyi commented Apr 3, 2025

Uh oh!

doris-robot commented Apr 3, 2025

Uh oh!

doris-robot commented Apr 3, 2025

Uh oh!

doris-robot commented Apr 3, 2025

Uh oh!

doris-robot commented Apr 3, 2025

Uh oh!

hello-stephen commented Apr 3, 2025

BE UT Coverage Report

Uh oh!

mymeiyi commented Apr 3, 2025

Uh oh!

doris-robot commented Apr 3, 2025

Uh oh!

doris-robot commented Apr 3, 2025

Uh oh!

doris-robot commented Apr 3, 2025

Uh oh!

doris-robot commented Apr 3, 2025

Uh oh!

doris-robot commented Apr 3, 2025

BE UT Coverage Report

Uh oh!

zhannngchen commented May 26, 2025

Uh oh!

github-actionsBot commented May 26, 2025

Uh oh!

hello-stephen commented May 26, 2025

Cloud UT Coverage Report

Uh oh!

doris-robot commented May 26, 2025

Uh oh!

doris-robot commented May 26, 2025

Uh oh!

mymeiyi commented Mar 24, 2025•
edited
Loading