[improve](cloud-mow)Make some mow calculating delete bitmap config more reasonable by hust-hhb · Pull Request #46750 · apache/doris

hust-hhb · 2025-01-10T04:22:17Z

Edit config to let calculate delete bitmap more fast.

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

…re reasonable

hello-stephen · 2025-01-10T04:22:23Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

hust-hhb · 2025-01-10T04:23:04Z

run buildall

doris-robot · 2025-01-10T05:03:23Z

TPC-H: Total hot run time: 32826 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cbbc531c2ea856550aa8d05bb613c136ef7ffb36, data reload: false
------ Round 1 ----------------------------------
q1	17639	6150	6021	6021
q2	2055	300	176	176
q3	10561	1209	756	756
q4	10246	876	444	444
q5	8643	2190	1980	1980
q6	219	175	145	145
q7	890	759	603	603
q8	9268	1372	1181	1181
q9	5221	4988	4973	4973
q10	6772	2290	1844	1844
q11	465	281	265	265
q12	345	368	228	228
q13	17764	3723	3074	3074
q14	232	227	219	219
q15	583	513	513	513
q16	639	601	602	601
q17	595	849	340	340
q18	6851	6425	6514	6425
q19	2283	958	554	554
q20	311	312	193	193
q21	2843	2195	1994	1994
q22	368	328	297	297
Total cold run time: 104793 ms
Total hot run time: 32826 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6332	6211	6292	6211
q2	238	327	238	238
q3	2257	2638	2302	2302
q4	1446	1814	1391	1391
q5	4377	4700	4848	4700
q6	186	190	139	139
q7	2068	1942	1799	1799
q8	2611	2811	2716	2716
q9	7315	7283	7251	7251
q10	3091	3303	2706	2706
q11	575	519	514	514
q12	717	803	638	638
q13	3519	3868	3315	3315
q14	288	304	274	274
q15	574	510	523	510
q16	659	706	645	645
q17	1208	1711	1309	1309
q18	7772	7599	7284	7284
q19	809	1121	1089	1089
q20	2021	2065	1880	1880
q21	5751	5035	4998	4998
q22	614	596	580	580
Total cold run time: 54428 ms
Total hot run time: 52489 ms

doris-robot · 2025-01-10T05:04:21Z

TeamCity be ut coverage result:
Function Coverage: 39.34% (10252/26058)
Line Coverage: 30.52% (87404/286398)
Region Coverage: 29.57% (44550/150678)
Branch Coverage: 26.11% (22802/87322)
Coverage Report: http://coverage.selectdb-in.cc/coverage/cbbc531c2ea856550aa8d05bb613c136ef7ffb36_cbbc531c2ea856550aa8d05bb613c136ef7ffb36/report/index.html

doris-robot · 2025-01-10T05:15:37Z

TPC-DS: Total hot run time: 195187 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cbbc531c2ea856550aa8d05bb613c136ef7ffb36, data reload: false
query1	1295	962	957	957
query2	6442	2284	2251	2251
query3	11109	4877	4953	4877
query4	32922	23566	23312	23312
query5	4232	597	433	433
query6	285	194	175	175
query7	4003	488	305	305
query8	304	257	245	245
query9	9252	2743	2768	2743
query10	465	323	250	250
query11	17784	15165	15128	15128
query12	166	108	103	103
query13	1572	521	397	397
query14	9845	7098	7425	7098
query15	267	207	192	192
query16	8053	604	482	482
query17	1554	772	619	619
query18	2184	414	357	357
query19	221	184	152	152
query20	122	126	120	120
query21	215	123	104	104
query22	4527	4384	4540	4384
query23	33797	33576	33139	33139
query24	6597	2274	2348	2274
query25	491	471	414	414
query26	787	295	161	161
query27	2087	472	333	333
query28	5706	2476	2494	2476
query29	662	571	421	421
query30	218	184	181	181
query31	939	894	813	813
query32	100	59	55	55
query33	482	356	314	314
query34	770	853	496	496
query35	798	820	744	744
query36	1017	1010	961	961
query37	130	102	80	80
query38	4192	4121	4006	4006
query39	1523	1493	1460	1460
query40	207	121	104	104
query41	55	50	52	50
query42	117	101	97	97
query43	532	546	490	490
query44	1357	834	829	829
query45	184	175	164	164
query46	877	1050	654	654
query47	1895	1918	1849	1849
query48	391	425	319	319
query49	728	515	390	390
query50	661	675	392	392
query51	7163	7037	6998	6998
query52	100	102	98	98
query53	225	255	186	186
query54	493	521	420	420
query55	90	79	83	79
query56	261	278	265	265
query57	1206	1197	1205	1197
query58	256	242	227	227
query59	3200	3285	3139	3139
query60	288	268	265	265
query61	118	116	128	116
query62	913	809	744	744
query63	228	188	187	187
query64	3609	1001	700	700
query65	3264	3238	3218	3218
query66	863	418	362	362
query67	16285	15853	15405	15405
query68	9300	715	515	515
query69	475	291	246	246
query70	1216	1159	1103	1103
query71	436	284	246	246
query72	6462	3818	3927	3818
query73	663	749	379	379
query74	10155	9055	8806	8806
query75	4095	3130	2655	2655
query76	3615	1160	755	755
query77	771	377	269	269
query78	9933	9904	9362	9362
query79	3535	836	590	590
query80	739	507	431	431
query81	485	276	242	242
query82	621	151	119	119
query83	196	170	153	153
query84	282	95	79	79
query85	754	353	305	305
query86	353	324	288	288
query87	4403	4435	4303	4303
query88	4650	2214	2168	2168
query89	408	326	284	284
query90	1804	190	193	190
query91	136	139	112	112
query92	64	57	55	55
query93	1973	867	530	530
query94	644	403	284	284
query95	330	259	253	253
query96	499	607	282	282
query97	2870	2941	2816	2816
query98	219	193	192	192
query99	1557	1534	1401	1401
Total cold run time: 296324 ms
Total hot run time: 195187 ms

doris-robot · 2025-01-10T05:20:56Z

ClickBench: Total hot run time: 31.45 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cbbc531c2ea856550aa8d05bb613c136ef7ffb36, data reload: false
query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.60	0.10	0.10
query5	0.42	0.41	0.40
query6	1.17	0.65	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.58	0.51	0.50
query10	0.55	0.57	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.61	0.60
query14	2.86	2.75	2.74
query15	0.90	0.84	0.82
query16	0.38	0.37	0.38
query17	1.09	0.99	1.05
query18	0.23	0.21	0.20
query19	1.94	1.88	1.97
query20	0.01	0.01	0.02
query21	15.36	0.97	0.59
query22	0.75	0.85	0.66
query23	15.20	1.45	0.53
query24	2.66	1.68	1.04
query25	0.17	0.19	0.15
query26	0.26	0.14	0.14
query27	0.05	0.06	0.05
query28	13.67	1.54	1.04
query29	12.58	3.89	3.23
query30	0.26	0.09	0.06
query31	2.82	0.57	0.38
query32	3.22	0.55	0.46
query33	3.12	3.22	3.12
query34	16.61	5.17	4.50
query35	4.57	4.52	4.51
query36	0.63	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.13	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.43 s
Total hot run time: 31.45 s

hust-hhb · 2025-01-13T08:05:27Z

run buildall

doris-robot · 2025-01-13T08:42:02Z

TPC-H: Total hot run time: 32378 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ff5389bea5ae0bd42b1cd8eb041755850d9dfdc7, data reload: false
------ Round 1 ----------------------------------
q1	17580	6164	6068	6068
q2	2053	298	174	174
q3	10425	1211	733	733
q4	10213	851	431	431
q5	7606	2175	1974	1974
q6	211	181	149	149
q7	874	757	597	597
q8	9241	1353	1172	1172
q9	5250	4871	4831	4831
q10	6791	2298	1853	1853
q11	478	282	268	268
q12	342	354	225	225
q13	17774	3627	2985	2985
q14	236	240	215	215
q15	581	516	510	510
q16	659	619	569	569
q17	577	842	324	324
q18	6837	6399	6338	6338
q19	2024	951	556	556
q20	312	319	185	185
q21	2938	2118	1914	1914
q22	363	342	307	307
Total cold run time: 103365 ms
Total hot run time: 32378 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6264	6222	6190	6190
q2	239	327	237	237
q3	2221	2633	2360	2360
q4	1442	1789	1370	1370
q5	4354	4761	4785	4761
q6	182	177	147	147
q7	2081	1950	1773	1773
q8	2636	2818	2740	2740
q9	7285	7148	7304	7148
q10	3008	3333	2718	2718
q11	566	508	497	497
q12	687	773	633	633
q13	3468	3818	3214	3214
q14	284	294	291	291
q15	569	515	503	503
q16	662	681	652	652
q17	1257	1735	1257	1257
q18	7638	7494	7405	7405
q19	828	1107	1077	1077
q20	1992	2017	1886	1886
q21	5776	5314	4900	4900
q22	651	612	551	551
Total cold run time: 54090 ms
Total hot run time: 52310 ms

doris-robot · 2025-01-13T08:52:19Z

TeamCity be ut coverage result:
Function Coverage: 40.35% (10514/26055)
Line Coverage: 31.10% (89086/286440)
Region Coverage: 30.22% (45554/150753)
Branch Coverage: 26.51% (23152/87342)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ff5389bea5ae0bd42b1cd8eb041755850d9dfdc7_ff5389bea5ae0bd42b1cd8eb041755850d9dfdc7/report/index.html

doris-robot · 2025-01-13T08:54:16Z

TPC-DS: Total hot run time: 194798 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ff5389bea5ae0bd42b1cd8eb041755850d9dfdc7, data reload: false
query1	1328	973	911	911
query2	6352	2392	2279	2279
query3	10936	4620	4584	4584
query4	32974	23547	23443	23443
query5	3585	627	461	461
query6	264	189	174	174
query7	3977	484	301	301
query8	309	231	216	216
query9	9489	2661	2654	2654
query10	474	312	251	251
query11	18339	15166	15044	15044
query12	164	106	104	104
query13	1558	519	395	395
query14	9918	8013	7552	7552
query15	275	220	200	200
query16	8563	649	454	454
query17	1615	767	579	579
query18	2108	431	348	348
query19	238	194	175	175
query20	118	123	128	123
query21	212	162	104	104
query22	4508	4641	4295	4295
query23	34024	33792	33186	33186
query24	6270	2276	2271	2271
query25	501	456	399	399
query26	763	273	157	157
query27	2060	467	327	327
query28	5244	2459	2431	2431
query29	641	559	421	421
query30	215	187	167	167
query31	932	871	795	795
query32	83	61	57	57
query33	507	355	297	297
query34	752	848	512	512
query35	828	827	784	784
query36	1053	1033	964	964
query37	128	101	75	75
query38	4216	4209	4200	4200
query39	1486	1442	1437	1437
query40	203	118	99	99
query41	49	53	53	53
query42	123	108	101	101
query43	496	534	495	495
query44	1314	827	811	811
query45	183	174	174	174
query46	883	1059	648	648
query47	1905	1882	1844	1844
query48	378	402	322	322
query49	735	491	402	402
query50	652	649	385	385
query51	7162	7106	7133	7106
query52	107	103	90	90
query53	230	255	183	183
query54	475	486	411	411
query55	83	80	77	77
query56	257	272	252	252
query57	1208	1207	1132	1132
query58	243	231	233	231
query59	3232	3263	3020	3020
query60	281	272	249	249
query61	114	110	114	110
query62	814	814	764	764
query63	229	191	199	191
query64	3243	1027	665	665
query65	3289	3315	3251	3251
query66	810	405	301	301
query67	16375	15728	15362	15362
query68	8747	688	510	510
query69	490	282	264	264
query70	1195	1132	1141	1132
query71	438	308	252	252
query72	6560	3668	3892	3668
query73	659	745	356	356
query74	10640	9186	8779	8779
query75	4460	3128	2685	2685
query76	3795	1172	763	763
query77	786	358	275	275
query78	10100	9850	9262	9262
query79	3647	784	571	571
query80	789	520	436	436
query81	486	272	241	241
query82	639	156	124	124
query83	196	190	151	151
query84	284	92	78	78
query85	761	332	305	305
query86	355	320	315	315
query87	4589	4483	4249	4249
query88	4291	2144	2167	2144
query89	420	318	296	296
query90	1889	183	187	183
query91	134	130	105	105
query92	70	54	53	53
query93	1640	858	528	528
query94	652	388	292	292
query95	331	267	261	261
query96	486	604	290	290
query97	2826	2919	2809	2809
query98	242	194	200	194
query99	1661	1521	1371	1371
Total cold run time: 296338 ms
Total hot run time: 194798 ms

doris-robot · 2025-01-13T08:59:38Z

ClickBench: Total hot run time: 31.04 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ff5389bea5ae0bd42b1cd8eb041755850d9dfdc7, data reload: false
query1	0.04	0.04	0.05
query2	0.08	0.04	0.03
query3	0.24	0.08	0.06
query4	1.62	0.10	0.10
query5	0.42	0.43	0.40
query6	1.14	0.69	0.67
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.60	0.49	0.52
query10	0.56	0.56	0.54
query11	0.15	0.11	0.10
query12	0.15	0.11	0.12
query13	0.63	0.62	0.60
query14	2.87	2.89	2.78
query15	0.94	0.86	0.86
query16	0.39	0.38	0.38
query17	1.02	1.03	1.05
query18	0.24	0.22	0.21
query19	2.03	1.87	2.01
query20	0.02	0.01	0.01
query21	15.35	0.94	0.59
query22	0.76	0.75	0.68
query23	15.26	1.47	0.64
query24	3.35	2.08	0.36
query25	0.29	0.22	0.05
query26	0.29	0.15	0.13
query27	0.07	0.06	0.05
query28	13.45	1.62	1.07
query29	12.58	3.97	3.27
query30	0.25	0.09	0.06
query31	2.80	0.59	0.39
query32	3.23	0.59	0.47
query33	3.22	3.13	3.11
query34	16.85	5.37	4.55
query35	4.61	4.49	4.61
query36	0.66	0.49	0.50
query37	0.10	0.06	0.06
query38	0.05	0.05	0.03
query39	0.03	0.03	0.02
query40	0.17	0.15	0.14
query41	0.08	0.02	0.02
query42	0.04	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 106.72 s
Total hot run time: 31.04 s

zhannngchen

LGTM

github-actions · 2025-01-13T13:53:07Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-01-13T13:53:10Z

PR approved by anyone and no changes requested.

…re reasonable (#46750)

…ap config more reasonable #46750 (#46986) Cherry-picked from #46750 Co-authored-by: huanghaibin <huanghaibin@selectdb.com>

…re reasonable (apache#46750)

[improve](cloud-mow)Make some mow calculating delete bitmap config mo…
2c36472
…re reasonable

edit
cbbc531

zhannngchen reviewed Jan 13, 2025
View reviewed changes

Comment threadbe/src/common/config.cpp Outdated

zhannngchen reviewed Jan 13, 2025
View reviewed changes

Comment threadbe/src/common/config.cpp Outdated

edit
ff5389b

zhannngchen approved these changes Jan 13, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jan 13, 2025

github-actionsBot added the reviewed label Jan 13, 2025

zhannngchen added the dev/3.0.x label Jan 14, 2025

gavinchou approved these changes Jan 14, 2025
View reviewed changes

gavinchou merged commit ef18747 into apache:masterJan 14, 2025

github-actionsBot pushed a commit that referenced this pull request Jan 14, 2025
[improve](cloud-mow)Make some mow calculating delete bitmap config mo…
59cb5e4
…re reasonable (#46750)

github-actionsBot mentioned this pull request Jan 14, 2025
branch-3.0: [improve](cloud-mow)Make some mow calculating delete bitmap config more reasonable #46750 #46986
Merged

dataroaring pushed a commit that referenced this pull request Jan 15, 2025
branch-3.0: [improve](cloud-mow)Make some mow calculating delete bitm…
d3f23c7
…ap config more reasonable #46750 (#46986) Cherry-picked from #46750 Co-authored-by: huanghaibin <huanghaibin@selectdb.com>

dataroaring added dev/3.0.4-merged and removed dev/3.0.x labels Jan 15, 2025

gavinchou mentioned this pull request Feb 18, 2025
Release Notes 3.0.4 #48013
Closed

lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
[improve](cloud-mow)Make some mow calculating delete bitmap config mo…
e3587ce
…re reasonable (apache#46750)

Uh oh!

Conversation

hust-hhb commented Jan 10, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jan 10, 2025

Uh oh!

hust-hhb commented Jan 10, 2025

Uh oh!

doris-robot commented Jan 10, 2025

Uh oh!

doris-robot commented Jan 10, 2025

Uh oh!

doris-robot commented Jan 10, 2025

Uh oh!

doris-robot commented Jan 10, 2025

Uh oh!

Uh oh!

Uh oh!

hust-hhb commented Jan 13, 2025

Uh oh!

doris-robot commented Jan 13, 2025

Uh oh!

doris-robot commented Jan 13, 2025

Uh oh!

doris-robot commented Jan 13, 2025

Uh oh!

doris-robot commented Jan 13, 2025

Uh oh!

zhannngchen left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jan 13, 2025

Uh oh!

github-actionsBot commented Jan 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants