[fix](cloud-mow) Make delete bitmap cache expired time more reasonable by hust-hhb · Pull Request #47295 · apache/doris

hust-hhb · 2025-01-21T13:51:03Z

after pr #46365, commit mow table may cost 1800s at most (rpc timeout*retry times), so the delete bitmap cache expiration time should be set to 1800s.This cache will be automatically deleted after the loading task is completed. Only when task fails, this cache will be clean in background.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-01-21T13:51:08Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

hust-hhb · 2025-01-21T13:51:33Z

run buildall

doris-robot · 2025-01-21T14:10:35Z

TPC-H: Total hot run time: 32227 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4110b69826ee04878629733470c19baddfda69a9, data reload: false
------ Round 1 ----------------------------------
q1	17743	5491	5390	5390
q2	2060	317	178	178
q3	10749	1278	753	753
q4	10521	991	515	515
q5	9295	2430	2230	2230
q6	200	179	138	138
q7	922	776	611	611
q8	9284	1390	1179	1179
q9	5210	4849	4901	4849
q10	6821	2360	1864	1864
q11	485	281	252	252
q12	337	354	215	215
q13	17805	3678	3081	3081
q14	248	227	212	212
q15	512	448	464	448
q16	619	631	603	603
q17	565	894	336	336
q18	7188	6533	6356	6356
q19	1617	973	547	547
q20	320	321	189	189
q21	2791	2267	1956	1956
q22	374	357	325	325
Total cold run time: 105666 ms
Total hot run time: 32227 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5545	5451	5489	5451
q2	236	347	238	238
q3	2280	2652	2316	2316
q4	1463	1830	1390	1390
q5	4371	4768	4794	4768
q6	171	165	131	131
q7	2148	1974	1853	1853
q8	2665	2886	2720	2720
q9	7432	7286	7329	7286
q10	3011	3268	2704	2704
q11	572	534	493	493
q12	712	770	597	597
q13	3528	3886	3317	3317
q14	301	288	303	288
q15	525	458	466	458
q16	644	687	652	652
q17	1238	1734	1274	1274
q18	7982	7599	7372	7372
q19	795	1102	1089	1089
q20	1996	2053	1882	1882
q21	5763	5250	5261	5250
q22	662	609	552	552
Total cold run time: 54040 ms
Total hot run time: 52081 ms

doris-robot · 2025-01-21T14:23:12Z

TPC-DS: Total hot run time: 193108 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4110b69826ee04878629733470c19baddfda69a9, data reload: false
query1	1330	967	919	919
query2	6358	2056	2010	2010
query3	11001	4663	4341	4341
query4	60878	29098	22978	22978
query5	5614	610	432	432
query6	415	196	176	176
query7	5575	495	296	296
query8	329	248	234	234
query9	8366	2612	2600	2600
query10	474	313	253	253
query11	17916	15039	15484	15039
query12	165	109	118	109
query13	1441	544	426	426
query14	11077	7105	6727	6727
query15	226	205	194	194
query16	7159	632	514	514
query17	1122	737	592	592
query18	1806	383	309	309
query19	195	186	153	153
query20	117	125	107	107
query21	215	125	104	104
query22	4483	4529	4569	4529
query23	34508	33193	33622	33193
query24	5641	2296	2269	2269
query25	514	481	397	397
query26	748	271	158	158
query27	2020	461	340	340
query28	4326	2488	2436	2436
query29	575	571	417	417
query30	217	187	165	165
query31	1020	868	836	836
query32	67	61	55	55
query33	473	379	320	320
query34	732	854	512	512
query35	825	876	748	748
query36	995	1069	970	970
query37	126	97	73	73
query38	4316	4377	4223	4223
query39	1484	1427	1456	1427
query40	201	117	103	103
query41	51	53	51	51
query42	121	104	100	100
query43	511	533	491	491
query44	1284	821	814	814
query45	183	175	173	173
query46	887	1082	669	669
query47	1884	1919	1859	1859
query48	400	413	316	316
query49	734	500	408	408
query50	661	680	398	398
query51	7077	7003	6914	6914
query52	104	100	89	89
query53	225	265	191	191
query54	487	501	427	427
query55	80	82	83	82
query56	268	281	253	253
query57	1193	1195	1189	1189
query58	250	253	238	238
query59	3005	3200	2897	2897
query60	289	265	248	248
query61	120	117	110	110
query62	692	707	658	658
query63	222	186	184	184
query64	2046	1097	654	654
query65	3242	3146	3151	3146
query66	733	386	295	295
query67	16115	15791	15566	15566
query68	5045	818	526	526
query69	498	296	265	265
query70	1241	1117	1157	1117
query71	423	281	257	257
query72	6023	3840	3845	3840
query73	799	744	347	347
query74	9435	9100	8654	8654
query75	3220	3147	2671	2671
query76	3674	1169	773	773
query77	520	362	271	271
query78	10066	10064	9250	9250
query79	3127	817	601	601
query80	1500	531	449	449
query81	546	294	242	242
query82	645	152	125	125
query83	271	164	154	154
query84	285	91	74	74
query85	762	351	319	319
query86	425	282	306	282
query87	4553	4540	4499	4499
query88	4486	2163	2138	2138
query89	400	348	281	281
query90	1547	189	188	188
query91	127	136	106	106
query92	66	56	55	55
query93	2821	874	521	521
query94	874	401	308	308
query95	335	263	245	245
query96	484	610	293	293
query97	2849	2892	2729	2729
query98	222	200	191	191
query99	1274	1360	1233	1233
Total cold run time: 316457 ms
Total hot run time: 193108 ms

doris-robot · 2025-01-21T14:28:32Z

ClickBench: Total hot run time: 30.54 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4110b69826ee04878629733470c19baddfda69a9, data reload: false
query1	0.03	0.03	0.05
query2	0.07	0.03	0.04
query3	0.25	0.07	0.06
query4	1.63	0.10	0.10
query5	0.43	0.43	0.40
query6	1.17	0.65	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.50	0.50
query10	0.56	0.56	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.10
query13	0.61	0.60	0.60
query14	2.74	2.86	2.73
query15	0.90	0.83	0.83
query16	0.38	0.39	0.39
query17	1.03	1.06	1.02
query18	0.23	0.22	0.20
query19	1.96	1.84	2.02
query20	0.01	0.01	0.01
query21	15.44	0.91	0.57
query22	0.76	0.77	0.83
query23	15.20	1.46	0.56
query24	3.02	0.80	1.54
query25	0.27	0.09	0.32
query26	0.26	0.15	0.14
query27	0.05	0.06	0.05
query28	14.00	1.07	0.44
query29	12.63	3.92	3.30
query30	0.25	0.09	0.06
query31	2.84	0.62	0.38
query32	3.22	0.58	0.46
query33	3.03	3.03	3.04
query34	16.34	5.10	4.48
query35	4.53	4.43	4.48
query36	0.65	0.48	0.48
query37	0.10	0.07	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.18	0.13	0.14
query41	0.09	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 105.94 s
Total hot run time: 30.54 s

doris-robot · 2025-01-21T14:44:53Z

TeamCity be ut coverage result:
Function Coverage: 41.56% (10834/26068)
Line Coverage: 31.96% (91563/286518)
Region Coverage: 31.09% (46899/150838)
Branch Coverage: 27.18% (23754/87386)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4110b69826ee04878629733470c19baddfda69a9_4110b69826ee04878629733470c19baddfda69a9/report/index.html

hust-hhb · 2025-01-24T07:13:41Z

run buildall

doris-robot · 2025-01-24T07:50:51Z

TPC-H: Total hot run time: 32175 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4110b69826ee04878629733470c19baddfda69a9, data reload: false
------ Round 1 ----------------------------------
q1	17594	5499	5349	5349
q2	2046	289	161	161
q3	10433	1223	758	758
q4	10221	964	512	512
q5	7558	2384	2131	2131
q6	186	163	135	135
q7	905	784	604	604
q8	9262	1367	1144	1144
q9	5136	4883	4870	4870
q10	6846	2342	1865	1865
q11	470	273	249	249
q12	336	350	221	221
q13	17754	3712	3083	3083
q14	227	231	208	208
q15	507	472	462	462
q16	622	601	591	591
q17	566	864	321	321
q18	6789	6528	6607	6528
q19	3238	933	517	517
q20	309	320	185	185
q21	2998	2226	1970	1970
q22	362	349	311	311
Total cold run time: 104365 ms
Total hot run time: 32175 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5584	5478	5514	5478
q2	234	320	228	228
q3	2254	2617	2281	2281
q4	1464	1826	1405	1405
q5	4335	4795	4664	4664
q6	176	161	133	133
q7	2041	1978	1840	1840
q8	2584	2822	2705	2705
q9	7344	7208	7248	7208
q10	3014	3270	2831	2831
q11	567	499	486	486
q12	652	732	597	597
q13	3558	3872	3302	3302
q14	302	290	269	269
q15	520	469	472	469
q16	644	669	657	657
q17	1227	1758	1273	1273
q18	7587	7359	7330	7330
q19	792	1124	1086	1086
q20	2012	2023	1888	1888
q21	5801	5278	4908	4908
q22	608	603	590	590
Total cold run time: 53300 ms
Total hot run time: 51628 ms

doris-robot · 2025-01-24T08:03:16Z

TPC-DS: Total hot run time: 191675 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4110b69826ee04878629733470c19baddfda69a9, data reload: false
query1	1309	955	940	940
query2	6203	2015	2021	2015
query3	11065	4547	4567	4547
query4	61831	32057	23124	23124
query5	5342	604	449	449
query6	346	209	190	190
query7	4949	494	307	307
query8	318	249	239	239
query9	6092	2609	2589	2589
query10	416	312	267	267
query11	15498	15027	15616	15027
query12	157	110	109	109
query13	1021	555	427	427
query14	11242	6598	7537	6598
query15	201	200	193	193
query16	6809	637	504	504
query17	1089	742	608	608
query18	1492	401	316	316
query19	202	196	165	165
query20	126	114	107	107
query21	216	131	102	102
query22	4466	4690	4683	4683
query23	33938	33309	33724	33309
query24	5653	2336	2305	2305
query25	494	453	409	409
query26	674	274	150	150
query27	1780	486	346	346
query28	3929	2448	2432	2432
query29	531	564	440	440
query30	217	184	160	160
query31	908	866	811	811
query32	74	62	59	59
query33	431	340	308	308
query34	754	861	523	523
query35	799	868	775	775
query36	993	1032	943	943
query37	117	103	73	73
query38	4324	4326	4284	4284
query39	1497	1449	1429	1429
query40	211	112	98	98
query41	51	47	49	47
query42	116	103	101	101
query43	506	521	516	516
query44	1381	850	863	850
query45	190	172	163	163
query46	890	1063	655	655
query47	1852	1905	1852	1852
query48	376	428	330	330
query49	719	509	400	400
query50	651	691	408	408
query51	4391	4311	4204	4204
query52	105	107	100	100
query53	235	260	193	193
query54	500	513	415	415
query55	81	80	82	80
query56	276	267	264	264
query57	1205	1216	1171	1171
query58	247	235	243	235
query59	3098	3325	2963	2963
query60	296	264	250	250
query61	118	113	119	113
query62	725	742	685	685
query63	234	194	187	187
query64	1756	1065	641	641
query65	3250	3170	3125	3125
query66	733	395	289	289
query67	16014	15891	15460	15460
query68	4924	824	548	548
query69	518	315	262	262
query70	1229	1206	1122	1122
query71	426	280	258	258
query72	6137	3836	3913	3836
query73	800	761	366	366
query74	10309	9077	8969	8969
query75	3213	3144	2696	2696
query76	4224	1183	763	763
query77	517	359	274	274
query78	9951	9941	9383	9383
query79	3043	802	613	613
query80	634	525	444	444
query81	497	287	243	243
query82	640	154	125	125
query83	203	165	149	149
query84	289	103	79	79
query85	764	356	295	295
query86	385	313	302	302
query87	4539	4621	4472	4472
query88	4141	2210	2170	2170
query89	407	331	293	293
query90	1612	191	190	190
query91	137	139	109	109
query92	65	56	53	53
query93	1157	857	543	543
query94	749	403	300	300
query95	340	269	253	253
query96	502	607	282	282
query97	2788	2893	2748	2748
query98	225	215	195	195
query99	1461	1343	1283	1283
Total cold run time: 304536 ms
Total hot run time: 191675 ms

doris-robot · 2025-01-24T08:08:36Z

ClickBench: Total hot run time: 31 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4110b69826ee04878629733470c19baddfda69a9, data reload: false
query1	0.04	0.04	0.03
query2	0.09	0.04	0.03
query3	0.24	0.07	0.07
query4	1.62	0.10	0.10
query5	0.42	0.41	0.41
query6	1.14	0.66	0.64
query7	0.02	0.02	0.01
query8	0.04	0.02	0.03
query9	0.59	0.51	0.52
query10	0.55	0.55	0.55
query11	0.14	0.11	0.10
query12	0.14	0.11	0.11
query13	0.60	0.59	0.59
query14	2.71	2.74	2.76
query15	0.88	0.84	0.82
query16	0.38	0.38	0.40
query17	1.08	1.09	1.07
query18	0.22	0.20	0.20
query19	1.99	1.89	1.98
query20	0.01	0.01	0.02
query21	15.35	0.95	0.57
query22	0.75	0.80	0.84
query23	15.11	1.43	0.52
query24	3.18	0.98	1.96
query25	0.17	0.11	0.16
query26	0.22	0.15	0.14
query27	0.09	0.04	0.04
query28	14.12	0.98	0.42
query29	12.54	3.99	3.32
query30	0.25	0.09	0.07
query31	2.83	0.62	0.39
query32	3.23	0.54	0.46
query33	2.92	3.02	3.01
query34	16.37	5.24	4.57
query35	4.62	4.56	4.55
query36	0.63	0.49	0.48
query37	0.09	0.05	0.06
query38	0.04	0.04	0.03
query39	0.03	0.03	0.02
query40	0.16	0.14	0.13
query41	0.07	0.02	0.02
query42	0.04	0.02	0.03
query43	0.03	0.03	0.02
Total cold run time: 105.74 s
Total hot run time: 31 s

doris-robot · 2025-01-24T08:14:40Z

TeamCity be ut coverage result:
Function Coverage: 41.97% (10946/26080)
Line Coverage: 32.24% (92445/286729)
Region Coverage: 31.36% (47354/150990)
Branch Coverage: 27.43% (23996/87470)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4110b69826ee04878629733470c19baddfda69a9_4110b69826ee04878629733470c19baddfda69a9/report/index.html

zhannngchen

LGTM

github-actions · 2025-01-24T11:23:12Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-01-24T11:23:14Z

PR approved by anyone and no changes requested.

dataroaring

LGTM

dataroaring

LGTM

dataroaring

LGTM

#47295) after pr #46365, commit mow table may cost 1800s at most (rpc timeout*retry times), so the delete bitmap cache expiration time should be set to 1800s.This cache will be automatically deleted after the loading task is completed. Only when task fails, this cache will be clean in background.

…re reasonable #47295 (#47490) Cherry-picked from #47295 Co-authored-by: huanghaibin <huanghaibin@selectdb.com>

apache#47295) after pr apache#46365, commit mow table may cost 1800s at most (rpc timeout*retry times), so the delete bitmap cache expiration time should be set to 1800s.This cache will be automatically deleted after the loading task is completed. Only when task fails, this cache will be clean in background.

[fix](cloud-mow) Make delete bitmap cache expired time more reasonable
4110b69

hust-hhb requested review from dataroaring and gavinchou as code owners January 21, 2025 13:51

zhannngchen approved these changes Jan 24, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jan 24, 2025

github-actionsBot added the reviewed label Jan 24, 2025

zhannngchen added dev/3.0.x p0_b labels Jan 24, 2025

dataroaring approved these changes Jan 27, 2025
View reviewed changes

zhannngchen merged commit 29055f2 into apache:masterJan 29, 2025

github-actionsBot mentioned this pull request Jan 29, 2025
branch-3.0: [fix](cloud-mow) Make delete bitmap cache expired time more reasonable #47295 #47490
Merged

dataroaring pushed a commit that referenced this pull request Jan 30, 2025
branch-3.0: [fix](cloud-mow) Make delete bitmap cache expired time mo…
a2827a2
…re reasonable #47295 (#47490) Cherry-picked from #47295 Co-authored-by: huanghaibin <huanghaibin@selectdb.com>

dataroaring added dev/3.0.4-merged and removed dev/3.0.x labels Jan 30, 2025

gavinchou mentioned this pull request Feb 18, 2025
Release Notes 3.0.4 #48013
Closed

Uh oh!

Conversation

hust-hhb commented Jan 21, 2025

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jan 21, 2025

Uh oh!

hust-hhb commented Jan 21, 2025

Uh oh!

doris-robot commented Jan 21, 2025

Uh oh!

doris-robot commented Jan 21, 2025

Uh oh!

doris-robot commented Jan 21, 2025

Uh oh!

doris-robot commented Jan 21, 2025

Uh oh!

hust-hhb commented Jan 24, 2025

Uh oh!

doris-robot commented Jan 24, 2025

Uh oh!

doris-robot commented Jan 24, 2025

Uh oh!

doris-robot commented Jan 24, 2025

Uh oh!

doris-robot commented Jan 24, 2025

Uh oh!

zhannngchen left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jan 24, 2025

Uh oh!

github-actionsBot commented Jan 24, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants