[Opt](cloud-mow) Mow load can force take MS lock by bobhan1 · Pull Request #53086 · apache/doris

bobhan1 · 2025-07-11T02:52:31Z

What problem does this PR solve?

This PR:

changes the mow load's backoff sleep time when get MS lock.
Let mow load force take MS lock after the waiting time exceeds a configurable threshold to reduce the load latency tail.

Test 1: cloud mode, 1 FE, 3 BE(8C 32GB), 5 threads, 120 buckets, 50rows/per load, cumu threads=8

with conf mow_load_force_take_ms_lock_threshold_ms=500

before	after

Test 2: cloud mode, 1 FE, 3 BE(8C 32GB), 1 threads, 120 buckets, 50rows/per load, cumu threads=8

with conf mow_load_force_take_ms_lock_threshold_ms=50

before	after

Note that the txn latency long tail is due to put object during flush phase.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-07-11T02:52:37Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

bobhan1 · 2025-07-16T03:58:25Z

run buildall

doris-robot · 2025-07-16T04:11:10Z

Cloud UT Coverage Report

Increment line coverage 11.69% (9/77) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.70% (1280/1606)
Line Coverage	65.47% (21592/32982)
Region Coverage	66.84% (10859/16247)
Branch Coverage	56.41% (5714/10130)

doris-robot · 2025-07-16T05:24:10Z

TPC-H: Total hot run time: 33943 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4bb4bf2cfeceb6ee37cb5cca11b7141d2f8577a8, data reload: false
------ Round 1 ----------------------------------
q1	17647	5274	5115	5115
q2	1935	285	192	192
q3	10401	1291	732	732
q4	10242	1024	534	534
q5	7695	2348	2355	2348
q6	180	162	134	134
q7	898	760	611	611
q8	9322	1334	1176	1176
q9	6824	5089	5152	5089
q10	6887	2365	1961	1961
q11	489	290	267	267
q12	335	359	221	221
q13	17774	3690	3055	3055
q14	225	245	211	211
q15	544	480	478	478
q16	424	434	368	368
q17	577	887	367	367
q18	7858	7319	7040	7040
q19	1214	933	558	558
q20	350	335	223	223
q21	3855	3137	2327	2327
q22	1060	1018	936	936
Total cold run time: 106736 ms
Total hot run time: 33943 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5128	5133	5151	5133
q2	243	326	218	218
q3	2157	2662	2279	2279
q4	1338	1804	1348	1348
q5	4218	4566	4712	4566
q6	207	167	132	132
q7	2077	1952	1825	1825
q8	2683	2618	2643	2618
q9	7408	7272	7412	7272
q10	3047	3284	2797	2797
q11	582	524	487	487
q12	688	778	652	652
q13	3584	4030	3511	3511
q14	289	317	286	286
q15	524	497	458	458
q16	443	489	459	459
q17	1186	1600	1361	1361
q18	7954	7786	7563	7563
q19	789	798	826	798
q20	2115	2074	1907	1907
q21	5095	4752	4459	4459
q22	1142	1093	1051	1051
Total cold run time: 52897 ms
Total hot run time: 51180 ms

doris-robot · 2025-07-16T05:35:52Z

TPC-DS: Total hot run time: 188765 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4bb4bf2cfeceb6ee37cb5cca11b7141d2f8577a8, data reload: false
query1	995	388	424	388
query2	6514	1667	1615	1615
query3	6734	211	217	211
query4	26848	23762	23496	23496
query5	4377	591	432	432
query6	300	220	211	211
query7	4626	501	286	286
query8	258	218	224	218
query9	8596	2642	2647	2642
query10	477	339	262	262
query11	15559	15070	14713	14713
query12	156	106	102	102
query13	1640	520	417	417
query14	8616	5947	5885	5885
query15	206	192	175	175
query16	7157	618	496	496
query17	1111	704	567	567
query18	1964	394	301	301
query19	187	185	195	185
query20	122	117	111	111
query21	207	124	109	109
query22	4301	4417	4559	4417
query23	34789	34232	33999	33999
query24	8377	2372	2355	2355
query25	547	438	415	415
query26	1241	263	151	151
query27	2767	521	344	344
query28	4312	2172	2120	2120
query29	757	542	437	437
query30	283	215	182	182
query31	923	843	776	776
query32	69	65	60	60
query33	554	363	310	310
query34	793	841	524	524
query35	768	821	748	748
query36	970	984	892	892
query37	112	99	78	78
query38	4124	4156	4060	4060
query39	1495	1441	1410	1410
query40	208	121	104	104
query41	55	54	51	51
query42	119	107	105	105
query43	492	511	482	482
query44	1305	848	817	817
query45	171	167	157	157
query46	837	1019	622	622
query47	1768	1810	1745	1745
query48	364	407	301	301
query49	756	492	383	383
query50	629	689	406	406
query51	5535	5522	5497	5497
query52	111	113	93	93
query53	215	255	180	180
query54	567	583	492	492
query55	85	75	77	75
query56	330	289	283	283
query57	1189	1190	1157	1157
query58	263	267	255	255
query59	2546	2666	2533	2533
query60	322	324	286	286
query61	127	119	129	119
query62	792	717	678	678
query63	224	183	184	183
query64	4320	1048	758	758
query65	4301	4222	4193	4193
query66	1141	424	329	329
query67	16203	15745	15659	15659
query68	6756	887	542	542
query69	493	324	274	274
query70	1206	1118	1147	1118
query71	415	319	293	293
query72	5976	4955	4929	4929
query73	707	685	351	351
query74	9101	8954	8859	8859
query75	3162	3197	2683	2683
query76	3177	1155	755	755
query77	474	386	295	295
query78	10158	10217	9390	9390
query79	2661	799	571	571
query80	653	491	484	484
query81	507	258	228	228
query82	491	121	95	95
query83	246	261	231	231
query84	267	108	83	83
query85	802	355	319	319
query86	378	300	285	285
query87	4538	4505	4389	4389
query88	4094	2309	2288	2288
query89	380	317	279	279
query90	1885	203	200	200
query91	138	141	111	111
query92	78	60	56	56
query93	2419	941	586	586
query94	732	386	313	313
query95	379	294	287	287
query96	522	570	280	280
query97	2687	2782	2637	2637
query98	235	213	205	205
query99	1414	1425	1306	1306
Total cold run time: 275938 ms
Total hot run time: 188765 ms

doris-robot · 2025-07-16T05:41:23Z

ClickBench: Total hot run time: 32.41 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4bb4bf2cfeceb6ee37cb5cca11b7141d2f8577a8, data reload: false
query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.24	0.07	0.08
query4	1.62	0.11	0.11
query5	0.45	0.46	0.43
query6	1.16	0.68	0.67
query7	0.03	0.02	0.02
query8	0.04	0.03	0.04
query9	0.61	0.53	0.51
query10	0.58	0.59	0.57
query11	0.17	0.11	0.12
query12	0.15	0.12	0.12
query13	0.62	0.62	0.61
query14	0.80	0.85	0.81
query15	0.89	0.88	0.89
query16	0.40	0.40	0.40
query17	1.10	1.11	1.10
query18	0.22	0.21	0.22
query19	1.93	1.84	1.91
query20	0.02	0.01	0.02
query21	15.38	0.89	0.54
query22	0.76	1.15	0.59
query23	15.07	1.39	0.62
query24	7.11	1.64	0.42
query25	0.43	0.24	0.16
query26	0.75	0.17	0.15
query27	0.06	0.07	0.05
query28	9.52	0.88	0.44
query29	12.57	4.02	3.38
query30	3.07	3.01	2.95
query31	2.82	0.60	0.38
query32	3.24	0.56	0.48
query33	3.11	3.17	3.16
query34	16.00	5.41	4.84
query35	4.82	4.84	4.84
query36	0.71	0.51	0.49
query37	0.09	0.07	0.06
query38	0.06	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.15	0.14
query41	0.08	0.02	0.03
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 107.11 s
Total hot run time: 32.41 s

doris-robot · 2025-07-16T06:45:16Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.52% (15738/27360)
Line Coverage	46.17% (140398/304087)
Region Coverage	35.46% (104680/295186)
Branch Coverage	38.07% (46137/121204)

bobhan1 · 2025-07-16T10:36:03Z

run buildall

hello-stephen · 2025-07-16T11:05:40Z

Cloud UT Coverage Report

Increment line coverage 11.69% (9/77) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	80.31% (1297/1615)
Line Coverage	65.65% (21707/33066)
Region Coverage	67.00% (10917/16293)
Branch Coverage	56.57% (5743/10152)

hello-stephen · 2025-07-16T11:53:27Z

FE UT Coverage Report

Increment line coverage 0.00% (0/23) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2025-07-16T11:56:21Z

TPC-H: Total hot run time: 34239 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a7faacafd00a6eae4615d799bf60d7579f44126b, data reload: false
------ Round 1 ----------------------------------
q1	17614	5264	5175	5175
q2	1948	293	177	177
q3	10399	1319	712	712
q4	10259	1020	514	514
q5	7811	2418	2398	2398
q6	184	160	134	134
q7	944	739	603	603
q8	9332	1345	1145	1145
q9	6864	5078	5209	5078
q10	6893	2387	2001	2001
q11	495	284	285	284
q12	338	350	214	214
q13	17789	3744	3089	3089
q14	226	237	217	217
q15	548	491	477	477
q16	428	451	378	378
q17	592	888	358	358
q18	7571	7263	7190	7190
q19	1409	975	567	567
q20	342	334	228	228
q21	4079	3180	2344	2344
q22	1017	1001	956	956
Total cold run time: 107082 ms
Total hot run time: 34239 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5210	5145	5156	5145
q2	246	327	221	221
q3	2259	2718	2281	2281
q4	1339	1807	1331	1331
q5	4390	4428	4594	4428
q6	221	172	123	123
q7	2059	1894	1866	1866
q8	2637	2662	2597	2597
q9	7384	7360	7251	7251
q10	3115	3293	2879	2879
q11	579	526	513	513
q12	866	760	633	633
q13	3617	4102	3317	3317
q14	295	305	277	277
q15	520	484	492	484
q16	457	493	440	440
q17	1191	1567	1435	1435
q18	7992	7971	7575	7575
q19	810	806	835	806
q20	1923	2041	1796	1796
q21	4757	4289	4357	4289
q22	1039	1043	982	982
Total cold run time: 52906 ms
Total hot run time: 50669 ms

doris-robot · 2025-07-16T12:07:54Z

TPC-DS: Total hot run time: 187357 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a7faacafd00a6eae4615d799bf60d7579f44126b, data reload: false
query1	991	389	419	389
query2	6555	1722	1694	1694
query3	6734	208	212	208
query4	26882	23900	23214	23214
query5	4327	575	423	423
query6	304	226	216	216
query7	4626	493	282	282
query8	269	230	229	229
query9	8671	2623	2634	2623
query10	490	324	269	269
query11	15185	15100	14841	14841
query12	157	108	104	104
query13	1660	514	404	404
query14	8633	5881	5821	5821
query15	195	195	159	159
query16	7170	623	442	442
query17	1192	697	570	570
query18	1980	397	294	294
query19	193	179	178	178
query20	117	117	110	110
query21	214	120	106	106
query22	4070	4209	4134	4134
query23	33931	33004	33066	33004
query24	8468	2440	2389	2389
query25	567	459	387	387
query26	1243	260	146	146
query27	2773	508	335	335
query28	4320	2166	2134	2134
query29	749	581	430	430
query30	281	214	192	192
query31	915	835	764	764
query32	70	61	61	61
query33	546	372	315	315
query34	781	852	510	510
query35	805	802	723	723
query36	959	980	910	910
query37	106	92	72	72
query38	4160	4214	4156	4156
query39	1542	1474	1430	1430
query40	211	117	101	101
query41	56	57	57	57
query42	120	102	105	102
query43	517	504	488	488
query44	1310	810	811	810
query45	174	169	162	162
query46	837	1021	630	630
query47	1786	1825	1767	1767
query48	381	414	310	310
query49	737	451	380	380
query50	639	703	414	414
query51	5466	5490	5388	5388
query52	121	110	103	103
query53	223	245	188	188
query54	563	576	484	484
query55	92	79	83	79
query56	301	311	279	279
query57	1244	1246	1152	1152
query58	262	253	249	249
query59	2660	2703	2631	2631
query60	343	308	288	288
query61	123	122	124	122
query62	802	718	661	661
query63	222	189	185	185
query64	4351	1023	734	734
query65	4266	4174	4176	4174
query66	1130	402	309	309
query67	16033	15660	15550	15550
query68	4604	905	536	536
query69	484	322	256	256
query70	1211	1138	1181	1138
query71	398	312	305	305
query72	5739	4862	4950	4862
query73	687	674	353	353
query74	8851	9163	9086	9086
query75	3205	3239	2728	2728
query76	3160	1149	720	720
query77	468	373	288	288
query78	9917	10130	9465	9465
query79	1993	787	577	577
query80	675	526	455	455
query81	501	265	218	218
query82	179	127	97	97
query83	244	261	230	230
query84	253	105	83	83
query85	757	362	317	317
query86	362	312	291	291
query87	4437	4479	4387	4387
query88	2861	2281	2274	2274
query89	391	315	275	275
query90	1792	200	204	200
query91	169	137	110	110
query92	68	60	57	57
query93	1163	929	593	593
query94	638	385	297	297
query95	383	309	300	300
query96	496	565	286	286
query97	2680	2756	2613	2613
query98	246	214	200	200
query99	1343	1417	1295	1295
Total cold run time: 268156 ms
Total hot run time: 187357 ms

doris-robot · 2025-07-16T12:13:30Z

ClickBench: Total hot run time: 33.13 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a7faacafd00a6eae4615d799bf60d7579f44126b, data reload: false
query1	0.04	0.04	0.03
query2	0.11	0.05	0.05
query3	0.29	0.06	0.06
query4	1.61	0.08	0.08
query5	0.43	0.42	0.41
query6	1.17	0.66	0.66
query7	0.03	0.02	0.03
query8	0.07	0.05	0.04
query9	0.64	0.52	0.52
query10	0.59	0.58	0.58
query11	0.26	0.12	0.13
query12	0.25	0.14	0.13
query13	0.66	0.63	0.64
query14	0.81	0.82	0.84
query15	0.97	0.91	0.88
query16	0.37	0.37	0.38
query17	1.09	1.10	1.05
query18	0.25	0.23	0.24
query19	1.97	1.85	1.92
query20	0.01	0.01	0.01
query21	15.34	0.96	0.66
query22	0.92	0.98	0.78
query23	14.73	1.54	0.74
query24	5.49	0.58	0.28
query25	0.18	0.09	0.09
query26	0.57	0.23	0.18
query27	0.09	0.09	0.09
query28	11.00	1.21	0.58
query29	12.61	4.08	3.39
query30	3.10	3.06	3.05
query31	2.83	0.62	0.43
query32	3.24	0.60	0.52
query33	3.09	3.16	3.29
query34	17.19	5.46	4.70
query35	4.81	4.76	4.81
query36	0.64	0.51	0.50
query37	0.20	0.18	0.18
query38	0.17	0.16	0.15
query39	0.05	0.04	0.04
query40	0.19	0.17	0.18
query41	0.11	0.06	0.05
query42	0.05	0.06	0.05
query43	0.05	0.05	0.04
Total cold run time: 108.27 s
Total hot run time: 33.13 s

doris-robot · 2025-07-16T13:33:58Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.50% (15721/27340)
Line Coverage	46.14% (140197/303881)
Region Coverage	35.40% (104432/295034)
Branch Coverage	38.03% (46066/121129)

mymeiyi · 2025-07-17T03:01:06Z

+ std::string key1 = mow_tablet_job_key({instance_id, table_id + 1, 0});
+ MowTabletJobPB mow_tablet_job;
+ std::unique_ptr<RangeGetIterator> it;
+ int64_t job_key_num = 0;


txn->remove(key0, key1)

+1, why not remove by range?

This reverts commit 2896b84.

doris-robot · 2025-07-17T09:29:35Z

ClickBench: Total hot run time: 33.28 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3fab85f2d463f06fc4b7c3d1edb1414aa79058cb, data reload: false
query1	0.04	0.04	0.03
query2	0.10	0.05	0.06
query3	0.29	0.05	0.06
query4	1.60	0.09	0.08
query5	0.42	0.42	0.41
query6	1.15	0.66	0.65
query7	0.04	0.02	0.02
query8	0.07	0.05	0.04
query9	0.63	0.52	0.53
query10	0.60	0.58	0.58
query11	0.25	0.12	0.13
query12	0.25	0.13	0.13
query13	0.64	0.63	0.63
query14	0.81	0.83	0.85
query15	0.99	0.90	0.90
query16	0.38	0.38	0.39
query17	1.09	1.07	1.08
query18	0.25	0.23	0.24
query19	1.97	1.86	1.92
query20	0.01	0.02	0.02
query21	15.42	0.97	0.66
query22	0.93	1.08	0.84
query23	14.67	1.55	0.75
query24	5.23	0.60	0.31
query25	0.16	0.10	0.10
query26	0.56	0.23	0.20
query27	0.08	0.08	0.09
query28	10.98	1.24	0.59
query29	12.54	4.14	3.43
query30	3.07	3.01	2.99
query31	2.82	0.62	0.44
query32	3.25	0.60	0.51
query33	3.18	3.13	3.30
query34	16.96	5.40	4.72
query35	4.82	4.76	4.76
query36	0.65	0.52	0.50
query37	0.20	0.18	0.18
query38	0.18	0.15	0.16
query39	0.05	0.04	0.04
query40	0.20	0.16	0.17
query41	0.11	0.05	0.05
query42	0.07	0.05	0.05
query43	0.05	0.05	0.04
Total cold run time: 107.76 s
Total hot run time: 33.28 s

doris-robot · 2025-07-17T12:09:41Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.42% (15770/27466)
Line Coverage	46.19% (141908/307243)
Region Coverage	35.69% (106821/299344)
Branch Coverage	38.19% (47149/123453)

zhannngchen

LGTM

github-actions · 2025-07-18T02:34:38Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-07-18T02:34:40Z

PR approved by anyone and no changes requested.

bobhan1 · 2025-07-18T03:04:25Z

run buildall

doris-robot · 2025-07-18T03:25:32Z

Cloud UT Coverage Report

Increment line coverage 20.00% (9/45) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	80.36% (1297/1614)
Line Coverage	65.72% (21710/33035)
Region Coverage	67.08% (10918/16276)
Branch Coverage	56.65% (5745/10142)

hello-stephen · 2025-07-18T04:13:47Z

FE UT Coverage Report

Increment line coverage 0.00% (0/27) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2025-07-18T04:38:00Z

TPC-H: Total hot run time: 34448 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b879cbb2110e024c8b2ff6659a468b0134f9c9f8, data reload: false
------ Round 1 ----------------------------------
q1	17658	5303	5265	5265
q2	1957	303	196	196
q3	10365	1314	735	735
q4	10266	1043	545	545
q5	7979	2464	2382	2382
q6	199	170	147	147
q7	925	764	607	607
q8	9336	1374	1089	1089
q9	7062	5142	5178	5142
q10	6921	2420	2001	2001
q11	490	308	297	297
q12	354	384	237	237
q13	17791	3697	3057	3057
q14	230	231	226	226
q15	570	517	510	510
q16	430	456	389	389
q17	601	877	354	354
q18	7582	7151	7078	7078
q19	1246	989	596	596
q20	346	356	229	229
q21	3911	2623	2359	2359
q22	1082	1047	1007	1007
Total cold run time: 107301 ms
Total hot run time: 34448 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5182	5163	5135	5135
q2	254	334	229	229
q3	2177	2651	2332	2332
q4	1311	1778	1358	1358
q5	4330	4388	4601	4388
q6	221	170	130	130
q7	2000	1976	1828	1828
q8	2639	2628	2573	2573
q9	7288	7450	7353	7353
q10	3116	3340	2869	2869
q11	606	551	513	513
q12	693	762	635	635
q13	3538	3914	3436	3436
q14	316	324	289	289
q15	565	521	520	520
q16	463	504	454	454
q17	1181	1596	1417	1417
q18	7899	7863	7683	7683
q19	836	901	902	901
q20	2045	2157	1899	1899
q21	5070	4843	4535	4535
q22	1130	1094	972	972
Total cold run time: 52860 ms
Total hot run time: 51449 ms

doris-robot · 2025-07-18T04:49:51Z

TPC-DS: Total hot run time: 190388 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b879cbb2110e024c8b2ff6659a468b0134f9c9f8, data reload: false
query1	994	389	442	389
query2	6571	1769	1718	1718
query3	6747	227	223	223
query4	27330	23773	23537	23537
query5	4516	709	556	556
query6	330	239	221	221
query7	4637	526	311	311
query8	395	336	325	325
query9	8658	2682	2679	2679
query10	491	374	311	311
query11	16016	15731	14819	14819
query12	196	128	129	128
query13	1704	578	433	433
query14	9931	6148	6135	6135
query15	225	198	184	184
query16	7645	686	506	506
query17	1200	733	590	590
query18	2036	442	332	332
query19	209	204	172	172
query20	148	136	127	127
query21	226	144	127	127
query22	4223	4314	4009	4009
query23	34154	33215	33174	33174
query24	8420	2476	2404	2404
query25	578	496	406	406
query26	1250	279	163	163
query27	2731	518	361	361
query28	4341	2211	2180	2180
query29	728	579	455	455
query30	305	235	212	212
query31	979	888	848	848
query32	84	84	77	77
query33	588	408	347	347
query34	841	895	541	541
query35	818	854	764	764
query36	998	998	935	935
query37	116	110	89	89
query38	4160	4192	4135	4135
query39	1533	1485	1469	1469
query40	234	138	127	127
query41	107	101	107	101
query42	128	116	114	114
query43	501	532	490	490
query44	1407	869	881	869
query45	195	185	183	183
query46	887	1029	671	671
query47	1780	1852	1786	1786
query48	403	449	340	340
query49	830	587	480	480
query50	695	729	440	440
query51	5524	5542	5399	5399
query52	122	120	106	106
query53	256	279	210	210
query54	614	601	531	531
query55	88	90	89	89
query56	336	362	322	322
query57	1207	1224	1187	1187
query58	302	298	299	298
query59	2649	2716	2517	2517
query60	355	347	336	336
query61	153	142	151	142
query62	810	729	661	661
query63	254	217	214	214
query64	4249	1078	720	720
query65	4410	4201	4189	4189
query66	1216	585	489	489
query67	16221	15716	15898	15716
query68	7942	935	566	566
query69	564	340	297	297
query70	1253	1126	1166	1126
query71	479	357	326	326
query72	5404	4744	4671	4671
query73	698	610	380	380
query74	8888	9301	8855	8855
query75	3842	3252	2760	2760
query76	3652	1165	741	741
query77	842	460	362	362
query78	10038	10179	9308	9308
query79	2011	804	681	681
query80	662	575	529	529
query81	483	285	240	240
query82	426	140	113	113
query83	279	282	277	277
query84	262	119	103	103
query85	828	391	414	391
query86	354	297	301	297
query87	4395	4513	4340	4340
query88	3594	2407	2386	2386
query89	405	343	306	306
query90	1962	238	241	238
query91	158	159	126	126
query92	95	73	70	70
query93	1116	960	607	607
query94	706	413	320	320
query95	397	323	311	311
query96	517	573	292	292
query97	2808	2780	2712	2712
query98	247	230	234	230
query99	1454	1435	1308	1308
Total cold run time: 279746 ms
Total hot run time: 190388 ms

doris-robot · 2025-07-18T04:55:29Z

ClickBench: Total hot run time: 33.83 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b879cbb2110e024c8b2ff6659a468b0134f9c9f8, data reload: false
query1	0.04	0.03	0.04
query2	0.11	0.06	0.06
query3	0.30	0.06	0.06
query4	1.60	0.09	0.09
query5	0.43	0.43	0.43
query6	1.16	0.72	0.68
query7	0.03	0.02	0.02
query8	0.07	0.06	0.06
query9	0.67	0.54	0.55
query10	0.60	0.58	0.60
query11	0.26	0.14	0.13
query12	0.26	0.14	0.14
query13	0.68	0.65	0.64
query14	0.82	0.82	0.84
query15	1.02	0.91	0.92
query16	0.39	0.38	0.40
query17	1.13	1.11	1.07
query18	0.25	0.24	0.24
query19	2.01	1.90	2.00
query20	0.02	0.01	0.01
query21	15.36	0.98	0.68
query22	0.93	1.05	0.82
query23	14.72	1.60	0.77
query24	5.23	0.61	0.30
query25	0.17	0.10	0.09
query26	0.56	0.24	0.19
query27	0.10	0.09	0.09
query28	11.00	1.27	0.59
query29	12.60	4.09	3.37
query30	3.23	3.07	3.05
query31	2.82	0.63	0.45
query32	3.24	0.64	0.55
query33	3.20	3.20	3.20
query34	16.67	5.54	4.76
query35	4.92	4.88	4.88
query36	0.66	0.54	0.52
query37	0.22	0.19	0.18
query38	0.18	0.16	0.17
query39	0.06	0.05	0.06
query40	0.24	0.19	0.16
query41	0.11	0.06	0.07
query42	0.08	0.07	0.07
query43	0.08	0.06	0.05
Total cold run time: 108.23 s
Total hot run time: 33.83 s

doris-robot · 2025-07-18T07:30:28Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.36% (15759/27473)
Line Coverage	46.11% (141730/307361)
Region Coverage	35.61% (106653/299538)
Branch Coverage	38.10% (47089/123597)

bobhan1 · 2025-07-18T07:57:28Z

run p0

bobhan1 · 2025-07-18T07:57:33Z

run external

dataroaring

LGTM

…53566) Cherry-picked from #53086 Co-authored-by: bobhan1 <baohan@selectdb.com>

…66082) Related PR: #53086 Problem Summary: Urgent MOW load lock requests currently replace any existing delete bitmap update lock after the force-take threshold. If a heavy schema change owns the lock, the load can invalidate the schema change and make its later commit fail. This change keeps active schema change locks protected in both v1 and v2 lock implementations. Urgent loads continue to preempt compaction and load locks, and stale schema change locks still follow the existing expiration cleanup path. The unit-test matrix directly verifies persisted lock ownership and v2 job-key cleanup for active schema change, expired schema change, compaction, and ordinary load owners. An isolated cloud Docker case pauses schema change after lock acquisition, limits the urgent load to two immediate lock attempts, and verifies that the load fails with the lock conflict while schema change retains ownership. ### Release note Urgent MOW loads no longer preempt active schema change delete bitmap locks.

…pache#66082) Related PR: apache#53086 Problem Summary: Urgent MOW load lock requests currently replace any existing delete bitmap update lock after the force-take threshold. If a heavy schema change owns the lock, the load can invalidate the schema change and make its later commit fail. This change keeps active schema change locks protected in both v1 and v2 lock implementations. Urgent loads continue to preempt compaction and load locks, and stale schema change locks still follow the existing expiration cleanup path. The unit-test matrix directly verifies persisted lock ownership and v2 job-key cleanup for active schema change, expired schema change, compaction, and ordinary load owners. An isolated cloud Docker case pauses schema change after lock acquisition, limits the urgent load to two immediate lock attempts, and verifies that the load fails with the lock conflict while schema change retains ownership. ### Release note Urgent MOW loads no longer preempt active schema change delete bitmap locks.

bobhan1 force-pushed the mow-load-force-take-ms-lock branch 4 times, most recently from 2896b84 to 4bb4bf2Compare July 16, 2025 03:57

bobhan1 marked this pull request as ready for review July 16, 2025 03:58

bobhan1 requested review from dataroaring, gavinchou and w41ter as code owners July 16, 2025 03:58

mymeiyi reviewed Jul 17, 2025
View reviewed changes

bobhan1 added 7 commits July 17, 2025 11:11

ms part
acd8a9e

tmp
451c81b

fe part
c3890a6

fix
71754bb

config mow load get dbm lock backoff random sleep time
b8ab6c4

log every txn latency
d64aa03

Revert "log every txn latency"
e8ff5cc
This reverts commit 2896b84.

fix
b879cbb

zhannngchen approved these changes Jul 18, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 18, 2025

github-actionsBot added the reviewed label Jul 18, 2025

mymeiyi approved these changes Jul 18, 2025
View reviewed changes

dataroaring approved these changes Jul 18, 2025
View reviewed changes

dataroaring merged commit 5d0e9c9 into apache:masterJul 18, 2025

github-actionsBot mentioned this pull request Jul 18, 2025
branch-3.1: [Opt](cloud-mow) Mow load can force take MS lock #53086 #53566
Merged

morrySnow pushed a commit that referenced this pull request Jul 21, 2025
branch-3.1: [Opt](cloud-mow) Mow load can force take MS lock#53086(#…
1fd4437
…53566) Cherry-picked from #53086 Co-authored-by: bobhan1 <baohan@selectdb.com>

morrySnow added dev/3.1.0-merged and removed dev/3.1.x labels Jul 21, 2025

bobhan1 mentioned this pull request Jul 27, 2026
[fix](cloud) Prevent urgent load from preempting schema change lock #66082
Merged

Uh oh!

Conversation

bobhan1 commented Jul 11, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Test 1: cloud mode, 1 FE, 3 BE(8C 32GB), 5 threads, 120 buckets, 50rows/per load, cumu threads=8

Test 2: cloud mode, 1 FE, 3 BE(8C 32GB), 1 threads, 120 buckets, 50rows/per load, cumu threads=8

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Jul 11, 2025

Uh oh!

bobhan1 commented Jul 16, 2025

Uh oh!

doris-robot commented Jul 16, 2025

Cloud UT Coverage Report

Uh oh!

doris-robot commented Jul 16, 2025

Uh oh!

doris-robot commented Jul 16, 2025

Uh oh!

doris-robot commented Jul 16, 2025

Uh oh!

doris-robot commented Jul 16, 2025

BE UT Coverage Report

Uh oh!

bobhan1 commented Jul 16, 2025

Uh oh!

hello-stephen commented Jul 16, 2025

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Jul 16, 2025

FE UT Coverage Report

Uh oh!

doris-robot commented Jul 16, 2025

Uh oh!

doris-robot commented Jul 16, 2025

Uh oh!

doris-robot commented Jul 16, 2025

Uh oh!

doris-robot commented Jul 16, 2025

BE UT Coverage Report

Uh oh!

mymeiyiJul 17, 2025

Choose a reason for hiding this comment

Uh oh!

zhannngchenJul 17, 2025

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Jul 17, 2025

Uh oh!

doris-robot commented Jul 17, 2025

BE UT Coverage Report

Uh oh!

zhannngchen left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jul 18, 2025

Uh oh!

github-actionsBot commented Jul 18, 2025

Uh oh!

bobhan1 commented Jul 18, 2025

Uh oh!

doris-robot commented Jul 18, 2025

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Jul 18, 2025

FE UT Coverage Report

Uh oh!

doris-robot commented Jul 18, 2025

Uh oh!

doris-robot commented Jul 18, 2025

Uh oh!

doris-robot commented Jul 18, 2025

Uh oh!

doris-robot commented Jul 18, 2025

BE UT Coverage Report

Uh oh!

bobhan1 commented Jul 18, 2025

bobhan1 commented Jul 11, 2025•
edited
Loading