[enhance](memtable) skip memtable in queue to speed up cancel by sollhui · Pull Request #53481 · apache/doris

sollhui · 2025-07-17T13:29:47Z

What problem does this PR solve?

At present, if load/update/delete is cancelled, it needs to wait for all submitted memtable task of related FlushToken to be completed, included:

flush task wait in thread pool queue
ongoing flushing task

It is not need wait flush task still in the thread pool queue which can speed up cancel load/update/delete.

Specific Example

For example, cancel load1. The current logic is to wait for all submit task(queue+flushing), but for those in the queue, they have to wait for all the previously queued ones to complete before it's their turn. Skip memtable in queue can speed up cancel for:

If there are still memtables for load2 and load3 in front of load1 memtable, cancel block until other load memtable been flushed.
Under more extreme conditions, the submit task count may add when wait submit task finish.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-07-17T13:29:53Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

sollhui · 2025-07-17T13:30:16Z

run buildall

doris-robot · 2025-07-17T13:48:33Z

TPC-H: Total hot run time: 33775 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9c065f65f3b9145bfd3de8de037e3aab570cdf1f, data reload: false
------ Round 1 ----------------------------------
q1	17617	5186	5048	5048
q2	1925	273	186	186
q3	10319	1272	727	727
q4	10220	981	530	530
q5	7504	2349	2322	2322
q6	179	163	127	127
q7	886	743	609	609
q8	9286	1291	1049	1049
q9	6724	5120	5097	5097
q10	6870	2390	1946	1946
q11	468	279	270	270
q12	333	353	213	213
q13	17748	3705	3071	3071
q14	220	227	210	210
q15	549	473	490	473
q16	419	430	377	377
q17	563	863	350	350
q18	7570	7289	7130	7130
q19	1209	949	562	562
q20	350	333	216	216
q21	3604	2518	2295	2295
q22	1053	978	967	967
Total cold run time: 105616 ms
Total hot run time: 33775 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5076	5118	5075	5075
q2	240	334	225	225
q3	2190	2686	2285	2285
q4	1392	1759	1368	1368
q5	4210	4268	4547	4268
q6	212	171	127	127
q7	2119	1992	1820	1820
q8	2609	2514	2481	2481
q9	7355	7352	7154	7154
q10	3105	3284	2850	2850
q11	563	516	517	516
q12	676	778	607	607
q13	3604	4002	3418	3418
q14	310	318	283	283
q15	543	477	454	454
q16	472	495	469	469
q17	1191	1552	1391	1391
q18	7843	7741	7657	7657
q19	778	806	912	806
q20	2018	2112	1843	1843
q21	4725	4368	4288	4288
q22	1056	996	978	978
Total cold run time: 52287 ms
Total hot run time: 50363 ms

doris-robot · 2025-07-17T14:00:14Z

TPC-DS: Total hot run time: 187078 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9c065f65f3b9145bfd3de8de037e3aab570cdf1f, data reload: false
query1	1001	381	402	381
query2	6509	1739	1777	1739
query3	6749	222	227	222
query4	26995	23793	23423	23423
query5	4339	579	425	425
query6	316	216	205	205
query7	4629	489	290	290
query8	271	221	210	210
query9	8606	2586	2618	2586
query10	491	360	289	289
query11	15767	14950	14837	14837
query12	172	112	106	106
query13	1648	526	404	404
query14	9489	5926	5848	5848
query15	208	201	170	170
query16	7619	627	448	448
query17	1225	686	568	568
query18	2023	412	294	294
query19	184	176	152	152
query20	118	113	111	111
query21	217	121	108	108
query22	4378	4130	4301	4130
query23	34205	33078	32962	32962
query24	8448	2384	2385	2384
query25	536	456	393	393
query26	1239	261	149	149
query27	2757	503	340	340
query28	4277	2143	2140	2140
query29	715	541	442	442
query30	287	220	194	194
query31	947	852	778	778
query32	80	70	65	65
query33	567	364	321	321
query34	799	845	518	518
query35	794	814	734	734
query36	949	989	915	915
query37	112	121	74	74
query38	4187	4200	4119	4119
query39	1474	1416	1433	1416
query40	212	115	106	106
query41	57	59	52	52
query42	118	105	111	105
query43	510	521	500	500
query44	1315	824	819	819
query45	177	170	165	165
query46	840	1006	636	636
query47	1746	1837	1723	1723
query48	370	430	307	307
query49	732	488	380	380
query50	631	683	413	413
query51	5533	5609	5474	5474
query52	112	108	96	96
query53	228	253	176	176
query54	574	584	509	509
query55	84	82	82	82
query56	292	315	283	283
query57	1175	1220	1154	1154
query58	262	259	278	259
query59	2594	2675	2618	2618
query60	329	329	300	300
query61	126	140	123	123
query62	822	718	685	685
query63	224	194	187	187
query64	4326	1019	662	662
query65	4338	4170	4173	4170
query66	1069	418	330	330
query67	16023	15720	15424	15424
query68	7860	867	523	523
query69	475	310	280	280
query70	1205	1105	1153	1105
query71	428	317	298	298
query72	5726	4709	4824	4709
query73	654	592	360	360
query74	9230	9125	8704	8704
query75	3411	3187	2688	2688
query76	3346	1190	716	716
query77	617	395	309	309
query78	9834	10110	9418	9418
query79	2243	832	576	576
query80	599	509	455	455
query81	479	260	242	242
query82	466	131	102	102
query83	276	254	237	237
query84	302	116	101	101
query85	885	442	415	415
query86	469	317	300	300
query87	4556	4474	4363	4363
query88	3525	2301	2296	2296
query89	386	331	296	296
query90	1901	216	214	214
query91	156	145	112	112
query92	75	64	59	59
query93	1851	936	583	583
query94	717	410	300	300
query95	366	300	284	284
query96	493	580	282	282
query97	2702	2739	2646	2646
query98	245	207	208	207
query99	1410	1390	1270	1270
Total cold run time: 277034 ms
Total hot run time: 187078 ms

doris-robot · 2025-07-17T14:05:43Z

ClickBench: Total hot run time: 32.52 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9c065f65f3b9145bfd3de8de037e3aab570cdf1f, data reload: false
query1	0.04	0.04	0.04
query2	0.08	0.04	0.05
query3	0.24	0.07	0.06
query4	1.62	0.11	0.11
query5	0.45	0.41	0.42
query6	1.15	0.66	0.66
query7	0.03	0.02	0.01
query8	0.04	0.03	0.04
query9	0.60	0.52	0.52
query10	0.59	0.57	0.59
query11	0.16	0.11	0.11
query12	0.15	0.12	0.11
query13	0.63	0.61	0.60
query14	0.80	0.82	0.82
query15	0.90	0.85	0.88
query16	0.37	0.40	0.40
query17	1.02	1.04	1.08
query18	0.22	0.21	0.22
query19	1.91	1.85	1.86
query20	0.01	0.00	0.01
query21	15.42	0.89	0.54
query22	0.76	1.22	0.67
query23	14.87	1.39	0.62
query24	6.81	1.06	0.85
query25	0.50	0.22	0.09
query26	0.62	0.17	0.15
query27	0.08	0.06	0.06
query28	9.40	0.93	0.44
query29	12.60	4.00	3.26
query30	3.09	3.04	2.99
query31	2.82	0.59	0.39
query32	3.24	0.59	0.46
query33	3.05	3.13	3.06
query34	16.13	5.44	4.86
query35	4.82	4.82	4.89
query36	0.68	0.50	0.48
query37	0.09	0.07	0.07
query38	0.05	0.04	0.03
query39	0.02	0.02	0.02
query40	0.17	0.16	0.13
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.04	0.04
Total cold run time: 106.38 s
Total hot run time: 32.52 s

hello-stephen · 2025-07-17T17:09:03Z

BE UT Coverage Report

Increment line coverage 86.67% (13/15) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.40% (15760/27456)
Line Coverage	46.17% (141741/307025)
Region Coverage	35.64% (106651/299226)
Branch Coverage	38.17% (47096/123387)

hello-stephen · 2025-07-17T17:15:21Z

BE Regression && UT Coverage Report

Increment line coverage 93.33% (14/15) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	81.13% (21854/26936)
Line Coverage	73.63% (225722/306555)
Region Coverage	61.27% (188086/306957)
Branch Coverage	65.02% (81130/124777)

sollhui · 2025-07-18T03:02:18Z

run buildall

doris-robot · 2025-07-18T03:49:19Z

TPC-H: Total hot run time: 34429 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit eb7213b2dc85d436134786cc3f2195cdf8b43de8, data reload: false
------ Round 1 ----------------------------------
q1	17591	5257	5154	5154
q2	1971	295	197	197
q3	10375	1353	730	730
q4	10283	1023	524	524
q5	8280	2484	2343	2343
q6	197	168	142	142
q7	921	766	626	626
q8	9343	1391	1150	1150
q9	6876	5158	5167	5158
q10	6895	2391	1998	1998
q11	503	299	288	288
q12	354	362	230	230
q13	17773	3786	3131	3131
q14	240	263	229	229
q15	593	521	514	514
q16	437	457	402	402
q17	601	890	370	370
q18	7597	7089	7146	7089
q19	1352	976	607	607
q20	348	360	230	230
q21	3984	3188	2341	2341
q22	1062	1057	976	976
Total cold run time: 107576 ms
Total hot run time: 34429 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5224	5191	5157	5157
q2	265	345	244	244
q3	2186	2703	2321	2321
q4	1369	1838	1330	1330
q5	4484	4491	4472	4472
q6	234	180	138	138
q7	2034	1931	1849	1849
q8	2654	2654	2578	2578
q9	7222	7329	7262	7262
q10	3111	3310	2909	2909
q11	602	527	514	514
q12	691	799	650	650
q13	3617	3967	3401	3401
q14	286	311	279	279
q15	546	511	521	511
q16	454	523	460	460
q17	1151	1550	1437	1437
q18	7908	7639	7533	7533
q19	805	832	1044	832
q20	2076	2265	1934	1934
q21	5051	4696	4254	4254
q22	1074	1027	976	976
Total cold run time: 53044 ms
Total hot run time: 51041 ms

doris-robot · 2025-07-18T04:01:10Z

TPC-DS: Total hot run time: 190132 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit eb7213b2dc85d436134786cc3f2195cdf8b43de8, data reload: false
query1	1008	407	405	405
query2	6535	1800	1725	1725
query3	6751	227	224	224
query4	26872	23332	23202	23202
query5	4477	718	541	541
query6	331	234	225	225
query7	4651	512	312	312
query8	398	351	319	319
query9	8645	2651	2683	2651
query10	533	360	308	308
query11	15561	15341	14915	14915
query12	185	128	126	126
query13	1691	574	459	459
query14	10101	6043	6070	6043
query15	223	197	180	180
query16	7649	647	465	465
query17	1331	731	603	603
query18	2031	431	332	332
query19	200	206	175	175
query20	140	131	133	131
query21	237	148	126	126
query22	4265	4306	4136	4136
query23	34610	33045	33318	33045
query24	8340	2393	2395	2393
query25	523	489	403	403
query26	1236	281	162	162
query27	2721	512	354	354
query28	4341	2187	2164	2164
query29	750	567	467	467
query30	305	237	205	205
query31	987	883	817	817
query32	90	81	77	77
query33	580	376	352	352
query34	818	863	530	530
query35	803	872	776	776
query36	1020	1014	896	896
query37	126	108	117	108
query38	4191	4179	4054	4054
query39	1548	1463	1477	1463
query40	233	141	128	128
query41	106	101	101	101
query42	128	117	116	116
query43	510	512	499	499
query44	1393	891	865	865
query45	207	184	177	177
query46	886	1033	650	650
query47	1805	1855	1736	1736
query48	400	464	331	331
query49	819	571	474	474
query50	678	728	423	423
query51	5573	5696	5394	5394
query52	118	122	116	116
query53	259	279	211	211
query54	616	605	530	530
query55	93	89	87	87
query56	317	346	323	323
query57	1218	1261	1161	1161
query58	305	288	309	288
query59	2648	2644	2587	2587
query60	369	344	363	344
query61	152	146	143	143
query62	822	739	698	698
query63	243	214	213	213
query64	4277	1126	712	712
query65	4297	4203	4220	4203
query66	1227	616	497	497
query67	15643	15761	15732	15732
query68	8742	933	565	565
query69	485	332	297	297
query70	1239	1177	1138	1138
query71	481	342	332	332
query72	5374	4754	4606	4606
query73	692	593	378	378
query74	9189	9225	9107	9107
query75	4151	3288	2734	2734
query76	3739	1166	752	752
query77	835	541	363	363
query78	10111	10433	9504	9504
query79	2130	858	627	627
query80	675	582	540	540
query81	515	279	244	244
query82	460	144	113	113
query83	284	295	278	278
query84	266	111	95	95
query85	845	395	351	351
query86	404	299	325	299
query87	4459	4483	4392	4392
query88	3631	2396	2382	2382
query89	406	354	313	313
query90	1876	233	233	233
query91	153	153	131	131
query92	88	74	69	69
query93	1587	984	606	606
query94	700	440	329	329
query95	397	312	303	303
query96	504	584	288	288
query97	2757	2809	2628	2628
query98	270	218	209	209
query99	1478	1450	1318	1318
Total cold run time: 281069 ms
Total hot run time: 190132 ms

doris-robot · 2025-07-18T04:06:40Z

ClickBench: Total hot run time: 32.67 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit eb7213b2dc85d436134786cc3f2195cdf8b43de8, data reload: false
query1	0.05	0.04	0.04
query2	0.07	0.04	0.05
query3	0.25	0.08	0.07
query4	1.63	0.11	0.11
query5	0.43	0.43	0.41
query6	1.14	0.65	0.69
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.60	0.53	0.51
query10	0.58	0.58	0.59
query11	0.16	0.12	0.11
query12	0.15	0.13	0.12
query13	0.63	0.62	0.61
query14	0.81	0.81	0.81
query15	0.90	0.91	0.88
query16	0.39	0.39	0.40
query17	1.05	1.09	1.06
query18	0.23	0.22	0.22
query19	2.01	1.86	1.92
query20	0.01	0.01	0.01
query21	15.39	0.92	0.55
query22	0.76	1.19	0.76
query23	14.78	1.40	0.62
query24	6.81	1.40	0.67
query25	0.50	0.21	0.10
query26	0.60	0.17	0.15
query27	0.06	0.06	0.05
query28	9.64	0.91	0.45
query29	12.57	3.93	3.33
query30	3.17	3.09	2.97
query31	2.83	0.62	0.39
query32	3.24	0.56	0.47
query33	3.11	3.10	3.09
query34	16.02	5.40	4.79
query35	4.84	4.89	4.84
query36	0.69	0.51	0.52
query37	0.10	0.07	0.08
query38	0.06	0.04	0.04
query39	0.04	0.03	0.03
query40	0.19	0.15	0.15
query41	0.08	0.04	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.05
Total cold run time: 106.74 s
Total hot run time: 32.67 s

sollhui · 2025-07-18T04:57:11Z

run buildall

doris-robot · 2025-07-18T05:15:49Z

TPC-H: Total hot run time: 34378 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e619f2e14ecbc95b4a6a2c8e8f990fcf89c119a3, data reload: false
------ Round 1 ----------------------------------
q1	17624	5283	5092	5092
q2	1954	298	201	201
q3	10437	1310	722	722
q4	10312	1013	536	536
q5	8591	2419	2413	2413
q6	202	168	136	136
q7	938	769	606	606
q8	9336	1298	1109	1109
q9	7325	5200	5171	5171
q10	7012	2458	1965	1965
q11	538	320	297	297
q12	381	376	240	240
q13	17792	3748	3069	3069
q14	260	246	227	227
q15	610	531	554	531
q16	452	463	403	403
q17	612	888	383	383
q18	7544	7131	7134	7131
q19	1217	985	594	594
q20	350	360	242	242
q21	3841	3180	2324	2324
q22	1086	1072	986	986
Total cold run time: 108414 ms
Total hot run time: 34378 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5261	5154	5163	5154
q2	282	353	246	246
q3	2195	2708	2332	2332
q4	1372	1799	1323	1323
q5	4297	4447	4600	4447
q6	232	187	140	140
q7	2049	2017	1787	1787
q8	2669	2639	2542	2542
q9	7310	7239	7215	7215
q10	3169	3327	2922	2922
q11	619	546	507	507
q12	712	775	633	633
q13	3600	4020	3442	3442
q14	298	316	283	283
q15	553	540	514	514
q16	458	516	464	464
q17	1213	1574	1409	1409
q18	7958	7583	7490	7490
q19	834	840	1335	840
q20	2043	2059	1911	1911
q21	5019	4637	4366	4366
q22	1099	1092	1010	1010
Total cold run time: 53242 ms
Total hot run time: 50977 ms

doris-robot · 2025-07-18T05:27:41Z

TPC-DS: Total hot run time: 189841 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e619f2e14ecbc95b4a6a2c8e8f990fcf89c119a3, data reload: false
query1	987	399	397	397
query2	6554	1796	1778	1778
query3	6763	227	222	222
query4	26226	23866	23004	23004
query5	4512	712	535	535
query6	321	251	253	251
query7	4639	511	309	309
query8	391	349	327	327
query9	8679	2644	2634	2634
query10	499	376	321	321
query11	15892	15136	14940	14940
query12	187	128	125	125
query13	1695	568	434	434
query14	9891	6004	5866	5866
query15	220	205	187	187
query16	7674	644	439	439
query17	1196	736	598	598
query18	2041	428	324	324
query19	201	197	171	171
query20	132	126	126	126
query21	232	149	131	131
query22	4174	4358	4279	4279
query23	34106	33341	33192	33192
query24	8462	2422	2436	2422
query25	559	471	404	404
query26	1241	290	159	159
query27	2736	521	360	360
query28	4326	2191	2161	2161
query29	729	567	447	447
query30	307	234	252	234
query31	965	865	847	847
query32	88	72	72	72
query33	588	393	334	334
query34	809	882	531	531
query35	836	831	770	770
query36	993	1016	903	903
query37	119	110	85	85
query38	4209	4263	4137	4137
query39	1541	1484	1558	1484
query40	237	141	123	123
query41	105	100	98	98
query42	130	123	118	118
query43	534	531	520	520
query44	1324	842	851	842
query45	197	183	185	183
query46	857	1034	656	656
query47	1820	1888	1768	1768
query48	410	440	330	330
query49	845	554	460	460
query50	665	719	443	443
query51	5517	5480	5531	5480
query52	118	114	109	109
query53	268	282	198	198
query54	595	588	527	527
query55	90	86	88	86
query56	318	339	320	320
query57	1221	1215	1198	1198
query58	302	299	305	299
query59	2648	2813	2672	2672
query60	372	345	343	343
query61	153	145	151	145
query62	810	743	663	663
query63	239	211	208	208
query64	4338	1090	714	714
query65	4318	4213	4193	4193
query66	1292	581	546	546
query67	16134	15640	15663	15640
query68	8983	913	559	559
query69	492	335	301	301
query70	1251	1167	1147	1147
query71	517	350	326	326
query72	5568	4767	4582	4582
query73	751	605	376	376
query74	9297	9124	8674	8674
query75	4202	3266	2765	2765
query76	3737	1166	727	727
query77	858	453	430	430
query78	9992	10277	9411	9411
query79	2615	845	633	633
query80	703	559	516	516
query81	487	278	256	256
query82	225	137	108	108
query83	378	292	275	275
query84	308	115	99	99
query85	792	409	355	355
query86	357	345	310	310
query87	4445	4561	4328	4328
query88	3067	2400	2381	2381
query89	438	344	317	317
query90	2127	232	234	232
query91	159	160	130	130
query92	88	74	69	69
query93	2166	1004	615	615
query94	686	437	320	320
query95	391	313	308	308
query96	517	582	290	290
query97	2740	2782	2657	2657
query98	242	221	219	219
query99	1431	1416	1303	1303
Total cold run time: 281587 ms
Total hot run time: 189841 ms

doris-robot · 2025-07-18T05:33:12Z

ClickBench: Total hot run time: 33.11 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e619f2e14ecbc95b4a6a2c8e8f990fcf89c119a3, data reload: false
query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.25	0.07	0.07
query4	1.62	0.11	0.11
query5	0.44	0.44	0.41
query6	1.13	0.66	0.68
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.61	0.54	0.52
query10	0.58	0.58	0.59
query11	0.16	0.12	0.11
query12	0.15	0.12	0.13
query13	0.63	0.61	0.61
query14	0.82	0.85	0.82
query15	0.90	0.89	0.87
query16	0.42	0.39	0.41
query17	1.06	1.06	1.11
query18	0.23	0.21	0.22
query19	2.00	1.85	1.88
query20	0.03	0.02	0.02
query21	15.36	0.90	0.53
query22	0.77	1.18	0.69
query23	14.94	1.40	0.61
query24	7.34	1.37	0.88
query25	0.45	0.14	0.13
query26	0.71	0.18	0.16
query27	0.07	0.05	0.06
query28	9.46	0.91	0.44
query29	12.55	4.01	3.35
query30	3.08	3.06	3.03
query31	2.83	0.58	0.39
query32	3.24	0.56	0.49
query33	3.15	3.18	3.19
query34	16.16	5.46	4.86
query35	4.84	4.89	4.87
query36	0.71	0.51	0.50
query37	0.10	0.07	0.08
query38	0.06	0.05	0.05
query39	0.04	0.04	0.03
query40	0.19	0.15	0.15
query41	0.09	0.04	0.04
query42	0.04	0.03	0.03
query43	0.05	0.04	0.05
Total cold run time: 107.46 s
Total hot run time: 33.11 s

doris-robot · 2025-07-18T08:34:37Z

BE UT Coverage Report

Increment line coverage 77.78% (14/18) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.36% (15759/27476)
Line Coverage	46.11% (141736/307393)
Region Coverage	35.61% (106688/299567)
Branch Coverage	38.11% (47105/123607)

hello-stephen · 2025-07-18T08:39:36Z

BE Regression && UT Coverage Report

Increment line coverage 83.33% (15/18) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	81.15% (21876/26956)
Line Coverage	73.76% (226380/306922)
Region Coverage	61.46% (188879/307297)
Branch Coverage	65.26% (81573/124997)

sollhui · 2025-07-20T15:20:05Z

run buildall

sollhui · 2025-07-20T15:24:41Z

run buildall

doris-robot · 2025-07-20T15:43:57Z

TPC-H: Total hot run time: 34503 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 323a1b7e6f1de9feb9cf8f01b787d371861b25fd, data reload: false
------ Round 1 ----------------------------------
q1	17629	5246	5099	5099
q2	1942	299	197	197
q3	10435	1272	729	729
q4	10312	1027	547	547
q5	8923	2448	2334	2334
q6	197	170	135	135
q7	942	759	612	612
q8	9314	1335	1117	1117
q9	6879	5166	5170	5166
q10	6924	2375	2014	2014
q11	486	313	291	291
q12	356	378	230	230
q13	17766	3665	3111	3111
q14	263	244	234	234
q15	600	522	508	508
q16	443	444	389	389
q17	588	884	370	370
q18	7648	7230	7234	7230
q19	1379	978	592	592
q20	345	344	242	242
q21	3829	3163	2355	2355
q22	1101	1091	1001	1001
Total cold run time: 108301 ms
Total hot run time: 34503 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5168	5130	5077	5077
q2	266	345	243	243
q3	2164	2646	2340	2340
q4	1351	1786	1334	1334
q5	4412	4388	4489	4388
q6	213	174	133	133
q7	2030	1961	1762	1762
q8	2633	2633	2477	2477
q9	7290	7408	7316	7316
q10	3149	3323	2843	2843
q11	586	508	511	508
q12	693	799	619	619
q13	3596	3961	3445	3445
q14	289	341	299	299
q15	576	528	504	504
q16	451	513	470	470
q17	1177	1603	1367	1367
q18	7994	7679	7535	7535
q19	801	799	892	799
q20	2011	2097	1929	1929
q21	4963	4431	4257	4257
q22	1053	1048	1033	1033
Total cold run time: 52866 ms
Total hot run time: 50678 ms

doris-robot · 2025-07-20T15:55:48Z

TPC-DS: Total hot run time: 191023 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 323a1b7e6f1de9feb9cf8f01b787d371861b25fd, data reload: false
query1	1014	399	414	399
query2	6545	1677	1704	1677
query3	6749	235	221	221
query4	26235	23800	23436	23436
query5	4519	747	606	606
query6	342	243	223	223
query7	4664	526	323	323
query8	399	354	340	340
query9	8634	2942	2898	2898
query10	518	388	324	324
query11	16181	15179	14812	14812
query12	191	138	129	129
query13	1692	579	446	446
query14	10308	6075	6098	6075
query15	225	211	177	177
query16	7657	628	497	497
query17	1615	762	597	597
query18	2061	451	341	341
query19	218	210	183	183
query20	141	139	133	133
query21	230	148	132	132
query22	4144	4290	4160	4160
query23	34202	33271	33162	33162
query24	8072	2447	2390	2390
query25	537	502	443	443
query26	1238	324	172	172
query27	2621	522	363	363
query28	4376	2218	2191	2191
query29	745	593	475	475
query30	300	237	202	202
query31	977	886	800	800
query32	97	88	89	88
query33	593	407	371	371
query34	800	870	569	569
query35	825	861	796	796
query36	961	1019	916	916
query37	127	111	94	94
query38	4252	4180	4084	4084
query39	1561	1495	1489	1489
query40	239	148	138	138
query41	117	104	103	103
query42	135	122	118	118
query43	520	525	482	482
query44	1377	866	882	866
query45	197	191	185	185
query46	889	1021	667	667
query47	1842	1902	1799	1799
query48	418	462	344	344
query49	803	574	482	482
query50	675	724	440	440
query51	5510	5521	5460	5460
query52	127	122	121	121
query53	250	287	219	219
query54	625	627	554	554
query55	112	101	97	97
query56	347	360	339	339
query57	1254	1245	1187	1187
query58	325	322	313	313
query59	2609	2693	2574	2574
query60	386	381	359	359
query61	155	153	158	153
query62	815	770	694	694
query63	250	224	214	214
query64	4286	1073	746	746
query65	4303	4230	4196	4196
query66	1271	597	542	542
query67	15969	15571	15438	15438
query68	8167	907	550	550
query69	486	417	314	314
query70	1276	1177	1150	1150
query71	490	355	359	355
query72	5340	4754	4805	4754
query73	757	650	376	376
query74	9270	8901	8922	8901
query75	3979	3145	2763	2763
query76	3734	1147	751	751
query77	856	459	395	395
query78	9999	10220	9391	9391
query79	2194	841	636	636
query80	752	603	536	536
query81	558	283	253	253
query82	486	146	118	118
query83	314	287	288	287
query84	306	117	89	89
query85	833	407	356	356
query86	361	337	312	312
query87	4435	4372	4394	4372
query88	2924	2395	2357	2357
query89	425	357	304	304
query90	1994	235	239	235
query91	153	158	133	133
query92	99	83	85	83
query93	1404	953	582	582
query94	691	407	317	317
query95	418	344	325	325
query96	495	603	289	289
query97	2748	2780	2655	2655
query98	259	236	234	234
query99	1446	1438	1286	1286
Total cold run time: 280021 ms
Total hot run time: 191023 ms

doris-robot · 2025-07-20T16:01:16Z

ClickBench: Total hot run time: 32.82 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 323a1b7e6f1de9feb9cf8f01b787d371861b25fd, data reload: false
query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.24	0.07	0.07
query4	1.62	0.11	0.10
query5	0.42	0.42	0.43
query6	1.18	0.67	0.67
query7	0.03	0.02	0.02
query8	0.06	0.04	0.04
query9	0.60	0.53	0.52
query10	0.59	0.58	0.58
query11	0.16	0.12	0.12
query12	0.16	0.12	0.12
query13	0.64	0.62	0.61
query14	0.83	0.82	0.80
query15	0.89	0.86	0.88
query16	0.39	0.38	0.40
query17	1.04	1.04	1.13
query18	0.23	0.22	0.21
query19	1.93	1.85	1.91
query20	0.02	0.02	0.01
query21	15.36	0.92	0.54
query22	0.82	1.25	0.78
query23	14.76	1.41	0.65
query24	6.71	1.05	0.67
query25	0.50	0.09	0.17
query26	0.68	0.17	0.14
query27	0.06	0.05	0.06
query28	9.24	0.96	0.44
query29	12.55	3.97	3.28
query30	3.16	3.13	3.07
query31	2.82	0.59	0.38
query32	3.23	0.55	0.48
query33	3.07	3.17	3.19
query34	16.02	5.46	4.83
query35	4.88	4.86	4.90
query36	0.69	0.51	0.50
query37	0.10	0.08	0.07
query38	0.06	0.05	0.05
query39	0.04	0.03	0.03
query40	0.19	0.15	0.15
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 106.24 s
Total hot run time: 32.82 s

doris-robot · 2025-07-20T18:56:21Z

BE UT Coverage Report

Increment line coverage 83.33% (20/24) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.45% (15892/27663)
Line Coverage	46.25% (142733/308631)
Region Coverage	35.74% (107563/300948)
Branch Coverage	38.23% (47438/124098)

hello-stephen · 2025-07-20T19:00:25Z

BE Regression && UT Coverage Report

Increment line coverage 87.50% (21/24) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	81.01% (21991/27147)
Line Coverage	73.65% (226973/308165)
Region Coverage	61.24% (189048/308677)
Branch Coverage	65.05% (81636/125490)

hello-stephen · 2025-07-21T04:19:16Z

BE Regression && UT Coverage Report

Increment line coverage 87.50% (21/24) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	81.00% (21989/27147)
Line Coverage	73.62% (226879/308165)
Region Coverage	61.21% (188936/308677)
Branch Coverage	65.01% (81583/125490)

hello-stephen · 2025-07-21T06:55:54Z

BE Regression && UT Coverage Report

Increment line coverage 87.50% (21/24) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	80.99% (21985/27147)
Line Coverage	73.63% (226888/308165)
Region Coverage	61.17% (188831/308677)
Branch Coverage	65.01% (81581/125490)

dataroaring

LGTM

github-actions · 2025-07-21T09:08:25Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-07-21T09:08:27Z

PR approved by anyone and no changes requested.

liaoxin01

LGTM

…#53481) At present, if load/update/delete is cancelled, it needs to wait for all submitted memtable task of related `FlushToken` to be completed, included: - flush task wait in thread pool queue - ongoing flushing task It is not need wait flush task still in the thread pool queue which can speed up cancel load/update/delete. <img width="1686" height="660" alt="image" src="https://github.com/user-attachments/assets/8eb8f43c-10d9-4b95-8933-d341f12857ce" /> For example, cancel load1. The current logic is to wait for all submit task(queue+flushing), but for those in the queue, they have to wait for all the previously queued ones to complete before it's their turn. Skip memtable in queue can speed up cancel for: - If there are still memtables for load2 and load3 in front of load1 memtable, cancel block until other load memtable been flushed. - Under more extreme conditions, the submit task count may add when wait submit task finish.

…ncel #53481 (#54122) pick #53481

## Problem When a queued memtable flush task starts after `FlushToken::cancel()` has already marked the token shutdown, `_flush_memtable()` can return before the old `flush_running_count++` path but still run the deferred `flush_running_count--`. This can drive the counter below zero and make `cancel()` wait forever for `flush_running_count == 0`. Related PR: #53481 ## Fix Move `flush_running_count++` to the top of `_flush_memtable()` before registering the deferred cleanup so the running-count accounting stays symmetric on every exit path. ## Validation - Reasoned from production stack and gdb evidence showing `flush_running_count = -1` while cancel was blocked in `_wait_running_task_finish()` - Not run locally: the BE UT environment in this workspace would require a full initial `ut_build_ASAN` build

…1684) ## Problem When a queued memtable flush task starts after `FlushToken::cancel()` has already marked the token shutdown, `_flush_memtable()` can return before the old `flush_running_count++` path but still run the deferred `flush_running_count--`. This can drive the counter below zero and make `cancel()` wait forever for `flush_running_count == 0`. Related PR: apache#53481 ## Fix Move `flush_running_count++` to the top of `_flush_memtable()` before registering the deferred cleanup so the running-count accounting stays symmetric on every exit path. ## Validation - Reasoned from production stack and gdb evidence showing `flush_running_count = -1` while cancel was blocked in `_wait_running_task_finish()` - Not run locally: the BE UT environment in this workspace would require a full initial `ut_build_ASAN` build

sollhui force-pushed the cancel_memtable branch from 9c065f6 to eb7213bCompare July 18, 2025 03:02

sollhui changed the title ~~[enhance](memtable) speed up cancel memtable~~[enhance](memtable) skip memtable in queue to speed up cancelJul 18, 2025

sollhui force-pushed the cancel_memtable branch from eb7213b to e619f2eCompare July 18, 2025 04:57

sollhui force-pushed the cancel_memtable branch from e619f2e to c1d2c88Compare July 20, 2025 15:19

skip memtable in queue to speed up cancel
323a1b7

sollhui force-pushed the cancel_memtable branch from c1d2c88 to 323a1b7Compare July 20, 2025 15:24

dataroaring approved these changes Jul 21, 2025
View reviewed changes

dataroaring added the dev/3.1.x label Jul 21, 2025

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 21, 2025

github-actionsBot added the reviewed label Jul 21, 2025

liaoxin01 approved these changes Jul 21, 2025
View reviewed changes

dataroaring merged commit b871f46 into apache:masterJul 21, 2025

github-actionsBot added the dev/3.1.x-conflict label Jul 21, 2025

sollhui mentioned this pull request Jul 30, 2025
branch-3.1: [enhance](memtable) skip memtable in queue to speed up cancel #53481 #54122
Merged
16 tasks

morrySnow pushed a commit that referenced this pull request Jul 31, 2025
branch-3.1: [enhance](memtable) skip memtable in queue to speed up ca…
eb10187
…ncel #53481 (#54122) pick #53481

morrySnow added dev/3.1.0-merged and removed dev/3.1.x dev/3.1.x-conflict labels Jul 31, 2025

liaoxin01 mentioned this pull request Mar 24, 2026
[fix](be) keep FlushToken running count symmetric on cancel #61684
Merged

Uh oh!

Conversation

sollhui commented Jul 17, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Specific Example

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jul 17, 2025

Uh oh!

sollhui commented Jul 17, 2025

Uh oh!

doris-robot commented Jul 17, 2025

Uh oh!

doris-robot commented Jul 17, 2025

Uh oh!

doris-robot commented Jul 17, 2025

Uh oh!

hello-stephen commented Jul 17, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Jul 17, 2025

BE Regression && UT Coverage Report

Uh oh!

sollhui commented Jul 18, 2025

Uh oh!

doris-robot commented Jul 18, 2025

Uh oh!

doris-robot commented Jul 18, 2025

Uh oh!

doris-robot commented Jul 18, 2025

Uh oh!

sollhui commented Jul 18, 2025

Uh oh!

doris-robot commented Jul 18, 2025

Uh oh!

doris-robot commented Jul 18, 2025

Uh oh!

doris-robot commented Jul 18, 2025

Uh oh!

doris-robot commented Jul 18, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Jul 18, 2025

BE Regression && UT Coverage Report

Uh oh!

sollhui commented Jul 20, 2025

Uh oh!

sollhui commented Jul 20, 2025

Uh oh!

doris-robot commented Jul 20, 2025

Uh oh!

doris-robot commented Jul 20, 2025

Uh oh!

doris-robot commented Jul 20, 2025

Uh oh!

doris-robot commented Jul 20, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Jul 20, 2025

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Jul 21, 2025

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Jul 21, 2025

BE Regression && UT Coverage Report

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jul 21, 2025

Uh oh!

github-actionsBot commented Jul 21, 2025

Uh oh!

liaoxin01 left a comment

Choose a reason for hiding this comment

Uh oh!

sollhui commented Jul 17, 2025•
edited
Loading