[fix](iceberg) Fix some issues in batch mode by wuwenchi · Pull Request #51185 · apache/doris

wuwenchi · 2025-05-23T07:13:34Z

What problem does this PR solve?

Problem Summary:

When adding splits to the queue, a retry mechanism is added to prevent blocking and sticking.
We use a separate thread to fetch splits. When the SQL execution ends early (such as limit), this thread needs to be handled. Now it is exited by throwing an exception. Moreover, this thread should use an independent thread pool to prevent system freezes caused by resource contention.
When there are multiple exceptions, overwriting would occur previously. Now we use addSuppressed to prevent exception loss.
When initializing the split assignment, a timeout mechanism is added to prevent the situation where the system gets stuck due to failing to obtain splits.
When the split assignment stop, if an exception is not null, an exception should be thrown externally.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-05-23T07:13:39Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

wuwenchi · 2025-05-23T08:14:37Z

run buildall

doris-robot · 2025-05-23T08:55:52Z

TPC-H: Total hot run time: 33586 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 73f3c1357d65a9d3157b106cfad000e60b0771e6, data reload: false
------ Round 1 ----------------------------------
q1	26010	5099	5006	5006
q2	2083	278	178	178
q3	10410	1253	725	725
q4	10221	1011	526	526
q5	7522	2365	2276	2276
q6	174	161	131	131
q7	908	737	618	618
q8	9313	1238	1072	1072
q9	6784	5112	5048	5048
q10	6863	2289	1893	1893
q11	487	279	277	277
q12	347	344	220	220
q13	17799	3649	3065	3065
q14	231	226	218	218
q15	528	493	491	491
q16	419	430	394	394
q17	598	845	354	354
q18	7623	7251	7067	7067
q19	1732	939	554	554
q20	332	327	222	222
q21	3584	2556	2288	2288
q22	1069	1010	963	963
Total cold run time: 115037 ms
Total hot run time: 33586 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5165	5152	5111	5111
q2	243	325	233	233
q3	2158	2645	2305	2305
q4	1368	1750	1350	1350
q5	4340	4324	4347	4324
q6	227	169	129	129
q7	1979	1942	1775	1775
q8	2557	2561	2453	2453
q9	7208	7212	7013	7013
q10	3084	3188	2751	2751
q11	573	517	505	505
q12	675	756	596	596
q13	3513	3913	3263	3263
q14	287	306	303	303
q15	534	487	483	483
q16	462	498	457	457
q17	1127	1546	1385	1385
q18	7822	7623	7407	7407
q19	807	835	1055	835
q20	1973	2116	1869	1869
q21	4774	4480	4379	4379
q22	1080	1064	1028	1028
Total cold run time: 51956 ms
Total hot run time: 49954 ms

doris-robot · 2025-05-23T09:08:05Z

TPC-DS: Total hot run time: 192355 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 73f3c1357d65a9d3157b106cfad000e60b0771e6, data reload: false
query1	1438	1098	1098	1098
query2	6320	1817	1802	1802
query3	10974	4564	4423	4423
query4	53535	24336	23029	23029
query5	5065	583	478	478
query6	331	213	202	202
query7	4873	503	290	290
query8	315	258	246	246
query9	5405	2695	2661	2661
query10	426	331	279	279
query11	15053	15062	14857	14857
query12	160	109	102	102
query13	1030	552	411	411
query14	10357	6582	6522	6522
query15	212	196	189	189
query16	7141	677	476	476
query17	1100	777	605	605
query18	1573	420	345	345
query19	201	214	173	173
query20	128	127	118	118
query21	211	129	108	108
query22	4399	4464	4430	4430
query23	34354	33525	33360	33360
query24	6576	2458	2407	2407
query25	477	481	413	413
query26	721	281	154	154
query27	2306	520	351	351
query28	2999	2139	2138	2138
query29	593	560	455	455
query30	272	217	197	197
query31	892	880	789	789
query32	74	64	63	63
query33	457	360	311	311
query34	787	859	534	534
query35	827	839	742	742
query36	932	1014	870	870
query37	112	101	76	76
query38	4246	4275	4209	4209
query39	1545	1447	1456	1447
query40	219	132	111	111
query41	55	55	53	53
query42	132	126	109	109
query43	499	511	479	479
query44	1368	867	850	850
query45	187	181	165	165
query46	882	1045	664	664
query47	1822	1844	1764	1764
query48	402	436	327	327
query49	718	524	450	450
query50	653	707	422	422
query51	4219	4240	4229	4229
query52	108	110	98	98
query53	238	268	185	185
query54	584	590	513	513
query55	87	89	85	85
query56	298	320	314	314
query57	1211	1213	1137	1137
query58	270	254	259	254
query59	2664	2785	2640	2640
query60	358	335	324	324
query61	133	134	132	132
query62	734	755	714	714
query63	232	194	193	193
query64	1875	1110	699	699
query65	4340	4177	4289	4177
query66	729	406	308	308
query67	15869	15623	15348	15348
query68	7150	845	532	532
query69	541	309	276	276
query70	1264	1107	1094	1094
query71	503	332	297	297
query72	5897	4829	4814	4814
query73	1521	652	362	362
query74	8934	9213	8697	8697
query75	4002	3212	2715	2715
query76	4228	1212	791	791
query77	717	392	289	289
query78	10149	10218	9300	9300
query79	2716	815	574	574
query80	705	507	439	439
query81	486	264	219	219
query82	446	126	96	96
query83	306	247	244	244
query84	296	118	94	94
query85	791	362	319	319
query86	365	290	271	271
query87	4413	4497	4276	4276
query88	3622	2301	2302	2301
query89	399	376	278	278
query90	1882	210	206	206
query91	144	145	108	108
query92	81	68	65	65
query93	1979	967	601	601
query94	663	416	304	304
query95	373	294	322	294
query96	509	567	280	280
query97	2750	2760	2628	2628
query98	228	210	200	200
query99	1482	1387	1287	1287
Total cold run time: 298845 ms
Total hot run time: 192355 ms

doris-robot · 2025-05-23T09:13:15Z

ClickBench: Total hot run time: 28.82 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 73f3c1357d65a9d3157b106cfad000e60b0771e6, data reload: false
query1	0.04	0.04	0.04
query2	0.13	0.11	0.11
query3	0.24	0.20	0.19
query4	1.60	0.20	0.11
query5	0.43	0.42	0.41
query6	1.15	0.66	0.65
query7	0.03	0.01	0.02
query8	0.04	0.04	0.04
query9	0.59	0.52	0.52
query10	0.55	0.57	0.57
query11	0.16	0.11	0.11
query12	0.14	0.12	0.11
query13	0.62	0.60	0.60
query14	0.79	0.80	0.81
query15	0.89	0.84	0.83
query16	0.37	0.40	0.38
query17	1.06	1.02	1.05
query18	0.23	0.21	0.21
query19	1.93	1.80	1.82
query20	0.01	0.01	0.02
query21	15.43	0.89	0.52
query22	0.78	1.12	0.74
query23	14.90	1.36	0.64
query24	6.79	1.11	0.57
query25	0.55	0.12	0.07
query26	0.56	0.16	0.13
query27	0.05	0.05	0.06
query28	9.05	0.93	0.45
query29	12.56	3.99	3.35
query30	0.25	0.09	0.07
query31	2.80	0.61	0.39
query32	3.23	0.54	0.47
query33	3.02	3.04	3.06
query34	15.63	5.03	4.49
query35	4.56	4.52	4.51
query36	0.66	0.49	0.48
query37	0.08	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 102.3 s
Total hot run time: 28.82 s

wuwenchi · 2025-05-28T07:52:21Z

run buildall

doris-robot · 2025-05-28T08:23:54Z

TPC-H: Total hot run time: 34151 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 69003e5128d0fc68883a2f8d39b434934689fa2f, data reload: false
------ Round 1 ----------------------------------
q1	26121	5150	5012	5012
q2	2065	278	189	189
q3	10458	1240	702	702
q4	10233	981	530	530
q5	7788	2397	2309	2309
q6	187	161	132	132
q7	890	733	615	615
q8	9324	1318	1110	1110
q9	6850	5089	5089	5089
q10	6841	2321	1944	1944
q11	501	304	284	284
q12	346	356	218	218
q13	17779	3695	3160	3160
q14	258	233	223	223
q15	533	487	476	476
q16	431	444	376	376
q17	620	859	383	383
q18	7922	7212	7204	7204
q19	1732	959	560	560
q20	349	352	228	228
q21	3774	3244	2442	2442
q22	1075	1025	965	965
Total cold run time: 116077 ms
Total hot run time: 34151 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5213	5062	5116	5062
q2	237	325	231	231
q3	2204	2819	2321	2321
q4	1355	1777	1447	1447
q5	4565	4437	4417	4417
q6	216	167	131	131
q7	2037	1944	1781	1781
q8	2539	2689	2544	2544
q9	7188	7055	7171	7055
q10	3028	3165	2749	2749
q11	586	532	511	511
q12	692	783	616	616
q13	3607	3906	3253	3253
q14	297	296	265	265
q15	533	485	464	464
q16	456	494	431	431
q17	1167	1581	1358	1358
q18	7852	7509	7373	7373
q19	807	801	911	801
q20	1972	2016	1863	1863
q21	4842	4312	4381	4312
q22	1083	1034	998	998
Total cold run time: 52476 ms
Total hot run time: 49983 ms

doris-robot · 2025-05-28T08:35:30Z

TPC-DS: Total hot run time: 185903 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 69003e5128d0fc68883a2f8d39b434934689fa2f, data reload: false
query1	1029	500	516	500
query2	6559	1810	1790	1790
query3	6747	220	225	220
query4	26670	23752	22914	22914
query5	4337	618	460	460
query6	300	214	200	200
query7	4621	483	293	293
query8	298	257	249	249
query9	8608	2630	2632	2630
query10	497	353	275	275
query11	15362	15085	14881	14881
query12	164	114	107	107
query13	1654	548	411	411
query14	8877	6307	6332	6307
query15	210	194	168	168
query16	7194	636	483	483
query17	1209	726	599	599
query18	1982	393	332	332
query19	196	195	165	165
query20	121	122	116	116
query21	215	131	109	109
query22	4073	4265	3961	3961
query23	34170	33169	33138	33138
query24	8355	2408	2465	2408
query25	554	466	402	402
query26	1236	272	157	157
query27	2729	502	340	340
query28	4325	2114	2104	2104
query29	775	563	447	447
query30	287	217	209	209
query31	941	862	746	746
query32	76	69	66	66
query33	549	375	315	315
query34	804	872	538	538
query35	814	804	732	732
query36	947	980	900	900
query37	119	104	79	79
query38	4067	4021	4043	4021
query39	1466	1411	1407	1407
query40	211	130	114	114
query41	69	62	56	56
query42	122	111	113	111
query43	486	501	463	463
query44	1358	814	818	814
query45	182	173	166	166
query46	870	1027	650	650
query47	1754	1775	1723	1723
query48	385	413	329	329
query49	780	511	448	448
query50	668	686	417	417
query51	4183	4113	4070	4070
query52	106	112	97	97
query53	227	260	208	208
query54	584	581	510	510
query55	89	88	86	86
query56	315	295	309	295
query57	1132	1159	1080	1080
query58	282	280	271	271
query59	2494	2605	2576	2576
query60	331	320	318	318
query61	130	129	128	128
query62	813	725	682	682
query63	246	195	194	194
query64	4418	1006	689	689
query65	4302	4283	4270	4270
query66	1148	418	321	321
query67	15924	15535	15238	15238
query68	8491	889	572	572
query69	481	304	278	278
query70	1251	1081	1035	1035
query71	466	336	310	310
query72	5561	4797	4681	4681
query73	735	648	357	357
query74	9141	9156	8735	8735
query75	3916	3224	2716	2716
query76	3650	1202	787	787
query77	781	396	304	304
query78	10156	10075	9324	9324
query79	2914	823	585	585
query80	636	533	460	460
query81	488	253	226	226
query82	501	130	93	93
query83	281	328	235	235
query84	295	112	96	96
query85	806	376	315	315
query86	391	309	289	289
query87	4489	4470	4356	4356
query88	3627	2340	2294	2294
query89	405	316	314	314
query90	1846	220	222	220
query91	145	158	116	116
query92	74	68	56	56
query93	2171	968	586	586
query94	676	421	307	307
query95	365	292	291	291
query96	564	566	282	282
query97	2799	2725	2705	2705
query98	238	211	208	208
query99	1451	1398	1267	1267
Total cold run time: 276537 ms
Total hot run time: 185903 ms

doris-robot · 2025-05-28T08:40:40Z

ClickBench: Total hot run time: 28.96 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 69003e5128d0fc68883a2f8d39b434934689fa2f, data reload: false
query1	0.04	0.04	0.04
query2	0.13	0.10	0.11
query3	0.26	0.20	0.20
query4	1.60	0.20	0.20
query5	0.48	0.45	0.45
query6	1.18	0.67	0.66
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.60	0.52	0.52
query10	0.57	0.58	0.57
query11	0.16	0.10	0.11
query12	0.15	0.12	0.11
query13	0.61	0.60	0.60
query14	0.79	0.82	0.81
query15	0.88	0.85	0.86
query16	0.39	0.39	0.39
query17	1.08	1.09	1.04
query18	0.23	0.21	0.22
query19	1.90	1.78	1.82
query20	0.02	0.01	0.01
query21	15.41	0.90	0.55
query22	0.76	1.24	0.89
query23	14.75	1.40	0.65
query24	7.52	1.82	0.37
query25	0.31	0.26	0.11
query26	0.63	0.16	0.14
query27	0.05	0.04	0.05
query28	8.76	0.90	0.44
query29	12.53	3.98	3.28
query30	0.24	0.09	0.07
query31	2.82	0.61	0.39
query32	3.23	0.55	0.47
query33	3.10	3.06	3.10
query34	15.78	5.10	4.51
query35	4.54	4.58	4.49
query36	0.66	0.50	0.48
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.03	0.03
query40	0.17	0.14	0.14
query41	0.08	0.03	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.02
Total cold run time: 102.72 s
Total hot run time: 28.96 s

wuwenchi · 2025-05-29T02:47:04Z

run buildall

doris-robot · 2025-05-29T04:28:49Z

TPC-H: Total hot run time: 33752 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 78f32380563be2bb4fad78d54c41414f304fac00, data reload: false
------ Round 1 ----------------------------------
q1	26134	5091	5072	5072
q2	2088	282	191	191
q3	10512	1242	701	701
q4	10233	998	514	514
q5	7732	2336	2324	2324
q6	181	162	128	128
q7	886	737	593	593
q8	9302	1218	1032	1032
q9	6782	5098	5111	5098
q10	6820	2301	1900	1900
q11	499	286	270	270
q12	343	345	218	218
q13	17761	3621	3123	3123
q14	228	226	208	208
q15	536	480	499	480
q16	442	445	370	370
q17	589	832	378	378
q18	7808	7178	7103	7103
q19	1649	949	543	543
q20	334	348	230	230
q21	3603	3153	2326	2326
q22	1051	1016	950	950
Total cold run time: 115513 ms
Total hot run time: 33752 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5082	5053	5003	5003
q2	247	325	232	232
q3	2154	2656	2244	2244
q4	1379	1748	1443	1443
q5	4557	4403	4366	4366
q6	212	168	128	128
q7	1964	1892	1754	1754
q8	2551	2553	2455	2455
q9	7124	7096	7156	7096
q10	3017	3174	2741	2741
q11	560	522	500	500
q12	682	769	627	627
q13	3541	3856	3322	3322
q14	285	291	282	282
q15	516	483	475	475
q16	447	494	450	450
q17	1134	1504	1396	1396
q18	7721	7587	7469	7469
q19	762	762	769	762
q20	1993	2091	1849	1849
q21	4685	4290	4286	4286
q22	1063	1032	1013	1013
Total cold run time: 51676 ms
Total hot run time: 49893 ms

doris-robot · 2025-05-29T04:40:20Z

TPC-DS: Total hot run time: 185650 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 78f32380563be2bb4fad78d54c41414f304fac00, data reload: false
query1	1029	492	488	488
query2	6549	1788	1819	1788
query3	6749	224	218	218
query4	26645	23320	23062	23062
query5	4390	666	478	478
query6	306	217	194	194
query7	4622	526	304	304
query8	276	233	205	205
query9	8628	2620	2651	2620
query10	490	326	276	276
query11	15623	15155	14834	14834
query12	157	108	107	107
query13	1656	534	410	410
query14	8721	5984	6073	5984
query15	201	198	168	168
query16	7110	647	417	417
query17	939	742	551	551
query18	1955	406	285	285
query19	192	182	194	182
query20	119	123	114	114
query21	211	138	111	111
query22	4195	4187	3896	3896
query23	34111	33023	33030	33023
query24	8539	2387	2386	2386
query25	549	468	415	415
query26	1254	286	157	157
query27	2743	526	347	347
query28	4337	2137	2108	2108
query29	791	593	446	446
query30	292	219	191	191
query31	924	883	776	776
query32	74	73	65	65
query33	576	374	320	320
query34	818	920	554	554
query35	809	850	728	728
query36	1002	1007	883	883
query37	127	106	83	83
query38	4023	4175	4035	4035
query39	1483	1442	1400	1400
query40	222	130	122	122
query41	73	64	92	64
query42	123	112	117	112
query43	490	517	491	491
query44	1356	823	825	823
query45	194	175	168	168
query46	904	1088	641	641
query47	1756	1796	1759	1759
query48	399	437	307	307
query49	750	522	404	404
query50	667	725	413	413
query51	4151	4193	4129	4129
query52	109	111	101	101
query53	231	279	183	183
query54	587	581	508	508
query55	89	88	82	82
query56	301	312	326	312
query57	1148	1137	1083	1083
query58	265	255	250	250
query59	2688	2638	2586	2586
query60	339	320	304	304
query61	131	144	131	131
query62	811	776	677	677
query63	238	196	186	186
query64	4369	987	669	669
query65	4380	4201	4208	4201
query66	1144	458	316	316
query67	15885	15406	15352	15352
query68	7791	909	578	578
query69	482	313	273	273
query70	1248	1120	1083	1083
query71	438	323	296	296
query72	5697	4765	4876	4765
query73	684	654	362	362
query74	8955	9056	9131	9056
query75	3402	3310	2669	2669
query76	3240	1304	758	758
query77	536	386	295	295
query78	10015	10168	9398	9398
query79	1458	862	586	586
query80	601	527	448	448
query81	489	266	222	222
query82	206	131	100	100
query83	255	256	240	240
query84	295	109	91	91
query85	756	363	309	309
query86	377	334	277	277
query87	4362	4495	4277	4277
query88	3004	2317	2305	2305
query89	420	341	283	283
query90	1977	213	205	205
query91	144	143	121	121
query92	75	65	62	62
query93	1600	976	593	593
query94	676	433	307	307
query95	377	296	278	278
query96	507	620	288	288
query97	2726	2825	2666	2666
query98	246	210	207	207
query99	1396	1483	1269	1269
Total cold run time: 271450 ms
Total hot run time: 185650 ms

doris-robot · 2025-05-29T04:45:33Z

ClickBench: Total hot run time: 29.04 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 78f32380563be2bb4fad78d54c41414f304fac00, data reload: false
query1	0.04	0.04	0.03
query2	0.12	0.11	0.11
query3	0.26	0.19	0.19
query4	1.60	0.19	0.12
query5	0.43	0.43	0.41
query6	1.16	0.65	0.66
query7	0.03	0.01	0.02
query8	0.04	0.04	0.03
query9	0.59	0.51	0.51
query10	0.55	0.59	0.57
query11	0.15	0.11	0.10
query12	0.15	0.11	0.12
query13	0.62	0.60	0.60
query14	0.79	0.81	0.81
query15	0.87	0.85	0.85
query16	0.39	0.39	0.38
query17	1.03	1.03	1.01
query18	0.23	0.21	0.21
query19	1.92	1.79	1.84
query20	0.01	0.01	0.02
query21	15.40	0.90	0.58
query22	0.78	1.16	0.77
query23	14.78	1.41	0.64
query24	6.87	1.60	0.65
query25	0.51	0.12	0.27
query26	0.73	0.18	0.14
query27	0.04	0.06	0.05
query28	9.25	0.90	0.47
query29	12.61	4.00	3.34
query30	0.25	0.10	0.06
query31	2.83	0.59	0.39
query32	3.23	0.55	0.46
query33	3.03	3.09	3.09
query34	15.84	5.08	4.50
query35	4.53	4.50	4.50
query36	0.66	0.50	0.47
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.02	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 102.79 s
Total hot run time: 29.04 s

morningman · 2025-05-29T09:45:01Z

- splitAssignment.addToQueue(Lists.newArrayList(createIcebergSplit(fileScanTask))));
-
- return null;
+ taskRef.set(planFileScanTask(scan));


You call planFileScanTask() twice?

BTW, the threadPoolWithPreAuth in ExternalCatalog.java is only used in

morningman · 2025-05-29T10:19:57Z

+ } catch (InterruptedException e) {
+ throw new UserException("Failed to offer batch split by interrupted", e);
+ }
+ if (needMoreSplit()) {


This logic is wrong

wuwenchi · 2025-05-30T07:57:16Z

run buildall

morningman · 2025-05-30T09:13:25Z

run buildall

doris-robot · 2025-05-30T10:41:31Z

TPC-H: Total hot run time: 33647 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3784bd9f23885ca49a79fe2b8d6801692747bd4d, data reload: false
------ Round 1 ----------------------------------
q1	26380	5100	4955	4955
q2	1960	272	172	172
q3	10313	1234	724	724
q4	10225	1000	519	519
q5	7508	2396	2323	2323
q6	179	168	132	132
q7	909	734	619	619
q8	9311	1242	1107	1107
q9	6738	5075	5020	5020
q10	6855	2310	1871	1871
q11	483	292	272	272
q12	342	356	220	220
q13	17788	3718	3128	3128
q14	251	231	221	221
q15	561	486	500	486
q16	428	438	384	384
q17	589	849	351	351
q18	7548	7156	6983	6983
q19	1926	964	571	571
q20	324	335	221	221
q21	3767	2562	2367	2367
q22	1054	1010	1001	1001
Total cold run time: 115439 ms
Total hot run time: 33647 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5193	5030	5073	5030
q2	238	308	222	222
q3	2168	2655	2328	2328
q4	1377	1779	1292	1292
q5	4431	4420	4391	4391
q6	224	167	131	131
q7	2008	1960	1854	1854
q8	2603	2545	2524	2524
q9	7168	7225	6875	6875
q10	2991	3221	2809	2809
q11	576	516	514	514
q12	700	790	599	599
q13	3585	3984	3377	3377
q14	280	301	272	272
q15	524	499	488	488
q16	475	504	435	435
q17	1125	1491	1397	1397
q18	7704	7546	7422	7422
q19	842	854	883	854
q20	2024	2067	1927	1927
q21	4718	4543	4520	4520
q22	1086	1057	1026	1026
Total cold run time: 52040 ms
Total hot run time: 50287 ms

doris-robot · 2025-05-30T10:53:39Z

TPC-DS: Total hot run time: 191375 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3784bd9f23885ca49a79fe2b8d6801692747bd4d, data reload: false
query1	1421	1118	1052	1052
query2	6099	1795	1762	1762
query3	11022	4434	4478	4434
query4	54109	25827	22913	22913
query5	5170	516	485	485
query6	373	209	213	209
query7	5023	516	287	287
query8	318	233	213	213
query9	6183	2676	2697	2676
query10	426	341	279	279
query11	15153	15035	14885	14885
query12	158	113	115	113
query13	1114	517	410	410
query14	10229	6305	6385	6305
query15	192	207	183	183
query16	7047	624	472	472
query17	1082	714	595	595
query18	1555	412	312	312
query19	202	196	186	186
query20	131	122	124	122
query21	206	123	103	103
query22	4301	4369	4218	4218
query23	34257	33241	33640	33241
query24	6738	2448	2462	2448
query25	487	492	409	409
query26	694	269	154	154
query27	2294	523	344	344
query28	3245	2163	2166	2163
query29	589	563	426	426
query30	274	218	198	198
query31	839	866	787	787
query32	75	64	64	64
query33	497	358	309	309
query34	776	866	551	551
query35	785	847	793	793
query36	956	975	883	883
query37	118	105	78	78
query38	4277	4324	4160	4160
query39	1533	1439	1455	1439
query40	213	118	112	112
query41	62	57	56	56
query42	136	116	106	106
query43	506	509	486	486
query44	1390	850	850	850
query45	195	187	178	178
query46	868	1051	648	648
query47	1873	1811	1773	1773
query48	387	428	346	346
query49	655	493	410	410
query50	652	703	416	416
query51	4191	4251	4240	4240
query52	113	114	98	98
query53	241	267	191	191
query54	591	590	523	523
query55	82	83	87	83
query56	318	304	301	301
query57	1157	1188	1097	1097
query58	264	268	258	258
query59	2696	2745	2671	2671
query60	335	341	312	312
query61	142	128	125	125
query62	749	716	672	672
query63	229	185	183	183
query64	1446	1000	669	669
query65	4207	4141	4148	4141
query66	709	399	296	296
query67	16002	15479	15458	15458
query68	5480	848	548	548
query69	529	299	331	299
query70	1220	1134	1055	1055
query71	427	317	295	295
query72	5709	4775	4851	4775
query73	1209	689	350	350
query74	9133	8921	8657	8657
query75	3331	3244	2699	2699
query76	3889	1186	731	731
query77	555	368	276	276
query78	9947	10046	9215	9215
query79	1952	823	585	585
query80	643	512	447	447
query81	494	276	214	214
query82	354	123	98	98
query83	262	247	236	236
query84	291	109	95	95
query85	778	406	316	316
query86	340	313	308	308
query87	4317	4393	4283	4283
query88	3297	2295	2310	2295
query89	401	310	279	279
query90	1827	212	212	212
query91	138	142	113	113
query92	72	68	58	58
query93	2655	943	578	578
query94	672	419	309	309
query95	363	295	289	289
query96	495	572	283	283
query97	2686	2770	2633	2633
query98	239	205	203	203
query99	1320	1377	1281	1281
Total cold run time: 295382 ms
Total hot run time: 191375 ms

doris-robot · 2025-05-30T10:58:51Z

ClickBench: Total hot run time: 29.01 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3784bd9f23885ca49a79fe2b8d6801692747bd4d, data reload: false
query1	0.04	0.03	0.03
query2	0.12	0.10	0.11
query3	0.25	0.19	0.19
query4	1.59	0.19	0.11
query5	0.46	0.42	0.42
query6	1.17	0.67	0.65
query7	0.03	0.02	0.01
query8	0.04	0.04	0.03
query9	0.58	0.53	0.51
query10	0.57	0.60	0.57
query11	0.15	0.11	0.10
query12	0.15	0.11	0.11
query13	0.61	0.60	0.59
query14	0.78	0.81	0.83
query15	0.87	0.87	0.85
query16	0.39	0.39	0.39
query17	1.06	1.03	1.07
query18	0.23	0.21	0.21
query19	1.90	1.82	1.85
query20	0.01	0.00	0.01
query21	15.39	0.89	0.56
query22	0.75	1.19	0.65
query23	14.99	1.39	0.65
query24	7.48	0.96	0.89
query25	0.52	0.26	0.09
query26	0.67	0.16	0.13
query27	0.04	0.05	0.05
query28	9.27	0.92	0.45
query29	12.55	4.08	3.35
query30	0.25	0.09	0.06
query31	2.82	0.59	0.38
query32	3.23	0.55	0.47
query33	3.07	3.02	3.11
query34	15.79	5.19	4.46
query35	4.50	4.56	4.48
query36	0.68	0.49	0.48
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.15	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 103.51 s
Total hot run time: 29.01 s

morningman

LGTM

github-actions · 2025-06-01T14:36:08Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-06-01T14:36:10Z

PR approved by anyone and no changes requested.

### What problem does this PR solve? Problem Summary: 1. When adding splits to the queue, a retry mechanism is added to prevent blocking and sticking. 2. We use a separate thread to fetch splits. When the SQL execution ends early (such as `limit`), this thread needs to be handled. Now it is exited by throwing an exception. Moreover, this thread should use an independent thread pool to prevent system freezes caused by resource contention. 3. When there are multiple exceptions, overwriting would occur previously. Now we use `addSuppressed` to prevent exception loss. 4. When initializing the split assignment, a timeout mechanism is added to prevent the situation where the system gets stuck due to failing to obtain splits. 5. When the split assignment stop, if an exception is not null, an exception should be thrown externally.

### What problem does this PR solve? Problem Summary: 1. When adding splits to the queue, a retry mechanism is added to prevent blocking and sticking. 2. We use a separate thread to fetch splits. When the SQL execution ends early (such as `limit`), this thread needs to be handled. Now it is exited by throwing an exception. Moreover, this thread should use an independent thread pool to prevent system freezes caused by resource contention. 3. When there are multiple exceptions, overwriting would occur previously. Now we use `addSuppressed` to prevent exception loss. 4. When initializing the split assignment, a timeout mechanism is added to prevent the situation where the system gets stuck due to failing to obtain splits. 5. When the split assignment stop, if an exception is not null, an exception should be thrown externally. (cherry picked from commit f6d6041)

This reverts commit 4e7a1c6.

…51185)"" This reverts commit fb1ef62.

…tables in multi-BE and batch mode. (#51694) ### What problem does this PR solve? Related PR: #51185 Problem Summary: In pr #51185, in batch mode + multiple BE scenarios, when allocating splits, if a BE is successfully allocated, it will exit the current allocation, resulting in missing splits and missing data in query results.

Problem Summary: 1. When adding splits to the queue, a retry mechanism is added to prevent blocking and sticking. 2. We use a separate thread to fetch splits. When the SQL execution ends early (such as `limit`), this thread needs to be handled. Now it is exited by throwing an exception. Moreover, this thread should use an independent thread pool to prevent system freezes caused by resource contention. 3. When there are multiple exceptions, overwriting would occur previously. Now we use `addSuppressed` to prevent exception loss. 4. When initializing the split assignment, a timeout mechanism is added to prevent the situation where the system gets stuck due to failing to obtain splits. 5. When the split assignment stop, if an exception is not null, an exception should be thrown externally.

…tables in multi-BE and batch mode. (apache#51694) ### What problem does this PR solve? Related PR: apache#51185 Problem Summary: In pr apache#51185, in batch mode + multiple BE scenarios, when allocating splits, if a BE is successfully allocated, it will exit the current allocation, resulting in missing splits and missing data in query results.

…52095) bp #46398 #49025 #49489 #50434 #51185 #51694 --------- Co-authored-by: wuwenchi <wuwenchi@selectdb.com> Co-authored-by: daidai <changyuwei@selectdb.com>

wuwenchi added 4 commits May 28, 2025 14:59

fix
0746800

fix
f52f050

fix
fecfb7c

fix
023d271

wuwenchi force-pushed the fix-batch-leak branch from 73f3c13 to 023d271Compare May 28, 2025 06:59

fix
69003e5

fix
78f3238

wuwenchi marked this pull request as ready for review May 29, 2025 02:47

fix some issue
76a0596

morningman reviewed May 29, 2025
View reviewed changes

morningman added 2 commits May 29, 2025 23:08

fix
44df17a

1
86f8ea0

fix ut
3784bd9

morningman added dev/3.0.x-experimental dev/3.1.x labels Jun 1, 2025

morningman approved these changes Jun 1, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jun 1, 2025

github-actionsBot added the reviewed label Jun 1, 2025

CalvinKirs approved these changes Jun 1, 2025
View reviewed changes

morningman merged commit f6d6041 into apache:masterJun 2, 2025

wuwenchi mentioned this pull request Jun 5, 2025
[opt](iceberg)support iceberg in batch mode for 3.0 #51515
Closed

hubgeter added a commit to hubgeter/doris that referenced this pull request Jun 13, 2025
Revert "[fix](iceberg) Fix some issues in batch mode (apache#51185)"
fb1ef62
This reverts commit 4e7a1c6.

hubgeter added a commit to hubgeter/doris that referenced this pull request Jun 13, 2025
Revert "Revert "[fix](iceberg) Fix some issues in batch mode (apache#…
9b3982f
…51185)"" This reverts commit fb1ef62.

hubgeter mentioned this pull request Jun 13, 2025
[fix](catalog)Fixed the issue of missing data when querying external tables in multi-BE and batch mode. #51694
Merged
16 tasks

This was referenced Jun 20, 2025
branch-3.1: [feat](iceberg) implement iceberg partition batch mode #52094
Closed
branch-3.1: [feat](iceberg) implement iceberg partition batch mode #52095
Merged

morrySnow added dev/3.1.0-merged and removed dev/3.1.x labels Jun 21, 2025

morningman removed the dev/3.0.x-experimental label Jul 4, 2025

Uh oh!

Conversation

wuwenchi commented May 23, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented May 23, 2025

Uh oh!

wuwenchi commented May 23, 2025

Uh oh!

doris-robot commented May 23, 2025

Uh oh!

doris-robot commented May 23, 2025

Uh oh!

doris-robot commented May 23, 2025

Uh oh!

wuwenchi commented May 28, 2025

Uh oh!

doris-robot commented May 28, 2025

Uh oh!

doris-robot commented May 28, 2025

Uh oh!

doris-robot commented May 28, 2025

Uh oh!

wuwenchi commented May 29, 2025

Uh oh!

doris-robot commented May 29, 2025

Uh oh!

doris-robot commented May 29, 2025

Uh oh!

doris-robot commented May 29, 2025

Uh oh!

morningmanMay 29, 2025

Choose a reason for hiding this comment

Uh oh!

morningmanMay 29, 2025

Choose a reason for hiding this comment

Uh oh!

morningmanMay 29, 2025

Choose a reason for hiding this comment

Uh oh!

wuwenchi commented May 30, 2025

Uh oh!

morningman commented May 30, 2025

Uh oh!

doris-robot commented May 30, 2025

Uh oh!

doris-robot commented May 30, 2025

Uh oh!

doris-robot commented May 30, 2025

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jun 1, 2025

Uh oh!

github-actionsBot commented Jun 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

wuwenchi commented May 23, 2025•
edited
Loading