[Opt] (multi-catalog) opt max scanner thread number in batch split mode. by kaka11chen · Pull Request #44635 · apache/doris

kaka11chen · 2024-11-26T12:22:37Z

What problem does this PR solve?

Problem Summary:

There's only one scan range for each backend in batch split mode. Each backend only starts up one ScanNode instance. However, when calculating the concurrency of scanners in the scan operator in batch split mode, it is not divided by 1, but by query_parallel_instance_num, resulting in poor performance of batch split mode.

Release note

There's only one scan range for each backend in batch split mode. Each backend only starts up one ScanNode instance. However, when calculating the concurrency of scanners in the scan operator in batch split mode, it is not divided by 1, but by query_parallel_instance_num, resulting in poor performance of batch split mode.

Fix calculation of maximum number of scanner threads in batch split mode.

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
1. Create a hive partition table which contains many hive partitions(> 1024);
2. Execute select * from par_table in doris.
3. Open the profile to view the number of file scan operators and the number of MaxScannerThreadNum.
  You should be able to see that each be node of the scan operator is 1, and there will be more MaxScannerThreadNum in the scan operator.
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-11-26T12:22:43Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

kaka11chen · 2024-11-26T12:23:06Z

run buildall

kaka11chen · 2024-11-26T12:58:10Z

run buildall

doris-robot · 2024-11-26T13:23:48Z

TeamCity be ut coverage result:
Function Coverage: 38.35% (9976/26014)
Line Coverage: 29.45% (83484/283518)
Region Coverage: 28.59% (42975/150295)
Branch Coverage: 25.20% (21829/86630)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b8c32972e9764c04ff0509ab9dfc209ce78fda75_b8c32972e9764c04ff0509ab9dfc209ce78fda75/report/index.html

doris-robot · 2024-11-26T13:35:07Z

TPC-H: Total hot run time: 39878 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b8c32972e9764c04ff0509ab9dfc209ce78fda75, data reload: false
------ Round 1 ----------------------------------
q1	17593	7436	7196	7196
q2	2038	174	177	174
q3	10593	1072	1170	1072
q4	10503	705	740	705
q5	7601	2690	2720	2690
q6	246	149	144	144
q7	968	607	589	589
q8	9511	1860	1957	1860
q9	6536	6453	6447	6447
q10	7030	2307	2294	2294
q11	479	258	255	255
q12	412	223	216	216
q13	18800	3035	3000	3000
q14	258	213	210	210
q15	578	541	529	529
q16	671	585	577	577
q17	976	552	649	552
q18	7318	6816	6761	6761
q19	1332	1008	996	996
q20	469	180	176	176
q21	4006	3384	3133	3133
q22	369	316	302	302
Total cold run time: 108287 ms
Total hot run time: 39878 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7318	7285	7295	7285
q2	336	248	238	238
q3	3095	3041	3151	3041
q4	2123	1923	1862	1862
q5	5764	5818	5913	5818
q6	230	146	150	146
q7	2354	1826	1778	1778
q8	3411	3577	3539	3539
q9	9076	8912	8870	8870
q10	3569	3531	3563	3531
q11	619	519	511	511
q12	851	634	666	634
q13	16659	3287	3212	3212
q14	308	268	276	268
q15	579	538	528	528
q16	683	655	656	655
q17	1861	1639	1610	1610
q18	8316	7834	7615	7615
q19	1818	1477	1500	1477
q20	2119	1867	1863	1863
q21	5576	5383	5239	5239
q22	632	557	562	557
Total cold run time: 77297 ms
Total hot run time: 60277 ms

doris-robot · 2024-11-26T13:47:46Z

TPC-DS: Total hot run time: 196384 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b8c32972e9764c04ff0509ab9dfc209ce78fda75, data reload: false
query1	1254	953	935	935
query2	6257	2026	2039	2026
query3	10772	3937	3866	3866
query4	67495	29897	23547	23547
query5	4980	468	465	465
query6	407	177	186	177
query7	5638	301	280	280
query8	312	216	209	209
query9	9171	2613	2644	2613
query10	449	248	247	247
query11	17437	15307	15820	15307
query12	153	101	110	101
query13	1574	441	447	441
query14	10920	7318	7224	7224
query15	233	198	195	195
query16	7215	455	487	455
query17	1310	586	591	586
query18	1844	320	312	312
query19	217	161	158	158
query20	125	114	110	110
query21	209	103	102	102
query22	4832	4707	4455	4455
query23	34814	34372	34493	34372
query24	5519	2532	2479	2479
query25	492	383	381	381
query26	647	144	144	144
query27	1895	273	301	273
query28	4475	2476	2485	2476
query29	682	443	411	411
query30	219	174	150	150
query31	986	818	855	818
query32	70	58	57	57
query33	438	289	288	288
query34	929	511	512	511
query35	855	748	724	724
query36	1099	960	969	960
query37	118	75	75	75
query38	4536	4335	4375	4335
query39	1497	1478	1491	1478
query40	198	99	98	98
query41	44	44	42	42
query42	112	102	101	101
query43	539	486	513	486
query44	1176	823	821	821
query45	183	166	167	166
query46	1145	731	695	695
query47	2077	1941	1904	1904
query48	427	320	320	320
query49	730	393	421	393
query50	847	394	393	393
query51	7376	7282	7056	7056
query52	96	85	91	85
query53	254	181	172	172
query54	505	405	387	387
query55	79	78	79	78
query56	244	249	250	249
query57	1291	1206	1167	1167
query58	241	212	211	211
query59	3217	2989	3155	2989
query60	285	241	285	241
query61	113	113	126	113
query62	803	675	663	663
query63	205	189	184	184
query64	1350	659	636	636
query65	3289	3213	3212	3212
query66	697	296	298	296
query67	16017	15785	15664	15664
query68	4048	576	558	558
query69	443	249	249	249
query70	1199	1144	1133	1133
query71	352	242	247	242
query72	6383	4129	4044	4044
query73	750	366	355	355
query74	10178	9014	8994	8994
query75	3391	2691	2627	2627
query76	1771	997	1080	997
query77	478	291	280	280
query78	10420	9353	9389	9353
query79	1521	602	644	602
query80	879	442	431	431
query81	500	224	236	224
query82	1280	118	118	118
query83	173	149	154	149
query84	283	68	69	68
query85	868	299	295	295
query86	330	309	299	299
query87	4658	4567	4550	4550
query88	3416	2188	2166	2166
query89	424	282	303	282
query90	2009	180	183	180
query91	132	102	104	102
query92	61	51	50	50
query93	1944	535	534	534
query94	767	298	287	287
query95	346	246	242	242
query96	601	278	270	270
query97	2869	2679	2685	2679
query98	217	199	191	191
query99	1795	1304	1268	1268
Total cold run time: 320101 ms
Total hot run time: 196384 ms

doris-robot · 2024-11-26T13:53:15Z

ClickBench: Total hot run time: 32.26 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b8c32972e9764c04ff0509ab9dfc209ce78fda75, data reload: false
query1	0.03	0.03	0.04
query2	0.06	0.04	0.03
query3	0.23	0.08	0.08
query4	1.62	0.11	0.10
query5	0.43	0.42	0.42
query6	1.14	0.66	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.51	0.50
query10	0.56	0.57	0.56
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.61	0.60
query14	2.73	2.74	2.73
query15	0.91	0.82	0.83
query16	0.38	0.38	0.38
query17	1.03	1.09	1.07
query18	0.22	0.21	0.21
query19	1.96	1.88	1.98
query20	0.01	0.01	0.01
query21	15.36	0.58	0.58
query22	2.48	2.03	1.53
query23	17.35	0.77	0.86
query24	2.93	0.77	1.47
query25	0.16	0.22	0.13
query26	0.48	0.14	0.13
query27	0.04	0.05	0.04
query28	10.62	1.09	1.07
query29	12.63	3.31	3.27
query30	0.25	0.07	0.07
query31	2.87	0.38	0.38
query32	3.29	0.47	0.48
query33	2.98	3.02	3.12
query34	16.83	4.43	4.52
query35	4.47	4.50	4.47
query36	0.66	0.49	0.48
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.71 s
Total hot run time: 32.26 s

github-actions · 2024-12-19T18:14:17Z

clang-tidy review says "All clean, LGTM! 👍"

kaka11chen · 2024-12-30T12:30:00Z

run buildall

doris-robot · 2024-12-30T13:25:41Z

TPC-H: Total hot run time: 32892 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a7db9f035e5a35892662493b25356fced4786461, data reload: false
------ Round 1 ----------------------------------
q1	17597	6181	6065	6065
q2	2050	335	176	176
q3	10375	1254	799	799
q4	10275	903	463	463
q5	8646	2222	2022	2022
q6	211	180	149	149
q7	893	773	611	611
q8	9251	1411	1204	1204
q9	5282	4953	5036	4953
q10	6801	2332	1851	1851
q11	495	287	271	271
q12	355	370	236	236
q13	17802	3606	2917	2917
q14	241	226	218	218
q15	556	511	514	511
q16	646	631	573	573
q17	569	860	328	328
q18	6947	6555	6315	6315
q19	2029	975	599	599
q20	321	325	192	192
q21	2889	2187	2121	2121
q22	361	352	318	318
Total cold run time: 104592 ms
Total hot run time: 32892 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6332	6191	6223	6191
q2	238	327	230	230
q3	2285	2703	2321	2321
q4	1491	1860	1400	1400
q5	4404	4799	4942	4799
q6	185	181	141	141
q7	2088	1999	1834	1834
q8	2652	2806	2707	2707
q9	7326	7300	7245	7245
q10	3066	3398	2783	2783
q11	574	508	494	494
q12	645	777	633	633
q13	3396	3750	3154	3154
q14	288	304	300	300
q15	571	504	510	504
q16	667	685	647	647
q17	1255	1728	1258	1258
q18	7659	7650	7337	7337
q19	923	1140	1132	1132
q20	2040	2065	1948	1948
q21	5712	5386	4977	4977
q22	639	622	571	571
Total cold run time: 54436 ms
Total hot run time: 52606 ms

doris-robot · 2024-12-30T13:38:01Z

TPC-DS: Total hot run time: 197373 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a7db9f035e5a35892662493b25356fced4786461, data reload: false
query1	1319	1011	927	927
query2	6345	2501	2384	2384
query3	10968	4806	4683	4683
query4	32552	23559	23430	23430
query5	4169	620	443	443
query6	281	204	204	204
query7	3984	481	300	300
query8	341	251	228	228
query9	9483	2728	2728	2728
query10	441	305	244	244
query11	17569	15294	15128	15128
query12	152	110	105	105
query13	1593	533	423	423
query14	9280	6786	7693	6786
query15	255	208	200	200
query16	8156	649	503	503
query17	1572	788	592	592
query18	2155	418	325	325
query19	210	191	172	172
query20	139	116	118	116
query21	208	126	111	111
query22	4831	4939	4701	4701
query23	34399	33908	34346	33908
query24	6384	2364	2459	2364
query25	487	455	389	389
query26	808	273	154	154
query27	2066	461	337	337
query28	5729	2532	2502	2502
query29	681	563	427	427
query30	209	191	149	149
query31	1032	883	866	866
query32	88	63	54	54
query33	465	361	299	299
query34	788	855	533	533
query35	823	812	745	745
query36	1010	1091	985	985
query37	118	110	78	78
query38	4266	4302	4227	4227
query39	1679	1479	1466	1466
query40	210	116	112	112
query41	46	47	44	44
query42	118	100	104	100
query43	553	537	500	500
query44	1330	818	823	818
query45	194	184	178	178
query46	894	1061	678	678
query47	2044	2071	1965	1965
query48	385	441	323	323
query49	715	483	417	417
query50	659	696	400	400
query51	7231	7194	7247	7194
query52	96	103	99	99
query53	241	262	190	190
query54	503	507	419	419
query55	84	83	77	77
query56	258	260	241	241
query57	1259	1229	1190	1190
query58	241	233	228	228
query59	3132	3418	3344	3344
query60	273	266	249	249
query61	111	111	111	111
query62	893	817	754	754
query63	243	199	196	196
query64	3757	1073	662	662
query65	3324	3281	3265	3265
query66	795	411	310	310
query67	16359	15915	15726	15726
query68	10042	767	515	515
query69	505	293	250	250
query70	1232	1188	1198	1188
query71	445	328	252	252
query72	5911	3832	3792	3792
query73	1160	761	367	367
query74	9714	8959	8712	8712
query75	4619	3161	2645	2645
query76	5627	1203	765	765
query77	1026	377	294	294
query78	10050	10362	9493	9493
query79	2699	895	602	602
query80	671	513	419	419
query81	480	267	232	232
query82	402	149	117	117
query83	187	158	146	146
query84	275	91	72	72
query85	736	447	303	303
query86	350	326	302	302
query87	4739	4470	4308	4308
query88	3191	2248	2243	2243
query89	425	330	300	300
query90	1975	192	189	189
query91	131	139	106	106
query92	69	55	51	51
query93	1231	880	542	542
query94	652	376	292	292
query95	330	336	248	248
query96	488	617	280	280
query97	2775	2825	2692	2692
query98	225	205	206	205
query99	1622	1567	1445	1445
Total cold run time: 296967 ms
Total hot run time: 197373 ms

doris-robot · 2024-12-30T13:43:20Z

ClickBench: Total hot run time: 31.22 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a7db9f035e5a35892662493b25356fced4786461, data reload: false
query1	0.04	0.04	0.04
query2	0.07	0.03	0.04
query3	0.23	0.07	0.06
query4	1.63	0.10	0.11
query5	0.41	0.42	0.40
query6	1.19	0.66	0.66
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.60	0.50	0.50
query10	0.56	0.57	0.54
query11	0.15	0.10	0.10
query12	0.15	0.11	0.11
query13	0.62	0.61	0.60
query14	2.84	2.74	2.88
query15	0.91	0.84	0.84
query16	0.37	0.38	0.40
query17	1.00	0.99	0.98
query18	0.22	0.21	0.21
query19	1.89	1.75	1.99
query20	0.02	0.01	0.01
query21	15.38	0.97	0.61
query22	0.76	0.72	0.69
query23	15.29	1.42	0.56
query24	2.52	1.07	0.93
query25	0.29	0.08	0.12
query26	0.28	0.14	0.13
query27	0.05	0.04	0.05
query28	13.92	1.53	1.05
query29	12.57	3.90	3.27
query30	0.25	0.09	0.07
query31	2.83	0.60	0.39
query32	3.22	0.54	0.46
query33	3.15	3.09	3.18
query34	16.83	5.07	4.49
query35	4.54	4.48	4.47
query36	0.68	0.49	0.48
query37	0.09	0.06	0.07
query38	0.05	0.03	0.03
query39	0.03	0.03	0.02
query40	0.16	0.14	0.14
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.99 s
Total hot run time: 31.22 s

doris-robot · 2024-12-30T14:27:38Z

TeamCity be ut coverage result:
Function Coverage: 38.89% (10122/26025)
Line Coverage: 29.88% (85505/286174)
Region Coverage: 29.02% (43721/150683)
Branch Coverage: 25.54% (22298/87308)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a7db9f035e5a35892662493b25356fced4786461_a7db9f035e5a35892662493b25356fced4786461/report/index.html

kaka11chen · 2025-01-16T03:17:07Z

run buildall

doris-robot · 2025-01-16T03:42:35Z

TPC-H: Total hot run time: 32422 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b612a1c1887a13372901cc82d7193070494f0335, data reload: false
------ Round 1 ----------------------------------
q1	12899	5523	5377	5377
q2	1853	314	181	181
q3	10055	1315	762	762
q4	9952	989	534	534
q5	4226	2448	2157	2157
q6	192	173	137	137
q7	893	757	629	629
q8	5101	1384	1190	1190
q9	5076	4960	4902	4902
q10	4868	2334	1917	1917
q11	407	283	259	259
q12	333	369	220	220
q13	3407	3710	3095	3095
q14	236	226	208	208
q15	527	485	463	463
q16	648	630	587	587
q17	559	862	331	331
q18	6767	6417	6415	6415
q19	626	954	545	545
q20	315	332	196	196
q21	2860	2187	1994	1994
q22	374	331	323	323
Total cold run time: 72174 ms
Total hot run time: 32422 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5503	5495	5499	5495
q2	239	342	237	237
q3	2327	2585	2343	2343
q4	1413	1840	1362	1362
q5	4354	4769	4686	4686
q6	165	159	127	127
q7	1980	1877	1763	1763
q8	2535	2710	2651	2651
q9	7026	6934	6897	6897
q10	2853	3116	2631	2631
q11	589	547	497	497
q12	656	726	579	579
q13	3310	3687	3106	3106
q14	264	282	262	262
q15	507	468	473	468
q16	668	678	640	640
q17	1210	1690	1230	1230
q18	7485	7122	7099	7099
q19	793	1084	1080	1080
q20	1955	1925	1847	1847
q21	5513	5070	4925	4925
q22	634	611	561	561
Total cold run time: 51979 ms
Total hot run time: 50486 ms

doris-robot · 2025-01-16T03:54:21Z

TPC-DS: Total hot run time: 188040 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b612a1c1887a13372901cc82d7193070494f0335, data reload: false
query1	968	379	378	378
query2	6534	1958	1989	1958
query3	6797	223	224	223
query4	33354	23242	23368	23242
query5	4359	615	454	454
query6	279	186	191	186
query7	4605	506	315	315
query8	290	223	217	217
query9	9385	2667	2681	2667
query10	475	324	261	261
query11	17818	15132	14940	14940
query12	155	112	110	110
query13	1680	543	415	415
query14	10407	7052	6849	6849
query15	245	206	191	191
query16	7368	633	464	464
query17	1621	757	587	587
query18	2030	395	303	303
query19	214	194	165	165
query20	126	158	108	108
query21	209	121	101	101
query22	4107	4447	4282	4282
query23	34048	33090	32856	32856
query24	6493	2248	2316	2248
query25	506	457	426	426
query26	779	276	156	156
query27	2018	482	329	329
query28	5278	2476	2429	2429
query29	634	580	428	428
query30	224	184	158	158
query31	946	872	799	799
query32	98	60	68	60
query33	505	358	316	316
query34	766	868	503	503
query35	810	822	730	730
query36	1018	1042	950	950
query37	122	105	81	81
query38	4168	4309	4093	4093
query39	1437	1412	1589	1412
query40	200	114	101	101
query41	66	51	50	50
query42	125	104	102	102
query43	522	513	490	490
query44	1419	819	823	819
query45	182	170	165	165
query46	915	1088	647	647
query47	1824	1800	1774	1774
query48	379	414	348	348
query49	762	489	406	406
query50	661	699	421	421
query51	6948	6889	6905	6889
query52	102	106	95	95
query53	231	269	192	192
query54	490	498	419	419
query55	82	82	86	82
query56	261	258	245	245
query57	1161	1149	1073	1073
query58	248	234	249	234
query59	2812	3011	2824	2824
query60	276	279	249	249
query61	140	118	121	118
query62	799	735	679	679
query63	229	197	192	192
query64	3870	1028	669	669
query65	3240	3130	3136	3130
query66	895	415	357	357
query67	15996	15642	15639	15639
query68	5480	846	525	525
query69	488	299	261	261
query70	1215	1169	1115	1115
query71	374	292	259	259
query72	5842	3843	3823	3823
query73	649	780	371	371
query74	9933	9032	8736	8736
query75	3206	3203	2726	2726
query76	3215	1193	789	789
query77	503	380	284	284
query78	10046	10072	9322	9322
query79	2740	828	618	618
query80	709	571	453	453
query81	494	294	248	248
query82	219	157	122	122
query83	168	182	151	151
query84	248	98	74	74
query85	756	356	374	356
query86	391	295	269	269
query87	4338	4442	4336	4336
query88	4830	2204	2175	2175
query89	403	331	298	298
query90	1955	197	201	197
query91	140	141	110	110
query92	73	57	56	56
query93	2336	881	531	531
query94	670	422	277	277
query95	347	280	264	264
query96	492	643	285	285
query97	2786	2903	2763	2763
query98	234	204	200	200
query99	1275	1407	1247	1247
Total cold run time: 284266 ms
Total hot run time: 188040 ms

doris-robot · 2025-01-16T03:59:39Z

ClickBench: Total hot run time: 30.28 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b612a1c1887a13372901cc82d7193070494f0335, data reload: false
query1	0.03	0.03	0.04
query2	0.07	0.04	0.03
query3	0.24	0.07	0.06
query4	1.63	0.11	0.11
query5	0.42	0.42	0.42
query6	1.17	0.65	0.67
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.60	0.51	0.49
query10	0.57	0.55	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.60	0.60	0.60
query14	2.75	2.84	2.88
query15	0.90	0.83	0.82
query16	0.38	0.38	0.38
query17	1.05	1.02	1.00
query18	0.23	0.22	0.20
query19	1.96	1.82	2.00
query20	0.01	0.01	0.01
query21	15.36	0.93	0.60
query22	0.77	0.87	0.62
query23	15.27	1.36	0.59
query24	3.21	1.10	0.45
query25	0.23	0.14	0.14
query26	0.27	0.14	0.15
query27	0.06	0.04	0.06
query28	13.70	1.01	0.43
query29	12.57	3.95	3.34
query30	0.26	0.09	0.06
query31	2.83	0.61	0.39
query32	3.23	0.55	0.46
query33	3.02	2.99	3.11
query34	16.68	5.23	4.48
query35	4.54	4.50	4.49
query36	0.63	0.49	0.49
query37	0.10	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.12	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.04
Total cold run time: 106.07 s
Total hot run time: 30.28 s

doris-robot · 2025-01-16T04:00:45Z

TeamCity be ut coverage result:
Function Coverage: 41.53% (10822/26060)
Line Coverage: 31.92% (91441/286467)
Region Coverage: 31.08% (46863/150799)
Branch Coverage: 27.17% (23733/87356)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b612a1c1887a13372901cc82d7193070494f0335_b612a1c1887a13372901cc82d7193070494f0335/report/index.html

morningman · 2025-01-16T09:01:52Z

+ // There's only one scan range for each backend in batch split mode. Each backend only starts up one ScanNode instance.
+ uint32_t shard_num =
+ std::min(config::doris_scanner_thread_pool_thread_num /
+ (_batch_split_mode ? 1 : state()->query_parallel_instance_num()),


I suggest to extract this _batch_split_mode ? 1 : state()->query_parallel_instance_num() to a method in file scan operator.

morningman · 2025-01-16T09:05:34Z


 template <typename Derived>
-Status ScanLocalState<Derived>::_start_scanners(
+Status ScanLocalState<Derived>::start_scanners(


The only difference between this and file_scan_operator is that file_scan_operator pass the query_parallel_instance_num separately.
So I suggest to create a new virtual method get_query_parallel_instance_num(),
and for scan_operator, it returns state()->query_parallel_instance_num().
For file_scan_operator, it returns _batch_split_mode ? 1 : state()->query_parallel_instance_num()

kaka11chen · 2025-01-17T09:13:38Z

run buildall

kaka11chen · 2025-01-17T09:26:33Z

run buildall

doris-robot · 2025-01-17T10:05:58Z

TPC-H: Total hot run time: 32841 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 19197e3be67250dfdc17c3955fcf745f09087d2c, data reload: false
------ Round 1 ----------------------------------
q1	19006	5683	5417	5417
q2	2063	311	192	192
q3	11471	1246	745	745
q4	10214	987	532	532
q5	7542	2339	2210	2210
q6	191	168	137	137
q7	921	762	633	633
q8	9416	1421	1186	1186
q9	6286	4851	4902	4851
q10	6886	2352	1911	1911
q11	464	290	256	256
q12	342	375	220	220
q13	17968	4013	3359	3359
q14	249	256	234	234
q15	545	493	469	469
q16	629	644	585	585
q17	576	865	343	343
q18	8165	6515	6562	6515
q19	3114	950	528	528
q20	303	322	189	189
q21	2768	2158	2013	2013
q22	382	342	316	316
Total cold run time: 109501 ms
Total hot run time: 32841 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5502	5454	5426	5426
q2	236	321	226	226
q3	2304	2677	2359	2359
q4	1383	1820	1391	1391
q5	4301	4834	4911	4834
q6	172	161	130	130
q7	2133	1984	1828	1828
q8	2624	2796	2683	2683
q9	7305	7251	7278	7251
q10	2898	3229	2783	2783
q11	581	553	497	497
q12	698	784	607	607
q13	3477	3857	3289	3289
q14	288	299	276	276
q15	520	481	455	455
q16	671	699	661	661
q17	1227	1749	1249	1249
q18	7632	7347	7394	7347
q19	801	1085	1098	1085
q20	2002	2050	1936	1936
q21	5566	5298	5111	5111
q22	612	612	605	605
Total cold run time: 52933 ms
Total hot run time: 52029 ms

doris-robot · 2025-01-17T10:18:03Z

TPC-DS: Total hot run time: 194319 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 19197e3be67250dfdc17c3955fcf745f09087d2c, data reload: false
query1	1315	964	956	956
query2	6150	2061	2061	2061
query3	11010	4450	4487	4450
query4	32832	23314	23262	23262
query5	3773	603	456	456
query6	281	191	189	189
query7	3985	502	314	314
query8	293	241	215	215
query9	9223	2584	2572	2572
query10	462	308	245	245
query11	17575	15191	14920	14920
query12	156	107	109	107
query13	1554	524	378	378
query14	9688	6847	7377	6847
query15	249	226	187	187
query16	7825	679	483	483
query17	1662	786	552	552
query18	2073	426	319	319
query19	229	199	173	173
query20	125	116	128	116
query21	206	122	106	106
query22	4650	4489	4530	4489
query23	34528	33346	33452	33346
query24	6741	2285	2366	2285
query25	472	473	400	400
query26	815	281	166	166
query27	2146	482	339	339
query28	5223	2472	2439	2439
query29	587	567	442	442
query30	211	198	170	170
query31	950	892	844	844
query32	71	64	59	59
query33	527	380	339	339
query34	777	849	524	524
query35	813	824	771	771
query36	1009	1029	979	979
query37	138	106	79	79
query38	4262	4359	4201	4201
query39	1490	1463	1432	1432
query40	207	118	103	103
query41	51	49	52	49
query42	130	107	99	99
query43	533	520	491	491
query44	1323	806	827	806
query45	189	179	166	166
query46	857	1041	647	647
query47	1882	1927	1867	1867
query48	387	410	333	333
query49	726	492	398	398
query50	650	670	396	396
query51	7083	7025	6994	6994
query52	103	109	101	101
query53	228	257	194	194
query54	511	514	419	419
query55	87	85	85	85
query56	259	277	235	235
query57	1209	1196	1141	1141
query58	268	241	248	241
query59	3235	3166	3059	3059
query60	292	280	267	267
query61	142	133	133	133
query62	834	745	663	663
query63	219	194	190	190
query64	2976	1021	666	666
query65	3248	3187	3152	3152
query66	919	396	305	305
query67	16050	15810	15340	15340
query68	4765	834	539	539
query69	510	282	272	272
query70	1207	1166	1151	1151
query71	399	291	273	273
query72	5975	3914	3783	3783
query73	650	741	354	354
query74	10267	9138	9203	9138
query75	3170	3182	2705	2705
query76	3131	1185	784	784
query77	463	351	294	294
query78	10237	10013	9345	9345
query79	3615	817	591	591
query80	707	535	456	456
query81	508	279	240	240
query82	615	146	130	130
query83	182	171	154	154
query84	238	92	79	79
query85	773	364	314	314
query86	399	316	310	310
query87	4430	4533	4303	4303
query88	5011	2144	2138	2138
query89	412	327	283	283
query90	1789	190	191	190
query91	133	141	120	120
query92	69	58	53	53
query93	2505	895	538	538
query94	687	456	303	303
query95	335	272	261	261
query96	482	612	276	276
query97	2815	2890	2701	2701
query98	233	200	206	200
query99	1305	1400	1289	1289
Total cold run time: 288691 ms
Total hot run time: 194319 ms

doris-robot · 2025-01-17T10:23:22Z

ClickBench: Total hot run time: 30.65 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 19197e3be67250dfdc17c3955fcf745f09087d2c, data reload: false
query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.11	0.10
query5	0.42	0.43	0.40
query6	1.20	0.65	0.64
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.51	0.50
query10	0.55	0.56	0.56
query11	0.15	0.11	0.11
query12	0.14	0.12	0.10
query13	0.60	0.60	0.61
query14	2.69	2.76	2.76
query15	0.89	0.83	0.83
query16	0.38	0.40	0.38
query17	1.07	1.00	0.98
query18	0.23	0.20	0.20
query19	2.00	1.85	1.96
query20	0.01	0.01	0.01
query21	15.37	0.93	0.58
query22	0.75	0.77	0.67
query23	15.34	1.39	0.60
query24	3.55	0.98	1.55
query25	0.12	0.15	0.09
query26	0.28	0.16	0.14
query27	0.05	0.07	0.04
query28	14.24	0.96	0.42
query29	12.61	3.94	3.27
query30	0.25	0.09	0.07
query31	2.84	0.59	0.38
query32	3.22	0.54	0.46
query33	2.99	3.01	3.07
query34	16.70	5.11	4.45
query35	4.53	4.47	4.48
query36	0.64	0.49	0.49
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.04	0.03	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.89 s
Total hot run time: 30.65 s

doris-robot · 2025-01-17T11:15:41Z

TeamCity be ut coverage result:
Function Coverage: 41.52% (10823/26064)
Line Coverage: 31.93% (91477/286507)
Region Coverage: 31.07% (46852/150807)
Branch Coverage: 27.16% (23726/87362)
Coverage Report: http://coverage.selectdb-in.cc/coverage/19197e3be67250dfdc17c3955fcf745f09087d2c_19197e3be67250dfdc17c3955fcf745f09087d2c/report/index.html

morningman

LGTM

github-actions · 2025-01-22T14:58:31Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-01-22T14:58:34Z

PR approved by anyone and no changes requested.

…de. (apache#44635) ### What problem does this PR solve? Problem Summary: There's only one scan range for each backend in batch split mode. Each backend only starts up one ScanNode instance. However, when calculating the concurrency of scanners in the scan operator in batch split mode, it is not divided by 1, but by `query_parallel_instance_num`, resulting in poor performance of batch split mode.

…de. (apache#44635) Problem Summary: There's only one scan range for each backend in batch split mode. Each backend only starts up one ScanNode instance. However, when calculating the concurrency of scanners in the scan operator in batch split mode, it is not divided by 1, but by `query_parallel_instance_num`, resulting in poor performance of batch split mode.

…de. (#50767) Cherry-pick #44635

…de. (apache#44635) ### What problem does this PR solve? Problem Summary: There's only one scan range for each backend in batch split mode. Each backend only starts up one ScanNode instance. However, when calculating the concurrency of scanners in the scan operator in batch split mode, it is not divided by 1, but by `query_parallel_instance_num`, resulting in poor performance of batch split mode.

…de. (apache#322) Cherry-pick apache#44635

kaka11chen force-pushed the batch_mode_max_scanner_thread_num branch from 0a75c6d to b8c3297Compare November 26, 2024 12:58

kaka11chen force-pushed the batch_mode_max_scanner_thread_num branch from b8c3297 to 5016ffaCompare December 19, 2024 18:09

kaka11chen force-pushed the batch_mode_max_scanner_thread_num branch from 5016ffa to a7db9f0Compare December 30, 2024 12:27

kaka11chen marked this pull request as ready for review December 30, 2024 14:55

kaka11chen force-pushed the batch_mode_max_scanner_thread_num branch from a7db9f0 to b612a1cCompare January 16, 2025 03:16

morningman reviewed Jan 16, 2025
View reviewed changes

kaka11chen force-pushed the batch_mode_max_scanner_thread_num branch 2 times, most recently from c473752 to 090ec3eCompare January 17, 2025 09:02

[Opt] opt max scanner thread number in batch split mode.
199a491

kaka11chen force-pushed the batch_mode_max_scanner_thread_num branch from 090ec3e to 8af9be6Compare January 17, 2025 09:13

Fix comments.
19197e3

morningman approved these changes Jan 22, 2025
View reviewed changes

morningman added dev/2.1.x-experimental dev/3.0.x-experimental labels Jan 22, 2025

github-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Jan 22, 2025

wuwenchi approved these changes Jan 22, 2025
View reviewed changes

morningman merged commit d7c99f8 into apache:masterJan 22, 2025

morningman added dev/2.1.x-lh and removed dev/2.1.x-lh labels Feb 17, 2025

kaka11chen mentioned this pull request May 9, 2025
[Opt] (multi-catalog) opt max scanner thread number in batch split mode. #50767
Merged
16 tasks

morningman added dev/3.0.x usercase Important user case type label and removed dev/3.0.x-experimental labels May 15, 2025

github-actionsBot added the dev/3.0.x-conflict label May 15, 2025

dataroaring pushed a commit that referenced this pull request May 15, 2025
[Opt] (multi-catalog) opt max scanner thread number in batch split mo…
49d0e72
…de. (#50767) Cherry-pick #44635

dataroaring added dev/3.0.6-merged and removed dev/3.0.x dev/3.0.x-conflict labels May 15, 2025

liaoxin01 pushed a commit to liaoxin01/doris that referenced this pull request Feb 9, 2026
[Opt] (multi-catalog) opt max scanner thread number in batch split mo…
4ea1ded
…de. (apache#322) Cherry-pick apache#44635

Uh oh!

Conversation

kaka11chen commented Nov 26, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

doris-robot commented Nov 26, 2024

Uh oh!

kaka11chen commented Nov 26, 2024

Uh oh!

kaka11chen commented Nov 26, 2024

Uh oh!

doris-robot commented Nov 26, 2024

Uh oh!

doris-robot commented Nov 26, 2024

Uh oh!

doris-robot commented Nov 26, 2024

Uh oh!

doris-robot commented Nov 26, 2024

Uh oh!

github-actionsBot commented Dec 19, 2024

Uh oh!

kaka11chen commented Dec 30, 2024

Uh oh!

doris-robot commented Dec 30, 2024

Uh oh!

doris-robot commented Dec 30, 2024

Uh oh!

doris-robot commented Dec 30, 2024

Uh oh!

doris-robot commented Dec 30, 2024

Uh oh!

kaka11chen commented Jan 16, 2025

Uh oh!

doris-robot commented Jan 16, 2025

Uh oh!

doris-robot commented Jan 16, 2025

Uh oh!

doris-robot commented Jan 16, 2025

Uh oh!

doris-robot commented Jan 16, 2025

Uh oh!

morningmanJan 16, 2025

Choose a reason for hiding this comment

Uh oh!

kaka11chenJan 16, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

morningmanJan 16, 2025

Choose a reason for hiding this comment

Uh oh!

kaka11chenJan 16, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kaka11chen commented Jan 17, 2025

Uh oh!

kaka11chen commented Jan 17, 2025

Uh oh!

doris-robot commented Jan 17, 2025

Uh oh!

doris-robot commented Jan 17, 2025

Uh oh!

doris-robot commented Jan 17, 2025

Uh oh!

doris-robot commented Jan 17, 2025

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jan 22, 2025

Uh oh!

github-actionsBot commented Jan 22, 2025

Uh oh!

Reviewers

Assignees

kaka11chen commented Nov 26, 2024•
edited
Loading

kaka11chenJan 16, 2025•
edited
Loading

kaka11chenJan 16, 2025•
edited
Loading