[opt](routine-load) optimize routine load task thread pool and related param(#32282) by sollhui · Pull Request #37443 · apache/doris

sollhui · 2024-07-08T06:25:44Z

pick (#32282)

doris-robot · 2024-07-08T06:25:49Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

sollhui · 2024-07-08T06:26:21Z

run buildall

github-actions · 2024-07-08T06:32:59Z

clang-tidy review says "All clean, LGTM! 👍"

sollhui · 2024-07-08T07:32:10Z

run buildall

github-actions · 2024-07-08T07:38:34Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-08T08:16:53Z

TPC-H: Total hot run time: 50589 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dce152eac2a0c724a4535c3f372b54e847abbf66, data reload: false
------ Round 1 ----------------------------------
q1	17541	4386	4381	4381
q2	2072	163	155	155
q3	10252	1927	1915	1915
q4	10104	1261	1354	1261
q5	8732	4000	3948	3948
q6	233	124	124	124
q7	2078	1596	1589	1589
q8	9380	2776	2745	2745
q9	11581	11352	10827	10827
q10	8663	3559	3577	3559
q11	429	246	247	246
q12	485	305	305	305
q13	18330	3944	4008	3944
q14	358	330	324	324
q15	513	459	455	455
q16	689	570	574	570
q17	1157	977	994	977
q18	7346	6803	6939	6803
q19	1816	1681	1645	1645
q20	554	304	321	304
q21	4440	4139	4079	4079
q22	535	435	433	433
Total cold run time: 117288 ms
Total hot run time: 50589 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4352	4324	4325	4324
q2	325	231	230	230
q3	4199	4180	4157	4157
q4	2792	2758	2775	2758
q5	7326	7198	7201	7198
q6	234	122	117	117
q7	3258	2828	2832	2828
q8	4447	4510	4544	4510
q9	17514	17127	17077	17077
q10	4220	4280	4268	4268
q11	757	690	678	678
q12	1027	883	856	856
q13	6990	3762	3771	3762
q14	449	436	414	414
q15	512	465	457	457
q16	731	677	694	677
q17	3825	3881	3838	3838
q18	8736	8737	8834	8737
q19	1747	1698	1664	1664
q20	2384	2094	2104	2094
q21	8492	8505	8451	8451
q22	1049	980	1020	980
Total cold run time: 85366 ms
Total hot run time: 80075 ms

doris-robot · 2024-07-08T08:29:40Z

TPC-DS: Total hot run time: 204807 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dce152eac2a0c724a4535c3f372b54e847abbf66, data reload: false
query1	955	388	378	378
query2	6527	2681	2775	2681
query3	6921	208	200	200
query4	21253	18060	17906	17906
query5	19748	6557	6484	6484
query6	289	219	234	219
query7	4146	311	305	305
query8	432	414	379	379
query9	3144	2706	2646	2646
query10	434	325	309	309
query11	11483	10783	10927	10783
query12	119	88	76	76
query13	5606	701	691	691
query14	17413	13749	14079	13749
query15	359	248	245	245
query16	6457	290	267	267
query17	1683	1462	882	882
query18	2300	411	408	408
query19	214	150	150	150
query20	81	77	80	77
query21	196	99	93	93
query22	5334	5203	5033	5033
query23	32735	32100	32115	32100
query24	6977	6575	6510	6510
query25	527	441	425	425
query26	603	164	163	163
query27	2002	299	308	299
query28	6206	2353	2316	2316
query29	2809	2755	2764	2755
query30	242	169	170	169
query31	938	733	760	733
query32	69	62	58	58
query33	410	266	251	251
query34	867	473	483	473
query35	1127	985	979	979
query36	1224	1182	1152	1152
query37	94	59	60	59
query38	3126	2957	2931	2931
query39	1378	1316	1315	1315
query40	203	97	96	96
query41	47	45	44	44
query42	81	82	80	80
query43	689	750	695	695
query44	1148	724	732	724
query45	251	238	238	238
query46	1256	945	966	945
query47	1737	1709	1638	1638
query48	1026	716	717	716
query49	652	374	367	367
query50	866	636	591	591
query51	4720	4647	4720	4647
query52	95	89	79	79
query53	456	326	327	326
query54	2644	2475	2479	2475
query55	95	76	78	76
query56	239	222	206	206
query57	1282	1154	1022	1022
query58	195	212	214	212
query59	4012	3985	3814	3814
query60	233	200	198	198
query61	96	92	96	92
query62	831	480	480	480
query63	488	345	343	343
query64	2441	1546	1485	1485
query65	3615	3569	3537	3537
query66	805	387	386	386
query67	16363	15676	15416	15416
query68	8575	685	636	636
query69	583	361	347	347
query70	1837	1550	1360	1360
query71	393	300	326	300
query72	6634	3538	3497	3497
query73	746	317	323	317
query74	6367	5820	5845	5820
query75	5063	3651	3705	3651
query76	4954	1142	1174	1142
query77	725	252	257	252
query78	12833	12142	12499	12142
query79	10781	632	658	632
query80	1120	393	409	393
query81	487	238	233	233
query82	1384	101	98	98
query83	173	134	132	132
query84	263	70	69	69
query85	875	337	337	337
query86	335	294	286	286
query87	3216	3034	3018	3018
query88	4925	2307	2310	2307
query89	468	303	303	303
query90	1901	205	222	205
query91	182	140	136	136
query92	59	53	53	53
query93	6270	594	582	582
query94	707	213	208	208
query95	1136	1069	1073	1069
query96	641	328	325	325
query97	6614	6339	6391	6339
query98	181	181	178	178
query99	2789	873	863	863
Total cold run time: 316211 ms
Total hot run time: 204807 ms

doris-robot · 2024-07-08T08:35:06Z

ClickBench: Total hot run time: 30.56 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dce152eac2a0c724a4535c3f372b54e847abbf66, data reload: false
query1	0.02	0.02	0.02
query2	0.07	0.03	0.02
query3	0.25	0.04	0.04
query4	1.79	0.08	0.06
query5	0.53	0.53	0.52
query6	1.27	0.61	0.62
query7	0.01	0.02	0.01
query8	0.03	0.02	0.02
query9	0.53	0.49	0.48
query10	0.54	0.54	0.52
query11	0.12	0.08	0.09
query12	0.12	0.10	0.09
query13	0.62	0.60	0.61
query14	0.80	0.78	0.80
query15	0.78	0.75	0.76
query16	0.36	0.36	0.36
query17	1.04	1.01	1.02
query18	0.22	0.26	0.25
query19	1.93	1.88	1.88
query20	0.01	0.01	0.01
query21	15.48	0.55	0.53
query22	2.16	1.98	1.57
query23	17.34	1.04	0.92
query24	5.20	4.32	0.90
query25	0.32	0.14	0.05
query26	0.92	0.15	0.16
query27	0.04	0.04	0.04
query28	4.92	0.78	0.73
query29	12.69	2.39	2.26
query30	0.61	0.51	0.52
query31	2.80	0.40	0.37
query32	3.39	0.50	0.49
query33	3.03	3.10	3.05
query34	15.26	4.80	4.81
query35	4.88	4.89	4.85
query36	1.05	1.00	1.01
query37	0.06	0.04	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.13	0.14
query41	0.07	0.01	0.01
query42	0.02	0.01	0.01
query43	0.02	0.01	0.02
Total cold run time: 101.51 s
Total hot run time: 30.56 s

doris-robot · 2024-07-08T08:38:21Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit dce152eac2a0c724a4535c3f372b54e847abbf66 with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.5 seconds inserted 10000000 Rows, about 465K ops/s

sollhui · 2024-07-08T09:23:04Z

run buildall

github-actions · 2024-07-08T09:30:37Z

clang-tidy review says "All clean, LGTM! 👍"

sollhui · 2024-07-08T09:34:15Z

run buildall

github-actions · 2024-07-08T09:40:46Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-08T10:07:26Z

TPC-H: Total hot run time: 50363 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f28a7797576fbce9a8cf20d05375e17379c01b56, data reload: false
------ Round 1 ----------------------------------
q1	17889	4405	4408	4405
q2	2043	159	151	151
q3	10432	1923	1966	1923
q4	10426	1225	1346	1225
q5	8454	3914	3964	3914
q6	242	126	125	125
q7	2080	1606	1603	1603
q8	9707	2753	2719	2719
q9	10502	10247	10250	10247
q10	8637	3522	3529	3522
q11	414	246	247	246
q12	475	313	312	312
q13	18468	4036	4188	4036
q14	367	340	343	340
q15	531	471	459	459
q16	694	623	591	591
q17	1148	946	918	918
q18	7591	7412	7175	7175
q19	2533	1652	1620	1620
q20	612	309	298	298
q21	4485	4178	4084	4084
q22	528	450	478	450
Total cold run time: 118258 ms
Total hot run time: 50363 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4288	4330	4300	4300
q2	327	240	220	220
q3	4177	4168	4152	4152
q4	2760	2746	2748	2746
q5	7184	7131	7156	7131
q6	234	120	120	120
q7	3245	2888	2848	2848
q8	4346	4451	4486	4451
q9	16855	16883	16814	16814
q10	4257	4247	4293	4247
q11	763	692	675	675
q12	1050	852	874	852
q13	7230	3731	3799	3731
q14	444	428	440	428
q15	498	459	458	458
q16	755	688	682	682
q17	3891	3793	3875	3793
q18	8720	8970	8652	8652
q19	1730	1678	1642	1642
q20	2381	2105	2117	2105
q21	8401	8516	8642	8516
q22	1057	987	964	964
Total cold run time: 84593 ms
Total hot run time: 79527 ms

doris-robot · 2024-07-08T10:20:13Z

TPC-DS: Total hot run time: 203358 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f28a7797576fbce9a8cf20d05375e17379c01b56, data reload: false
query1	922	427	378	378
query2	6548	2746	2628	2628
query3	6920	210	210	210
query4	20884	18159	18001	18001
query5	19756	6467	6533	6467
query6	296	217	230	217
query7	4154	304	309	304
query8	452	428	415	415
query9	3131	2674	2625	2625
query10	423	308	295	295
query11	11326	10720	10762	10720
query12	129	74	82	74
query13	5605	721	688	688
query14	18096	13598	13182	13182
query15	357	254	244	244
query16	6471	287	267	267
query17	1572	1524	885	885
query18	2274	413	411	411
query19	209	159	150	150
query20	76	85	83	83
query21	194	97	97	97
query22	5158	4969	5007	4969
query23	32463	32020	31892	31892
query24	6991	6674	6655	6655
query25	541	432	436	432
query26	506	164	160	160
query27	1770	291	296	291
query28	6143	2366	2325	2325
query29	2878	2843	2914	2843
query30	251	167	172	167
query31	889	731	740	731
query32	69	64	64	64
query33	409	274	263	263
query34	849	479	483	479
query35	1150	916	899	899
query36	1410	1067	1196	1067
query37	92	60	65	60
query38	3067	2984	2955	2955
query39	1362	1319	1306	1306
query40	208	97	95	95
query41	46	43	45	43
query42	90	89	82	82
query43	689	717	759	717
query44	1125	704	712	704
query45	253	240	234	234
query46	1255	962	983	962
query47	1777	1753	1786	1753
query48	991	728	713	713
query49	618	381	385	381
query50	861	587	603	587
query51	4814	4687	4622	4622
query52	96	74	88	74
query53	462	339	332	332
query54	2678	2427	2470	2427
query55	108	91	87	87
query56	239	238	213	213
query57	1175	1065	1023	1023
query58	200	205	188	188
query59	4142	4442	3784	3784
query60	217	214	220	214
query61	97	93	98	93
query62	813	466	465	465
query63	487	349	346	346
query64	2522	1525	1501	1501
query65	3657	3554	3509	3509
query66	760	384	393	384
query67	15433	15722	15319	15319
query68	11377	666	648	648
query69	577	357	332	332
query70	1969	1323	1299	1299
query71	407	317	310	310
query72	6574	3487	3498	3487
query73	1345	323	311	311
query74	6309	5825	5904	5825
query75	5435	3730	3708	3708
query76	6978	1114	1168	1114
query77	1197	256	243	243
query78	12583	12045	11654	11654
query79	9027	636	636	636
query80	959	396	401	396
query81	464	234	232	232
query82	882	99	98	98
query83	165	133	131	131
query84	249	71	73	71
query85	871	337	342	337
query86	344	290	318	290
query87	3358	3016	2999	2999
query88	4285	2310	2280	2280
query89	480	283	284	283
query90	1943	216	203	203
query91	184	141	139	139
query92	58	58	51	51
query93	6950	604	553	553
query94	701	209	211	209
query95	1108	1078	1068	1068
query96	650	324	322	322
query97	6466	6405	6354	6354
query98	199	178	173	173
query99	2954	851	866	851
Total cold run time: 318654 ms
Total hot run time: 203358 ms

doris-robot · 2024-07-08T10:25:41Z

ClickBench: Total hot run time: 30.57 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f28a7797576fbce9a8cf20d05375e17379c01b56, data reload: false
query1	0.03	0.02	0.02
query2	0.07	0.02	0.02
query3	0.24	0.06	0.05
query4	1.78	0.06	0.06
query5	0.54	0.52	0.52
query6	1.24	0.63	0.67
query7	0.01	0.00	0.01
query8	0.04	0.03	0.02
query9	0.53	0.48	0.48
query10	0.54	0.53	0.55
query11	0.12	0.08	0.09
query12	0.11	0.09	0.10
query13	0.62	0.62	0.61
query14	0.78	0.78	0.78
query15	0.79	0.76	0.77
query16	0.39	0.39	0.36
query17	1.02	0.99	1.02
query18	0.23	0.27	0.23
query19	1.88	1.88	1.84
query20	0.02	0.01	0.01
query21	15.52	0.54	0.55
query22	2.41	2.63	1.55
query23	16.95	1.18	0.92
query24	5.41	1.18	1.06
query25	0.40	0.08	0.04
query26	0.60	0.15	0.15
query27	0.05	0.04	0.03
query28	7.52	0.78	0.71
query29	12.62	2.19	2.34
query30	0.63	0.50	0.51
query31	2.81	0.40	0.38
query32	3.37	0.49	0.50
query33	3.07	3.07	3.09
query34	15.25	4.83	4.79
query35	4.85	4.86	4.82
query36	1.03	1.01	1.02
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.07	0.01	0.01
query42	0.02	0.01	0.01
query43	0.03	0.01	0.02
Total cold run time: 103.86 s
Total hot run time: 30.57 s

doris-robot · 2024-07-08T10:28:56Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit f28a7797576fbce9a8cf20d05375e17379c01b56 with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 21.9 seconds inserted 10000000 Rows, about 456K ops/s

sollhui · 2024-07-08T11:45:00Z

run buildall

github-actions · 2024-07-08T11:51:28Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-08T12:16:54Z

TPC-H: Total hot run time: 49761 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b39ecd8498748e6f0e68519975cf79f9627f8ee8, data reload: false
------ Round 1 ----------------------------------
q1	17727	4399	4382	4382
q2	2080	158	150	150
q3	10460	1842	1959	1842
q4	10356	1224	1322	1224
q5	8508	3915	3920	3915
q6	227	124	149	124
q7	2044	1627	1616	1616
q8	9263	2750	2687	2687
q9	10998	10291	10338	10291
q10	8643	3508	3509	3508
q11	426	248	238	238
q12	473	310	304	304
q13	18355	3940	4031	3940
q14	359	339	321	321
q15	503	454	458	454
q16	674	577	585	577
q17	1126	952	975	952
q18	7296	6834	6859	6834
q19	1770	1649	1589	1589
q20	521	295	292	292
q21	4420	4096	4086	4086
q22	513	435	445	435
Total cold run time: 116742 ms
Total hot run time: 49761 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4339	4322	4256	4256
q2	327	232	232	232
q3	4173	4130	4131	4130
q4	2755	2739	2746	2739
q5	7190	7116	7133	7116
q6	265	125	122	122
q7	3201	2818	2848	2818
q8	4367	4456	4477	4456
q9	17316	17090	17124	17090
q10	4240	4306	4275	4275
q11	784	691	693	691
q12	1051	850	870	850
q13	7061	3767	3747	3747
q14	457	428	427	427
q15	507	456	458	456
q16	725	686	707	686
q17	3864	3831	3817	3817
q18	8747	8664	8805	8664
q19	1709	1701	1636	1636
q20	2333	2145	2133	2133
q21	8587	8459	8474	8459
q22	1046	960	1005	960
Total cold run time: 85044 ms
Total hot run time: 79760 ms

doris-robot · 2024-07-08T12:29:43Z

TPC-DS: Total hot run time: 204133 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b39ecd8498748e6f0e68519975cf79f9627f8ee8, data reload: false
query1	948	427	389	389
query2	6530	2786	2660	2660
query3	6913	212	207	207
query4	21306	18058	17983	17983
query5	19747	6488	6476	6476
query6	288	220	228	220
query7	4174	300	315	300
query8	430	389	399	389
query9	3160	2733	2663	2663
query10	414	300	312	300
query11	11410	10944	10754	10754
query12	125	74	75	74
query13	5610	700	690	690
query14	17413	13744	13197	13197
query15	358	244	258	244
query16	6460	286	265	265
query17	1743	1449	890	890
query18	2311	406	413	406
query19	219	151	157	151
query20	78	77	80	77
query21	190	110	102	102
query22	5211	5127	5170	5127
query23	32513	32022	32020	32020
query24	6950	6507	6478	6478
query25	505	434	435	434
query26	529	163	158	158
query27	1883	295	295	295
query28	6222	2391	2359	2359
query29	2838	2620	2738	2620
query30	244	162	167	162
query31	930	760	779	760
query32	72	66	60	60
query33	421	265	265	265
query34	842	475	489	475
query35	1123	932	947	932
query36	1289	1163	1107	1107
query37	91	65	61	61
query38	3100	2899	2967	2899
query39	1397	1345	1343	1343
query40	210	98	93	93
query41	45	44	43	43
query42	89	85	84	84
query43	663	717	636	636
query44	1117	722	723	722
query45	249	235	238	235
query46	1221	977	980	977
query47	2087	1692	1742	1692
query48	1015	704	715	704
query49	622	379	382	379
query50	868	593	635	593
query51	4750	4620	4674	4620
query52	96	74	80	74
query53	459	324	317	317
query54	2663	2456	2445	2445
query55	89	87	81	81
query56	229	222	222	222
query57	1185	1134	1117	1117
query58	207	213	202	202
query59	4170	4112	3992	3992
query60	219	209	226	209
query61	101	97	100	97
query62	921	448	418	418
query63	495	343	342	342
query64	2448	1560	1459	1459
query65	3612	3526	3519	3519
query66	774	389	389	389
query67	16623	17106	15183	15183
query68	8256	670	658	658
query69	594	356	361	356
query70	1652	1585	1513	1513
query71	407	298	320	298
query72	6554	3537	3507	3507
query73	745	341	330	330
query74	6367	5839	5910	5839
query75	4497	3700	3666	3666
query76	4639	1162	1199	1162
query77	583	256	260	256
query78	12789	12865	11882	11882
query79	13208	637	634	634
query80	858	394	395	394
query81	507	243	233	233
query82	1042	101	101	101
query83	188	135	135	135
query84	263	72	70	70
query85	849	341	343	341
query86	340	300	288	288
query87	3297	3024	3080	3024
query88	5914	2378	2368	2368
query89	442	320	295	295
query90	2091	215	201	201
query91	172	141	142	141
query92	63	53	51	51
query93	6057	573	605	573
query94	990	218	213	213
query95	1114	1052	1042	1042
query96	651	343	335	335
query97	6532	6464	6387	6387
query98	194	177	173	173
query99	2846	945	831	831
Total cold run time: 317915 ms
Total hot run time: 204133 ms

doris-robot · 2024-07-08T12:35:10Z

ClickBench: Total hot run time: 30.24 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b39ecd8498748e6f0e68519975cf79f9627f8ee8, data reload: false
query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.04	0.05
query4	1.80	0.06	0.06
query5	0.53	0.52	0.52
query6	1.25	0.62	0.62
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.52	0.47	0.46
query10	0.54	0.52	0.53
query11	0.12	0.08	0.08
query12	0.11	0.09	0.09
query13	0.62	0.61	0.61
query14	0.79	0.78	0.80
query15	0.78	0.75	0.77
query16	0.36	0.37	0.37
query17	1.01	1.01	1.00
query18	0.22	0.24	0.22
query19	1.94	1.83	1.87
query20	0.02	0.01	0.02
query21	15.45	0.56	0.56
query22	2.10	2.05	1.50
query23	16.88	0.87	1.05
query24	7.94	0.65	0.61
query25	0.40	0.15	0.04
query26	0.66	0.17	0.16
query27	0.05	0.04	0.04
query28	5.97	0.78	0.74
query29	12.76	2.33	2.32
query30	0.54	0.53	0.52
query31	2.82	0.38	0.38
query32	3.37	0.50	0.49
query33	3.13	3.08	3.06
query34	15.25	4.81	4.81
query35	4.86	4.86	4.85
query36	1.07	1.01	1.02
query37	0.06	0.04	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.01
query40	0.17	0.14	0.15
query41	0.07	0.01	0.01
query42	0.02	0.01	0.02
query43	0.02	0.02	0.02
Total cold run time: 104.64 s
Total hot run time: 30.24 s

doris-robot · 2024-07-08T12:38:24Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit b39ecd8498748e6f0e68519975cf79f9627f8ee8 with default session variables
Stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.5 seconds inserted 10000000 Rows, about 465K ops/s

…d param(#32282)

sollhui · 2024-07-08T12:40:08Z

run buildall

github-actions · 2024-07-08T12:51:27Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-08T12:58:55Z

TPC-H: Total hot run time: 50808 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 67ad7b8a04236255f78d4ce131259f295903cad0, data reload: false
------ Round 1 ----------------------------------
q1	17540	4391	4388	4388
q2	2072	159	156	156
q3	10254	1928	1935	1928
q4	10118	1258	1333	1258
q5	8420	3902	3932	3902
q6	240	124	126	124
q7	2097	1653	1620	1620
q8	9275	2741	2732	2732
q9	10677	10865	10894	10865
q10	8666	3502	3530	3502
q11	411	249	252	249
q12	475	317	312	312
q13	18366	3998	4069	3998
q14	354	339	320	320
q15	507	461	452	452
q16	680	569	573	569
q17	1136	961	959	959
q18	7249	6992	6997	6992
q19	1810	1667	1604	1604
q20	569	313	299	299
q21	4413	4165	4134	4134
q22	543	457	445	445
Total cold run time: 115872 ms
Total hot run time: 50808 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4336	4342	4313	4313
q2	328	228	226	226
q3	4201	4178	4210	4178
q4	2766	2760	2780	2760
q5	7205	7133	7093	7093
q6	239	122	118	118
q7	3303	2903	2889	2889
q8	4408	4493	4527	4493
q9	16839	16892	16748	16748
q10	4270	4305	4273	4273
q11	799	683	709	683
q12	1026	868	854	854
q13	6269	3747	3765	3747
q14	457	412	417	412
q15	508	450	460	450
q16	744	687	682	682
q17	3859	3865	3890	3865
q18	8898	8822	8840	8822
q19	1742	1734	1635	1635
q20	2401	2099	2156	2099
q21	8440	8386	8543	8386
q22	1068	962	1007	962
Total cold run time: 84106 ms
Total hot run time: 79688 ms

doris-robot · 2024-07-08T13:04:11Z

TeamCity be ut coverage result:
Function Coverage: 37.89% (8115/21418)
Line Coverage: 29.56% (66505/224948)
Region Coverage: 29.03% (34269/118062)
Branch Coverage: 24.90% (17602/70700)
Coverage Report: http://coverage.selectdb-in.cc/coverage/67ad7b8a04236255f78d4ce131259f295903cad0_67ad7b8a04236255f78d4ce131259f295903cad0/report/index.html

doris-robot · 2024-07-08T13:11:37Z

TPC-DS: Total hot run time: 203523 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 67ad7b8a04236255f78d4ce131259f295903cad0, data reload: false
query1	953	401	384	384
query2	6535	2839	2572	2572
query3	6924	212	210	210
query4	20197	17929	17928	17928
query5	19726	6552	6533	6533
query6	299	223	233	223
query7	4159	305	306	305
query8	464	400	468	400
query9	3219	2738	2682	2682
query10	437	283	287	283
query11	11327	10612	10617	10612
query12	133	78	74	74
query13	5599	701	696	696
query14	17968	13180	13715	13180
query15	367	240	250	240
query16	6474	284	268	268
query17	1708	1448	867	867
query18	2324	419	412	412
query19	209	144	154	144
query20	83	79	81	79
query21	195	98	93	93
query22	5121	5034	4820	4820
query23	32515	31957	32112	31957
query24	7084	6556	6500	6500
query25	528	431	439	431
query26	613	163	162	162
query27	2017	307	303	303
query28	6214	2380	2338	2338
query29	2887	2691	2830	2691
query30	247	165	169	165
query31	914	732	741	732
query32	70	64	63	63
query33	412	259	261	259
query34	861	482	488	482
query35	1122	952	904	904
query36	1248	1416	1074	1074
query37	92	62	63	62
query38	3099	2905	2941	2905
query39	1370	1336	1320	1320
query40	210	94	90	90
query41	48	46	44	44
query42	86	88	78	78
query43	701	643	690	643
query44	1120	717	731	717
query45	245	238	244	238
query46	1244	975	986	975
query47	1780	1806	1638	1638
query48	1038	730	746	730
query49	616	371	375	371
query50	870	596	597	596
query51	4780	4572	4632	4572
query52	105	91	86	86
query53	455	333	321	321
query54	2679	2465	2479	2465
query55	89	77	89	77
query56	239	219	213	213
query57	1285	1112	1044	1044
query58	225	233	211	211
query59	4301	4224	3955	3955
query60	211	211	205	205
query61	98	92	118	92
query62	822	458	481	458
query63	493	342	344	342
query64	2422	1554	1508	1508
query65	3611	3569	3600	3569
query66	818	384	384	384
query67	16487	15837	16912	15837
query68	9005	673	658	658
query69	580	359	351	351
query70	1696	1446	1511	1446
query71	414	309	308	308
query72	6565	3474	3451	3451
query73	744	331	328	328
query74	6322	5880	5806	5806
query75	5385	3684	3738	3684
query76	5506	1151	1185	1151
query77	959	249	269	249
query78	12619	11963	11500	11500
query79	8589	643	621	621
query80	1200	406	411	406
query81	492	239	230	230
query82	1729	99	97	97
query83	170	138	133	133
query84	260	72	69	69
query85	879	327	332	327
query86	326	297	325	297
query87	3301	3012	3037	3012
query88	5066	2379	2368	2368
query89	396	275	302	275
query90	1939	218	220	218
query91	179	152	156	152
query92	61	55	51	51
query93	5383	566	584	566
query94	706	209	212	209
query95	1114	1053	1077	1053
query96	657	332	349	332
query97	6505	6352	6388	6352
query98	196	177	164	164
query99	2869	847	907	847
Total cold run time: 314604 ms
Total hot run time: 203523 ms

doris-robot · 2024-07-08T13:17:06Z

ClickBench: Total hot run time: 31.43 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 67ad7b8a04236255f78d4ce131259f295903cad0, data reload: false
query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.24	0.05	0.05
query4	1.79	0.08	0.08
query5	0.54	0.53	0.53
query6	1.24	0.62	0.62
query7	0.01	0.01	0.02
query8	0.03	0.02	0.02
query9	0.53	0.50	0.47
query10	0.55	0.52	0.53
query11	0.12	0.09	0.09
query12	0.11	0.09	0.09
query13	0.64	0.62	0.62
query14	0.80	0.78	0.78
query15	0.79	0.76	0.77
query16	0.36	0.37	0.36
query17	1.03	1.03	1.02
query18	0.21	0.24	0.27
query19	1.93	1.81	1.81
query20	0.02	0.01	0.01
query21	15.48	0.54	0.54
query22	2.34	2.30	1.80
query23	17.39	1.02	0.87
query24	5.77	2.45	1.42
query25	0.38	0.09	0.06
query26	0.82	0.16	0.16
query27	0.04	0.05	0.04
query28	5.63	0.78	0.74
query29	12.78	2.31	2.33
query30	0.58	0.51	0.53
query31	2.80	0.38	0.39
query32	3.36	0.50	0.49
query33	3.03	3.08	3.12
query34	15.29	4.80	4.83
query35	4.85	4.83	4.84
query36	1.06	1.02	1.02
query37	0.06	0.04	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.17	0.14	0.14
query41	0.07	0.02	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 103.02 s
Total hot run time: 31.43 s

doris-robot · 2024-07-08T13:20:21Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 67ad7b8a04236255f78d4ce131259f295903cad0 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 22.0 seconds inserted 10000000 Rows, about 454K ops/s

…d param(apache#32282) (apache#37443) pick (apache#32282)

sollhui changed the title ~~[opt](routine-load) optimize routine load task thread pool and relate…~~[opt](routine-load) optimize routine load task thread pool and related param(#32282)Jul 8, 2024

[opt](routine-load) optimize routine load task thread pool and relate…
67ad7b8
…d param(#32282)

dataroaring merged commit 2eb85bc into apache:branch-2.0Jul 9, 2024

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
[opt](routine-load) optimize routine load task thread pool and relate…
ce4492c
…d param(apache#32282) (apache#37443) pick (apache#32282)

Uh oh!

Conversation

sollhui commented Jul 8, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doris-robot commented Jul 8, 2024

Uh oh!

sollhui commented Jul 8, 2024

Uh oh!

github-actionsBot commented Jul 8, 2024

Uh oh!

sollhui commented Jul 8, 2024

Uh oh!

github-actionsBot commented Jul 8, 2024

Uh oh!

doris-robot commented Jul 8, 2024

Uh oh!

doris-robot commented Jul 8, 2024

Uh oh!

doris-robot commented Jul 8, 2024

Uh oh!

doris-robot commented Jul 8, 2024

Uh oh!

sollhui commented Jul 8, 2024

Uh oh!

github-actionsBot commented Jul 8, 2024

Uh oh!

sollhui commented Jul 8, 2024

Uh oh!

github-actionsBot commented Jul 8, 2024

Uh oh!

doris-robot commented Jul 8, 2024

Uh oh!

doris-robot commented Jul 8, 2024

Uh oh!

doris-robot commented Jul 8, 2024

Uh oh!

doris-robot commented Jul 8, 2024

Uh oh!

sollhui commented Jul 8, 2024

Uh oh!

github-actionsBot commented Jul 8, 2024

Uh oh!

doris-robot commented Jul 8, 2024

Uh oh!

doris-robot commented Jul 8, 2024

Uh oh!

doris-robot commented Jul 8, 2024

Uh oh!

doris-robot commented Jul 8, 2024

Uh oh!

sollhui commented Jul 8, 2024

Uh oh!

github-actionsBot commented Jul 8, 2024

Uh oh!

doris-robot commented Jul 8, 2024

Uh oh!

doris-robot commented Jul 8, 2024

Uh oh!

doris-robot commented Jul 8, 2024

Uh oh!

doris-robot commented Jul 8, 2024

Uh oh!

doris-robot commented Jul 8, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sollhui commented Jul 8, 2024•
edited
Loading