[improve](routine-load) optimize routine load job auto resume policy by sollhui · Pull Request #35266 · apache/doris

sollhui · 2024-05-23T05:15:14Z

Proposed changes

When some exception occur, such as get offset failed for network isolation or be node down when upgrade, job will pause unexpectedly. Therefore, Doris introduce auto resume to keep job stable in #2958.

But currently there are some issues with this policy:

If the fault recovery time is relatively long, such as the downtime of BE, then after auto resume three times, it will be completely unable to automatically resume. But in reality, it can be restored by automatically resume.

Improvement measures:

Unrestricted retry attempts to solve the problem of automatic resume caused by long recovery time.
Add the backoff algorithm to solve the problem of some resume being invalid, such as Kafka having problems, but Doris still having a high resume frequency. The maximum backoff time is five minutes to avoid the difficulty of recovery caused by too much backoff time.

doris-robot · 2024-05-23T05:15:19Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

sollhui · 2024-05-23T05:15:26Z

run buildall

doris-robot · 2024-05-23T05:33:48Z

TPC-H: Total hot run time: 39787 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d599284c17eb3678171dac2b254cec50143ac52c, data reload: false
------ Round 1 ----------------------------------
q1	17897	4401	4218	4218
q2	2018	199	195	195
q3	10455	1238	1206	1206
q4	10147	856	756	756
q5	7482	2620	2640	2620
q6	211	129	131	129
q7	950	603	586	586
q8	9244	2059	2066	2059
q9	8776	6464	6414	6414
q10	8976	3667	3663	3663
q11	451	243	236	236
q12	433	217	214	214
q13	17900	2978	2950	2950
q14	266	214	234	214
q15	516	469	482	469
q16	526	379	374	374
q17	962	653	714	653
q18	8061	7559	7499	7499
q19	7486	1559	1509	1509
q20	653	317	321	317
q21	4979	3234	3908	3234
q22	341	274	272	272
Total cold run time: 118730 ms
Total hot run time: 39787 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4400	4167	4183	4167
q2	379	278	259	259
q3	3015	2750	2701	2701
q4	1861	1568	1537	1537
q5	5265	5261	5291	5261
q6	208	122	123	122
q7	2098	1686	1756	1686
q8	3199	3326	3311	3311
q9	8346	8287	8331	8287
q10	3923	3696	3632	3632
q11	584	474	480	474
q12	758	611	589	589
q13	17186	2992	2933	2933
q14	284	255	262	255
q15	524	475	469	469
q16	471	415	443	415
q17	1796	1484	1473	1473
q18	7730	7638	7363	7363
q19	1645	1559	1586	1559
q20	1960	1802	1788	1788
q21	4843	4686	4838	4686
q22	562	531	486	486
Total cold run time: 71037 ms
Total hot run time: 53453 ms

doris-robot · 2024-05-23T05:44:44Z

TPC-DS: Total hot run time: 169313 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d599284c17eb3678171dac2b254cec50143ac52c, data reload: false
query1	913	387	363	363
query2	6467	2550	2488	2488
query3	6644	209	211	209
query4	19040	17269	17202	17202
query5	4156	411	432	411
query6	243	156	156	156
query7	4625	294	287	287
query8	240	187	183	183
query9	8623	2367	2335	2335
query10	449	287	257	257
query11	10437	10041	10131	10041
query12	136	91	90	90
query13	1659	366	364	364
query14	10043	7266	6976	6976
query15	248	163	176	163
query16	8236	287	279	279
query17	2086	563	556	556
query18	3081	292	288	288
query19	211	165	166	165
query20	145	91	95	91
query21	192	137	132	132
query22	4413	4253	3935	3935
query23	33659	32983	33103	32983
query24	12137	2852	2833	2833
query25	644	345	351	345
query26	1739	157	156	156
query27	2969	323	315	315
query28	7601	2024	2041	2024
query29	1022	610	598	598
query30	317	170	174	170
query31	956	758	780	758
query32	98	52	53	52
query33	757	287	259	259
query34	1009	476	476	476
query35	742	590	576	576
query36	1066	930	914	914
query37	271	67	70	67
query38	2904	2771	2760	2760
query39	856	785	796	785
query40	283	126	127	126
query41	49	47	50	47
query42	103	94	99	94
query43	593	572	524	524
query44	1252	710	734	710
query45	181	158	161	158
query46	1075	715	752	715
query47	1874	1768	1786	1768
query48	361	293	295	293
query49	1183	378	380	378
query50	770	387	397	387
query51	6938	6838	6786	6786
query52	101	86	91	86
query53	347	281	291	281
query54	959	423	413	413
query55	74	73	72	72
query56	260	233	239	233
query57	1138	1051	1066	1051
query58	228	213	210	210
query59	3443	3408	2954	2954
query60	288	245	250	245
query61	94	88	86	86
query62	636	449	466	449
query63	310	292	276	276
query64	9755	2165	1779	1779
query65	3149	3085	3104	3085
query66	1391	332	322	322
query67	15317	14979	14912	14912
query68	4542	531	531	531
query69	445	263	266	263
query70	1155	1072	1083	1072
query71	429	263	267	263
query72	7643	4284	2530	2530
query73	708	323	320	320
query74	6159	5643	5686	5643
query75	3420	2646	2642	2642
query76	2915	1065	1000	1000
query77	424	313	263	263
query78	10207	9732	9937	9732
query79	2588	523	512	512
query80	1077	427	429	427
query81	531	245	245	245
query82	869	101	98	98
query83	250	165	166	165
query84	237	87	84	84
query85	1326	266	265	265
query86	468	291	321	291
query87	3316	3112	3133	3112
query88	4381	2326	2379	2326
query89	478	384	377	377
query90	2008	185	183	183
query91	118	94	95	94
query92	56	47	50	47
query93	1579	507	506	506
query94	1209	181	182	181
query95	404	310	309	309
query96	587	262	264	262
query97	3257	3022	3069	3022
query98	231	215	225	215
query99	1094	887	851	851
Total cold run time: 277297 ms
Total hot run time: 169313 ms

doris-robot · 2024-05-23T05:50:16Z

ClickBench: Total hot run time: 30.29 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d599284c17eb3678171dac2b254cec50143ac52c, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.66	0.07	0.08
query5	0.50	0.49	0.51
query6	1.13	0.74	0.73
query7	0.02	0.02	0.02
query8	0.06	0.04	0.04
query9	0.55	0.50	0.48
query10	0.54	0.54	0.54
query11	0.16	0.11	0.12
query12	0.14	0.12	0.12
query13	0.58	0.59	0.59
query14	0.76	0.76	0.76
query15	0.81	0.81	0.80
query16	0.36	0.36	0.38
query17	1.02	1.04	0.97
query18	0.23	0.24	0.25
query19	1.90	1.78	1.75
query20	0.01	0.01	0.01
query21	15.71	0.67	0.67
query22	4.67	7.55	1.56
query23	18.28	1.41	1.36
query24	2.00	0.22	0.20
query25	0.14	0.08	0.08
query26	0.25	0.17	0.17
query27	0.08	0.08	0.08
query28	13.35	1.02	0.99
query29	13.11	3.23	3.25
query30	0.25	0.06	0.07
query31	2.85	0.39	0.38
query32	3.30	0.48	0.47
query33	2.88	2.96	2.89
query34	16.84	4.40	4.40
query35	4.51	4.51	4.51
query36	0.66	0.47	0.46
query37	0.18	0.15	0.15
query38	0.16	0.16	0.14
query39	0.04	0.04	0.03
query40	0.17	0.14	0.16
query41	0.10	0.04	0.04
query42	0.06	0.05	0.05
query43	0.05	0.03	0.04
Total cold run time: 110.42 s
Total hot run time: 30.29 s

sollhui · 2024-05-23T06:52:02Z

run buildall

doris-robot · 2024-05-23T07:31:39Z

TPC-H: Total hot run time: 41657 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 15416770af32580da3cefd3106aad66c7f7c9d8b, data reload: false
------ Round 1 ----------------------------------
q1	17618	4251	4169	4169
q2	2000	191	188	188
q3	10454	1236	1188	1188
q4	10191	792	767	767
q5	7477	2751	2709	2709
q6	216	131	137	131
q7	961	592	618	592
q8	9226	2133	2084	2084
q9	9174	6726	6659	6659
q10	9328	3970	3843	3843
q11	427	249	244	244
q12	517	224	230	224
q13	17395	3193	3319	3193
q14	272	211	228	211
q15	512	476	472	472
q16	486	405	387	387
q17	973	689	705	689
q18	8416	7886	7793	7793
q19	5748	1592	1529	1529
q20	634	316	317	316
q21	5180	3992	4033	3992
q22	347	277	284	277
Total cold run time: 117552 ms
Total hot run time: 41657 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4539	4404	4394	4394
q2	401	275	271	271
q3	3170	2959	2725	2725
q4	1876	1581	1626	1581
q5	5475	5480	5481	5480
q6	221	126	126	126
q7	2233	1822	1816	1816
q8	3226	3344	3384	3344
q9	8667	8696	8606	8606
q10	3940	3865	3804	3804
q11	600	485	485	485
q12	824	638	625	625
q13	16060	3132	3177	3132
q14	300	275	262	262
q15	521	486	495	486
q16	485	447	423	423
q17	1755	1478	1463	1463
q18	7716	7593	7440	7440
q19	1699	1537	1538	1537
q20	2003	1806	1776	1776
q21	5013	4713	4789	4713
q22	572	482	501	482
Total cold run time: 71296 ms
Total hot run time: 54971 ms

doris-robot · 2024-05-23T07:42:20Z

TPC-DS: Total hot run time: 173065 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 15416770af32580da3cefd3106aad66c7f7c9d8b, data reload: false
query1	925	370	363	363
query2	6441	2562	2368	2368
query3	6643	221	209	209
query4	19244	17277	17203	17203
query5	4158	426	422	422
query6	253	155	159	155
query7	4657	304	288	288
query8	242	192	203	192
query9	8705	2362	2339	2339
query10	461	281	271	271
query11	10646	10058	10054	10054
query12	142	108	86	86
query13	1626	364	363	363
query14	10202	7660	7771	7660
query15	219	173	178	173
query16	7709	284	259	259
query17	1696	519	500	500
query18	1814	266	264	264
query19	197	154	157	154
query20	90	84	88	84
query21	202	138	136	136
query22	4344	3938	3940	3938
query23	33750	32957	33024	32957
query24	6904	2864	2781	2781
query25	580	353	375	353
query26	704	153	153	153
query27	2135	316	331	316
query28	4472	2034	2047	2034
query29	878	612	603	603
query30	258	173	175	173
query31	969	756	742	742
query32	90	53	57	53
query33	514	257	261	257
query34	870	468	477	468
query35	699	625	620	620
query36	1064	939	879	879
query37	116	74	72	72
query38	2901	2818	2804	2804
query39	837	792	799	792
query40	194	123	126	123
query41	43	41	44	41
query42	104	94	98	94
query43	590	554	554	554
query44	1061	715	729	715
query45	175	170	164	164
query46	1059	732	720	720
query47	1836	1768	1764	1764
query48	370	303	306	303
query49	839	375	387	375
query50	769	383	384	383
query51	6877	6799	6734	6734
query52	108	90	93	90
query53	361	294	290	290
query54	542	427	423	423
query55	77	74	73	73
query56	262	268	254	254
query57	1140	1061	1077	1061
query58	224	211	225	211
query59	3379	3147	3289	3147
query60	274	252	251	251
query61	93	85	86	85
query62	600	466	444	444
query63	314	289	290	289
query64	8504	2291	1790	1790
query65	3162	3105	3131	3105
query66	816	321	319	319
query67	15335	14744	14941	14744
query68	4517	531	536	531
query69	436	268	266	266
query70	1184	1134	1118	1118
query71	367	275	261	261
query72	7570	5753	5387	5387
query73	720	322	319	319
query74	6015	5574	5604	5574
query75	3298	2626	2637	2626
query76	2333	1065	1163	1065
query77	393	271	267	267
query78	10213	10041	9969	9969
query79	1505	509	527	509
query80	1050	442	427	427
query81	548	241	242	241
query82	957	98	95	95
query83	239	170	171	170
query84	245	88	86	86
query85	902	264	261	261
query86	384	289	299	289
query87	3361	3127	3137	3127
query88	3109	2325	2331	2325
query89	488	408	404	404
query90	1983	186	189	186
query91	177	97	96	96
query92	59	49	50	49
query93	1114	514	494	494
query94	1014	185	185	185
query95	414	308	317	308
query96	570	265	273	265
query97	3219	3045	3108	3045
query98	242	214	209	209
query99	1181	827	839	827
Total cold run time: 257272 ms
Total hot run time: 173065 ms

doris-robot · 2024-05-23T07:47:49Z

ClickBench: Total hot run time: 29.95 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 15416770af32580da3cefd3106aad66c7f7c9d8b, data reload: false
query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.66	0.08	0.08
query5	0.50	0.50	0.51
query6	1.13	0.72	0.72
query7	0.02	0.01	0.02
query8	0.06	0.05	0.04
query9	0.53	0.47	0.49
query10	0.55	0.54	0.54
query11	0.16	0.11	0.11
query12	0.14	0.11	0.12
query13	0.61	0.60	0.60
query14	0.78	0.78	0.78
query15	0.83	0.80	0.80
query16	0.36	0.37	0.38
query17	0.97	1.00	1.00
query18	0.23	0.21	0.26
query19	1.88	1.74	1.79
query20	0.01	0.01	0.02
query21	15.56	0.69	0.67
query22	4.07	7.65	1.40
query23	18.26	1.40	1.18
query24	1.71	0.29	0.23
query25	0.14	0.08	0.08
query26	0.27	0.16	0.16
query27	0.08	0.07	0.08
query28	13.29	0.99	1.00
query29	12.91	3.32	3.25
query30	0.24	0.06	0.05
query31	2.87	0.38	0.38
query32	3.30	0.47	0.46
query33	2.93	2.90	2.89
query34	17.10	4.40	4.46
query35	4.47	4.49	4.53
query36	0.67	0.48	0.46
query37	0.17	0.16	0.15
query38	0.15	0.15	0.14
query39	0.04	0.04	0.04
query40	0.16	0.13	0.15
query41	0.09	0.05	0.04
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.34 s
Total hot run time: 29.95 s

sollhui · 2024-05-23T09:08:05Z

run buildall

doris-robot · 2024-05-23T09:32:55Z

TPC-H: Total hot run time: 40809 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b3ed0e0b5ee19cc26b23c259553b0144f61ae26c, data reload: false
------ Round 1 ----------------------------------
q1	17701	4489	4232	4232
q2	2018	190	201	190
q3	10758	1303	1237	1237
q4	10226	787	806	787
q5	7524	2698	2832	2698
q6	224	136	135	135
q7	963	616	592	592
q8	9572	2105	2088	2088
q9	9322	6619	6591	6591
q10	9251	3681	3690	3681
q11	452	238	230	230
q12	462	224	229	224
q13	17767	3005	3002	3002
q14	249	221	221	221
q15	502	494	478	478
q16	527	375	383	375
q17	960	699	711	699
q18	8113	7404	7401	7401
q19	3629	1505	1512	1505
q20	673	301	309	301
q21	4986	3907	3869	3869
q22	344	292	273	273
Total cold run time: 116223 ms
Total hot run time: 40809 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4282	4295	4189	4189
q2	382	269	259	259
q3	3005	2771	2702	2702
q4	1871	1533	1570	1533
q5	5237	5206	5277	5206
q6	210	122	125	122
q7	2120	1726	1707	1707
q8	3178	3288	3269	3269
q9	8310	8325	8285	8285
q10	3876	3672	3678	3672
q11	584	485	485	485
q12	756	577	566	566
q13	16344	3005	2998	2998
q14	292	274	287	274
q15	512	485	472	472
q16	463	419	408	408
q17	1743	1462	1481	1462
q18	7719	7493	7476	7476
q19	1794	1535	1586	1535
q20	2007	1791	1765	1765
q21	4882	4611	4707	4611
q22	578	484	487	484
Total cold run time: 70145 ms
Total hot run time: 53480 ms

doris-robot · 2024-05-23T09:43:58Z

TPC-DS: Total hot run time: 169486 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b3ed0e0b5ee19cc26b23c259553b0144f61ae26c, data reload: false
query1	927	371	371	371
query2	6473	2487	2373	2373
query3	6691	202	203	202
query4	19254	17546	17559	17546
query5	4167	411	418	411
query6	251	149	150	149
query7	4578	308	291	291
query8	237	185	178	178
query9	8488	2463	2458	2458
query10	439	269	269	269
query11	10573	10223	10134	10134
query12	138	95	88	88
query13	1663	363	357	357
query14	9249	7315	6621	6621
query15	219	164	172	164
query16	7783	268	262	262
query17	1665	524	553	524
query18	1950	281	264	264
query19	227	155	160	155
query20	92	94	83	83
query21	202	125	128	125
query22	4166	3991	3948	3948
query23	33818	33211	32988	32988
query24	12255	2791	2854	2791
query25	652	363	356	356
query26	1808	158	162	158
query27	3069	345	311	311
query28	7814	2078	2089	2078
query29	1151	616	585	585
query30	305	176	174	174
query31	964	746	748	746
query32	94	53	52	52
query33	771	265	260	260
query34	1043	477	473	473
query35	730	604	599	599
query36	1103	931	905	905
query37	264	70	69	69
query38	2926	2763	2799	2763
query39	867	791	788	788
query40	279	124	120	120
query41	49	43	42	42
query42	101	93	95	93
query43	602	543	524	524
query44	1196	723	745	723
query45	176	162	160	160
query46	1081	758	700	700
query47	1850	1783	1779	1779
query48	366	287	301	287
query49	1216	377	385	377
query50	771	408	394	394
query51	6748	6653	6616	6616
query52	98	95	89	89
query53	355	281	275	275
query54	979	427	425	425
query55	96	70	72	70
query56	261	241	241	241
query57	1113	1011	1077	1011
query58	234	233	203	203
query59	3432	3081	3403	3081
query60	276	255	241	241
query61	91	84	88	84
query62	665	437	443	437
query63	306	279	276	276
query64	9896	2292	1793	1793
query65	3225	3101	3078	3078
query66	1389	348	348	348
query67	15254	14875	14869	14869
query68	5707	542	537	537
query69	504	283	295	283
query70	1201	1161	1078	1078
query71	466	272	264	264
query72	7884	2850	2670	2670
query73	745	327	326	326
query74	6104	5753	5667	5667
query75	3903	2608	2648	2608
query76	3677	937	991	937
query77	663	272	269	269
query78	10330	9970	9691	9691
query79	5724	514	515	514
query80	1108	461	446	446
query81	524	257	247	247
query82	777	96	99	96
query83	213	180	181	180
query84	272	89	92	89
query85	1487	321	308	308
query86	469	291	318	291
query87	3320	3089	3115	3089
query88	5106	2458	2439	2439
query89	505	390	378	378
query90	1991	329	184	184
query91	163	96	93	93
query92	62	50	49	49
query93	5072	510	498	498
query94	1136	187	189	187
query95	406	310	306	306
query96	605	275	277	275
query97	3187	3012	3027	3012
query98	235	229	210	210
query99	1225	860	862	860
Total cold run time: 285942 ms
Total hot run time: 169486 ms

doris-robot · 2024-05-23T09:49:29Z

ClickBench: Total hot run time: 30.31 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b3ed0e0b5ee19cc26b23c259553b0144f61ae26c, data reload: false
query1	0.04	0.04	0.03
query2	0.08	0.03	0.04
query3	0.22	0.05	0.04
query4	1.68	0.07	0.07
query5	0.50	0.50	0.51
query6	1.12	0.72	0.71
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.49	0.50
query10	0.55	0.54	0.54
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.60	0.60	0.60
query14	0.78	0.77	0.77
query15	0.82	0.80	0.80
query16	0.38	0.36	0.36
query17	1.03	1.02	1.02
query18	0.24	0.23	0.25
query19	1.90	1.69	1.82
query20	0.02	0.01	0.01
query21	15.46	0.69	0.69
query22	4.74	7.31	1.54
query23	18.29	1.39	1.32
query24	1.98	0.24	0.21
query25	0.15	0.08	0.09
query26	0.26	0.16	0.17
query27	0.08	0.08	0.07
query28	13.26	1.01	1.00
query29	12.69	3.30	3.27
query30	0.25	0.06	0.06
query31	2.87	0.38	0.37
query32	3.30	0.48	0.47
query33	2.82	2.97	2.87
query34	17.11	4.45	4.45
query35	4.49	4.52	4.52
query36	0.64	0.47	0.47
query37	0.18	0.16	0.17
query38	0.16	0.15	0.15
query39	0.04	0.04	0.03
query40	0.17	0.14	0.14
query41	0.08	0.04	0.05
query42	0.06	0.04	0.05
query43	0.03	0.03	0.04
Total cold run time: 109.99 s
Total hot run time: 30.31 s

sollhui · 2024-05-24T00:22:12Z

run buildall

sollhui · 2024-05-24T00:56:57Z

run buildall

doris-robot · 2024-05-24T01:15:21Z

TPC-H: Total hot run time: 40944 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b3ed0e0b5ee19cc26b23c259553b0144f61ae26c, data reload: false
------ Round 1 ----------------------------------
q1	4841	4683	4316	4316
q2	434	189	189	189
q3	5932	1113	1146	1113
q4	9086	750	767	750
q5	2694	2935	2797	2797
q6	225	136	135	135
q7	938	598	615	598
q8	2005	2059	2087	2059
q9	7255	6517	6464	6464
q10	3877	3735	3757	3735
q11	354	250	242	242
q12	372	214	218	214
q13	16349	3013	3123	3013
q14	265	221	226	221
q15	507	488	476	476
q16	500	403	392	392
q17	995	734	662	662
q18	8192	7695	7642	7642
q19	1587	1541	1469	1469
q20	532	329	318	318
q21	5035	3869	3915	3869
q22	346	270	281	270
Total cold run time: 72321 ms
Total hot run time: 40944 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4238	4244	4205	4205
q2	372	264	259	259
q3	2985	2757	2752	2752
q4	1875	1596	1585	1585
q5	5287	5348	5294	5294
q6	222	127	126	126
q7	2139	1734	1792	1734
q8	3209	3354	3305	3305
q9	8328	8302	8385	8302
q10	3891	3691	3650	3650
q11	583	497	489	489
q12	766	581	601	581
q13	13573	2997	2973	2973
q14	279	267	271	267
q15	526	475	473	473
q16	480	422	421	421
q17	1786	1478	1449	1449
q18	7587	7633	7549	7549
q19	1644	1542	1559	1542
q20	1953	1803	1760	1760
q21	4848	4888	4757	4757
q22	570	487	490	487
Total cold run time: 67141 ms
Total hot run time: 53960 ms

doris-robot · 2024-05-24T01:26:03Z

TPC-DS: Total hot run time: 169354 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b3ed0e0b5ee19cc26b23c259553b0144f61ae26c, data reload: false
query1	915	385	372	372
query2	6453	2422	2266	2266
query3	6657	212	208	208
query4	19743	17352	17233	17233
query5	4215	430	417	417
query6	243	156	151	151
query7	4580	307	299	299
query8	237	193	182	182
query9	8501	2365	2353	2353
query10	475	295	260	260
query11	10616	10233	10083	10083
query12	133	85	85	85
query13	1634	359	364	359
query14	10225	6853	6914	6853
query15	207	177	166	166
query16	7802	262	260	260
query17	1716	541	517	517
query18	1970	271	272	271
query19	194	155	159	155
query20	90	86	82	82
query21	198	129	123	123
query22	4246	3975	3899	3899
query23	33445	33138	32939	32939
query24	7862	2846	2859	2846
query25	576	372	360	360
query26	704	159	152	152
query27	2224	317	320	317
query28	5453	2024	2042	2024
query29	835	589	590	589
query30	283	169	176	169
query31	942	775	743	743
query32	92	51	54	51
query33	552	285	274	274
query34	869	475	483	475
query35	716	615	606	606
query36	1083	901	916	901
query37	108	70	74	70
query38	2898	2809	2797	2797
query39	834	819	819	819
query40	199	130	130	130
query41	53	48	45	45
query42	105	99	106	99
query43	586	567	535	535
query44	1125	724	743	724
query45	175	163	169	163
query46	1076	733	732	732
query47	1839	1777	1803	1777
query48	396	296	294	294
query49	1042	374	388	374
query50	784	404	386	386
query51	6981	6740	6696	6696
query52	102	89	93	89
query53	350	287	298	287
query54	540	427	418	418
query55	74	74	73	73
query56	263	243	254	243
query57	1141	1010	1092	1010
query58	222	201	214	201
query59	3329	3294	3234	3234
query60	274	250	247	247
query61	93	93	91	91
query62	613	450	441	441
query63	309	284	288	284
query64	8550	2235	1798	1798
query65	3196	3120	3103	3103
query66	802	328	330	328
query67	15095	14807	14894	14807
query68	4537	551	544	544
query69	437	263	261	261
query70	1151	1106	1063	1063
query71	331	261	267	261
query72	7869	5310	2737	2737
query73	722	320	321	320
query74	6069	5626	5709	5626
query75	3254	2679	2670	2670
query76	2332	975	925	925
query77	406	266	268	266
query78	10247	9769	9779	9769
query79	2287	521	511	511
query80	1117	434	430	430
query81	530	254	242	242
query82	637	92	95	92
query83	243	174	176	174
query84	239	89	81	81
query85	1635	279	270	270
query86	501	315	304	304
query87	3363	3092	3165	3092
query88	4188	2343	2318	2318
query89	482	391	393	391
query90	1961	196	192	192
query91	121	99	110	99
query92	66	52	54	52
query93	2419	516	495	495
query94	1251	196	198	196
query95	489	310	309	309
query96	590	263	261	261
query97	3236	2985	3026	2985
query98	244	221	222	221
query99	1242	862	870	862
Total cold run time: 264036 ms
Total hot run time: 169354 ms

doris-robot · 2024-05-24T01:31:34Z

ClickBench: Total hot run time: 30.66 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b3ed0e0b5ee19cc26b23c259553b0144f61ae26c, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.10	0.07
query5	0.47	0.48	0.50
query6	1.13	0.72	0.73
query7	0.02	0.01	0.01
query8	0.04	0.04	0.04
query9	0.54	0.49	0.48
query10	0.54	0.55	0.54
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.60
query14	0.77	0.78	0.79
query15	0.83	0.81	0.80
query16	0.36	0.38	0.37
query17	0.97	1.03	1.02
query18	0.22	0.26	0.24
query19	1.88	1.84	1.79
query20	0.01	0.01	0.01
query21	15.76	0.67	0.66
query22	4.24	7.90	1.78
query23	18.43	1.35	1.29
query24	1.82	0.25	0.20
query25	0.14	0.08	0.08
query26	0.26	0.17	0.17
query27	0.08	0.08	0.09
query28	13.34	1.02	0.99
query29	12.92	3.32	3.27
query30	0.24	0.06	0.06
query31	2.88	0.39	0.37
query32	3.29	0.47	0.47
query33	2.86	2.92	2.92
query34	17.07	4.47	4.51
query35	4.57	4.51	4.49
query36	0.68	0.49	0.49
query37	0.18	0.15	0.14
query38	0.15	0.15	0.15
query39	0.04	0.04	0.03
query40	0.16	0.14	0.14
query41	0.10	0.05	0.05
query42	0.05	0.05	0.04
query43	0.04	0.03	0.03
Total cold run time: 109.99 s
Total hot run time: 30.66 s

sollhui · 2024-05-24T03:16:25Z

run buildall

doris-robot · 2024-05-24T03:53:11Z

TPC-H: Total hot run time: 40927 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0b1c310b5bb8ee9d98a473597ba01a11ff72e3a4, data reload: false
------ Round 1 ----------------------------------
q1	17591	4749	4236	4236
q2	2018	193	192	192
q3	10457	1261	1161	1161
q4	10210	858	731	731
q5	7486	2774	2690	2690
q6	229	135	133	133
q7	951	611	624	611
q8	9227	2126	2105	2105
q9	9116	6675	6718	6675
q10	9282	3879	3846	3846
q11	475	241	236	236
q12	480	227	216	216
q13	17423	3219	3245	3219
q14	252	212	217	212
q15	505	469	463	463
q16	500	389	403	389
q17	979	724	702	702
q18	8402	7808	7744	7744
q19	4731	1522	1538	1522
q20	637	321	323	321
q21	5124	4044	3252	3252
q22	347	281	271	271
Total cold run time: 116422 ms
Total hot run time: 40927 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4607	4426	4407	4407
q2	371	276	271	271
q3	3186	2947	2968	2947
q4	1931	1572	1605	1572
q5	5435	5482	5489	5482
q6	216	130	132	130
q7	2147	1847	1815	1815
q8	3241	3390	3392	3390
q9	8540	8642	8617	8617
q10	4066	3726	3791	3726
q11	584	533	499	499
q12	799	592	620	592
q13	17030	3133	3145	3133
q14	295	267	270	267
q15	541	486	495	486
q16	500	447	449	447
q17	1805	1475	1469	1469
q18	7684	7497	7507	7497
q19	1658	1472	1555	1472
q20	2007	1786	1771	1771
q21	4885	4740	4603	4603
q22	576	482	502	482
Total cold run time: 72104 ms
Total hot run time: 55075 ms

doris-robot · 2024-05-24T04:03:32Z

TPC-DS: Total hot run time: 168773 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0b1c310b5bb8ee9d98a473597ba01a11ff72e3a4, data reload: false
query1	915	368	372	368
query2	6445	2536	2340	2340
query3	6650	204	207	204
query4	19780	17496	17357	17357
query5	4101	411	410	410
query6	245	158	159	158
query7	4590	296	296	296
query8	241	187	186	186
query9	8466	2387	2365	2365
query10	445	279	273	273
query11	10672	10103	10040	10040
query12	138	85	86	85
query13	1630	360	354	354
query14	9200	8186	6906	6906
query15	205	188	169	169
query16	7225	267	267	267
query17	1337	515	525	515
query18	1909	286	269	269
query19	209	173	155	155
query20	95	84	88	84
query21	194	136	129	129
query22	4181	3948	3935	3935
query23	33431	32926	33068	32926
query24	5049	2772	2888	2772
query25	492	349	359	349
query26	692	154	156	154
query27	1825	322	314	314
query28	3649	2028	2050	2028
query29	825	614	600	600
query30	243	176	177	176
query31	918	750	728	728
query32	60	53	57	53
query33	459	291	263	263
query34	841	462	461	461
query35	720	593	587	587
query36	1013	902	870	870
query37	107	67	70	67
query38	2901	2790	2720	2720
query39	874	802	790	790
query40	191	128	121	121
query41	49	44	43	43
query42	104	94	95	94
query43	576	554	555	554
query44	1037	713	744	713
query45	182	161	164	161
query46	1045	727	716	716
query47	1852	1786	1768	1768
query48	363	287	281	281
query49	764	399	379	379
query50	767	383	380	380
query51	6873	6809	6725	6725
query52	97	91	87	87
query53	347	287	284	284
query54	523	437	421	421
query55	74	70	69	69
query56	257	237	254	237
query57	1115	1038	1062	1038
query58	222	206	209	206
query59	3638	3136	3130	3130
query60	272	256	247	247
query61	115	110	120	110
query62	569	442	436	436
query63	305	281	283	281
query64	2510	1757	1673	1673
query65	3163	3086	3110	3086
query66	674	364	328	328
query67	15070	14799	15041	14799
query68	4591	524	543	524
query69	503	266	274	266
query70	1113	1091	1133	1091
query71	433	274	263	263
query72	7658	2763	2551	2551
query73	731	325	317	317
query74	5980	5588	5583	5583
query75	3272	2633	2639	2633
query76	2217	992	949	949
query77	387	261	263	261
query78	10132	9958	9688	9688
query79	1237	505	514	505
query80	946	437	420	420
query81	527	237	236	236
query82	1132	96	96	96
query83	237	178	174	174
query84	244	84	82	82
query85	908	326	312	312
query86	409	291	309	291
query87	3327	3133	3246	3133
query88	3467	2308	2305	2305
query89	466	382	396	382
query90	1934	180	182	180
query91	120	95	95	95
query92	59	49	47	47
query93	987	505	491	491
query94	1102	193	179	179
query95	399	306	303	303
query96	570	256	259	256
query97	3220	2999	2993	2993
query98	234	218	218	218
query99	1180	859	861	859
Total cold run time: 245448 ms
Total hot run time: 168773 ms

doris-robot · 2024-05-24T04:09:01Z

ClickBench: Total hot run time: 30.6 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0b1c310b5bb8ee9d98a473597ba01a11ff72e3a4, data reload: false
query1	0.04	0.03	0.03
query2	0.09	0.03	0.04
query3	0.24	0.05	0.05
query4	1.68	0.06	0.07
query5	0.48	0.50	0.49
query6	1.12	0.71	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.53	0.48	0.49
query10	0.56	0.54	0.54
query11	0.16	0.11	0.11
query12	0.14	0.12	0.12
query13	0.59	0.59	0.59
query14	0.77	0.77	0.78
query15	0.84	0.82	0.80
query16	0.38	0.35	0.37
query17	0.96	1.04	1.02
query18	0.22	0.25	0.23
query19	1.72	1.74	1.72
query20	0.01	0.01	0.01
query21	15.64	0.67	0.66
query22	4.40	6.73	2.08
query23	18.28	1.46	1.27
query24	1.38	0.35	0.20
query25	0.14	0.09	0.08
query26	0.25	0.16	0.16
query27	0.08	0.08	0.07
query28	13.50	1.01	1.00
query29	13.80	3.26	3.24
query30	0.24	0.06	0.05
query31	2.88	0.38	0.37
query32	3.29	0.46	0.47
query33	2.85	2.95	2.90
query34	17.05	4.39	4.42
query35	4.46	4.46	4.59
query36	0.66	0.46	0.45
query37	0.18	0.15	0.15
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.16	0.14	0.14
query41	0.09	0.04	0.04
query42	0.06	0.04	0.05
query43	0.03	0.04	0.04
Total cold run time: 110.22 s
Total hot run time: 30.6 s

dataroaring · 2024-06-05T01:12:37Z

run buildall

doris-robot · 2024-06-05T01:46:56Z

TPC-H: Total hot run time: 40861 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a9493a1e3f496afd7e9bfda6c322cabe81d1ce43, data reload: false
------ Round 1 ----------------------------------
q1	17633	4283	4254	4254
q2	2023	190	189	189
q3	10462	1206	1082	1082
q4	10188	883	785	785
q5	7488	2708	2691	2691
q6	224	140	138	138
q7	981	625	607	607
q8	9359	2105	2112	2105
q9	9261	6739	6647	6647
q10	9400	3859	3898	3859
q11	445	248	247	247
q12	428	224	229	224
q13	18218	3227	3236	3227
q14	268	207	222	207
q15	507	476	496	476
q16	524	428	403	403
q17	966	721	620	620
q18	8275	7794	7792	7792
q19	7571	1507	1410	1410
q20	645	324	338	324
q21	5092	3240	4011	3240
q22	402	334	336	334
Total cold run time: 120360 ms
Total hot run time: 40861 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4631	4446	4404	4404
q2	361	267	270	267
q3	3194	2996	2856	2856
q4	1885	1598	1656	1598
q5	5496	5511	5538	5511
q6	221	130	131	130
q7	2188	1845	1858	1845
q8	3212	3349	3357	3349
q9	8618	8669	8738	8669
q10	4061	3802	3685	3685
q11	578	487	488	487
q12	813	655	639	639
q13	15897	3099	3115	3099
q14	330	269	259	259
q15	522	467	473	467
q16	484	427	458	427
q17	1813	1500	1495	1495
q18	8138	7651	7359	7359
q19	1738	1515	1460	1460
q20	3049	1842	1790	1790
q21	4831	4575	4786	4575
q22	767	545	525	525
Total cold run time: 72827 ms
Total hot run time: 54896 ms

doris-robot · 2024-06-05T01:57:48Z

TPC-DS: Total hot run time: 170686 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a9493a1e3f496afd7e9bfda6c322cabe81d1ce43, data reload: false
query1	912	377	366	366
query2	6466	2431	2238	2238
query3	6638	209	212	209
query4	20808	17371	17316	17316
query5	4173	447	462	447
query6	249	151	156	151
query7	4578	306	288	288
query8	313	271	272	271
query9	8625	2465	2423	2423
query10	437	294	296	294
query11	10610	9986	10153	9986
query12	132	93	87	87
query13	1632	362	357	357
query14	9345	7586	7451	7451
query15	224	195	186	186
query16	7513	266	267	266
query17	1373	511	510	510
query18	1879	297	271	271
query19	195	164	154	154
query20	92	83	83	83
query21	209	128	127	127
query22	4219	4087	4032	4032
query23	33370	32881	33068	32881
query24	11178	2860	2843	2843
query25	611	352	360	352
query26	1342	156	158	156
query27	3018	319	323	319
query28	7514	2065	2068	2065
query29	912	606	593	593
query30	282	147	146	146
query31	971	765	745	745
query32	93	53	55	53
query33	772	321	270	270
query34	948	470	478	470
query35	742	660	628	628
query36	1086	939	924	924
query37	150	70	76	70
query38	2875	2752	2749	2749
query39	862	797	773	773
query40	201	123	120	120
query41	54	50	49	49
query42	114	95	99	95
query43	587	552	542	542
query44	1205	730	740	730
query45	200	166	167	166
query46	1064	757	715	715
query47	1833	1764	1782	1764
query48	374	301	312	301
query49	1083	403	429	403
query50	776	394	403	394
query51	6838	6668	6695	6668
query52	103	91	87	87
query53	352	294	293	293
query54	843	447	439	439
query55	76	74	71	71
query56	275	253	249	249
query57	1110	1035	1078	1035
query58	254	250	253	250
query59	3533	3300	3291	3291
query60	314	266	270	266
query61	86	110	110	110
query62	689	442	450	442
query63	310	290	287	287
query64	8868	2208	1712	1712
query65	3152	3068	3102	3068
query66	881	333	342	333
query67	15441	15033	14940	14940
query68	4575	568	549	549
query69	489	311	317	311
query70	1161	1157	1130	1130
query71	397	291	289	289
query72	7260	2861	2660	2660
query73	759	334	332	332
query74	5860	5517	5614	5517
query75	3380	2663	2663	2663
query76	2568	923	908	908
query77	469	301	291	291
query78	10399	9966	10081	9966
query79	2511	521	523	521
query80	2151	491	477	477
query81	568	227	220	220
query82	625	105	99	99
query83	280	201	176	176
query84	264	94	90	90
query85	1700	319	306	306
query86	487	326	274	274
query87	3342	3129	3057	3057
query88	3897	2471	2449	2449
query89	486	388	371	371
query90	1827	191	190	190
query91	135	111	109	109
query92	59	50	52	50
query93	3035	521	505	505
query94	1253	203	200	200
query95	425	320	325	320
query96	595	269	277	269
query97	3172	3001	2967	2967
query98	249	223	219	219
query99	1303	836	831	831
Total cold run time: 273645 ms
Total hot run time: 170686 ms

doris-robot · 2024-06-05T02:03:21Z

ClickBench: Total hot run time: 31.17 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a9493a1e3f496afd7e9bfda6c322cabe81d1ce43, data reload: false
query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.07
query4	1.66	0.08	0.07
query5	0.50	0.50	0.49
query6	1.13	0.72	0.74
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.53	0.49	0.48
query10	0.54	0.55	0.54
query11	0.15	0.12	0.11
query12	0.14	0.12	0.12
query13	0.59	0.59	0.59
query14	0.79	0.78	0.79
query15	0.84	0.81	0.82
query16	0.38	0.37	0.36
query17	1.02	0.95	0.97
query18	0.22	0.25	0.26
query19	1.76	1.79	1.72
query20	0.01	0.02	0.01
query21	15.49	0.69	0.66
query22	4.18	7.20	2.35
query23	18.62	1.33	1.31
query24	1.86	0.24	0.22
query25	0.15	0.09	0.09
query26	0.26	0.17	0.18
query27	0.08	0.08	0.08
query28	13.27	1.03	0.99
query29	13.42	3.38	3.33
query30	0.24	0.06	0.06
query31	2.86	0.37	0.40
query32	3.28	0.48	0.46
query33	2.89	2.95	2.91
query34	17.25	4.43	4.47
query35	4.46	4.50	4.57
query36	0.69	0.51	0.47
query37	0.17	0.16	0.15
query38	0.15	0.14	0.14
query39	0.04	0.04	0.03
query40	0.16	0.15	0.15
query41	0.09	0.04	0.05
query42	0.05	0.04	0.04
query43	0.03	0.03	0.04
Total cold run time: 110.37 s
Total hot run time: 31.17 s

sollhui · 2024-06-05T11:38:17Z

run buildall

doris-robot · 2024-06-05T12:31:14Z

TPC-H: Total hot run time: 40628 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e139ddd1b0a688e7c35cd5b9717eb76a6f4abc4f, data reload: false
------ Round 1 ----------------------------------
q1	17624	5282	4235	4235
q2	2020	189	194	189
q3	10453	1142	1118	1118
q4	10205	737	827	737
q5	7465	2737	2758	2737
q6	224	134	134	134
q7	962	623	611	611
q8	9234	2123	2080	2080
q9	9254	6671	6751	6671
q10	9627	3853	3864	3853
q11	433	228	233	228
q12	503	245	239	239
q13	17496	3175	3250	3175
q14	266	209	218	209
q15	513	479	473	473
q16	489	376	398	376
q17	990	666	608	608
q18	8424	7754	7710	7710
q19	1706	1395	1598	1395
q20	616	320	316	316
q21	5124	3194	3274	3194
q22	395	345	340	340
Total cold run time: 114023 ms
Total hot run time: 40628 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4520	4433	4451	4433
q2	390	268	262	262
q3	3059	2945	2891	2891
q4	1964	1727	1617	1617
q5	5275	5435	5494	5435
q6	225	126	127	126
q7	2144	1806	1822	1806
q8	3202	3330	3332	3330
q9	8574	8555	8532	8532
q10	4077	3848	3842	3842
q11	607	474	491	474
q12	784	577	608	577
q13	16251	3062	3167	3062
q14	301	256	276	256
q15	509	476	495	476
q16	466	419	418	418
q17	1794	1499	1495	1495
q18	7995	7796	7373	7373
q19	1809	1631	1547	1547
q20	3033	1764	1767	1764
q21	4722	4743	4759	4743
q22	640	553	556	553
Total cold run time: 72341 ms
Total hot run time: 55012 ms

liaoxin01

LGTM

github-actions · 2024-06-07T11:13:24Z

PR approved by at least one committer and no changes requested.

sollhui · 2024-06-07T12:13:02Z

run buildall

doris-robot · 2024-06-07T12:49:58Z

TPC-H: Total hot run time: 39550 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0e639e78d9d94ea8fd8a1245c84717525e87e5fd, data reload: false
------ Round 1 ----------------------------------
q1	17630	4315	4246	4246
q2	2020	196	217	196
q3	10427	1096	1035	1035
q4	10195	830	836	830
q5	7457	2628	2589	2589
q6	219	135	137	135
q7	940	612	607	607
q8	9220	2034	2039	2034
q9	8745	6471	6460	6460
q10	8996	3727	3731	3727
q11	436	237	234	234
q12	425	233	222	222
q13	17758	3028	2984	2984
q14	260	216	216	216
q15	514	481	472	472
q16	515	388	372	372
q17	947	656	757	656
q18	7955	7420	7303	7303
q19	7602	1431	1516	1431
q20	657	323	320	320
q21	4936	3146	3969	3146
q22	392	335	339	335
Total cold run time: 118246 ms
Total hot run time: 39550 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4433	4234	4252	4234
q2	375	256	268	256
q3	2989	2951	2907	2907
q4	1942	1689	1766	1689
q5	5599	5488	5465	5465
q6	228	128	131	128
q7	2166	1847	1817	1817
q8	3305	3384	3404	3384
q9	8647	8699	8784	8699
q10	4072	3873	3698	3698
q11	603	505	490	490
q12	840	656	635	635
q13	17085	3195	3110	3110
q14	312	261	282	261
q15	517	477	478	477
q16	476	431	431	431
q17	1800	1504	1469	1469
q18	8090	7900	7701	7701
q19	1766	1602	1520	1520
q20	3048	1930	1838	1838
q21	5130	5044	4776	4776
q22	801	564	554	554
Total cold run time: 74224 ms
Total hot run time: 55539 ms

doris-robot · 2024-06-07T13:00:57Z

TPC-DS: Total hot run time: 173546 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0e639e78d9d94ea8fd8a1245c84717525e87e5fd, data reload: false
query1	927	374	375	374
query2	6443	2597	2351	2351
query3	6626	207	205	205
query4	21038	17832	17421	17421
query5	3601	480	466	466
query6	250	167	162	162
query7	4587	295	287	287
query8	321	303	292	292
query9	8489	2399	2387	2387
query10	585	294	289	289
query11	10607	9940	10016	9940
query12	124	90	91	90
query13	1644	379	366	366
query14	9753	7842	7371	7371
query15	241	195	190	190
query16	7744	288	264	264
query17	1838	559	510	510
query18	1946	279	268	268
query19	189	154	158	154
query20	90	82	84	82
query21	205	129	130	129
query22	4215	4121	4010	4010
query23	33712	33527	33458	33458
query24	11025	2923	2938	2923
query25	622	373	365	365
query26	1249	158	165	158
query27	3078	332	327	327
query28	7743	2105	2079	2079
query29	909	650	596	596
query30	263	157	159	157
query31	971	767	773	767
query32	93	52	55	52
query33	736	304	276	276
query34	1010	488	494	488
query35	754	644	624	624
query36	1147	978	964	964
query37	161	74	72	72
query38	2972	2795	2824	2795
query39	907	863	873	863
query40	224	126	125	125
query41	53	52	71	52
query42	120	108	102	102
query43	570	514	565	514
query44	1249	726	741	726
query45	201	160	164	160
query46	1100	714	702	702
query47	1858	1757	1750	1750
query48	373	296	292	292
query49	845	415	430	415
query50	778	390	398	390
query51	6791	6681	6743	6681
query52	110	91	92	91
query53	364	292	295	292
query54	871	443	439	439
query55	73	75	73	73
query56	291	258	270	258
query57	1108	1066	1047	1047
query58	262	249	243	243
query59	3415	3321	3047	3047
query60	301	266	296	266
query61	123	86	83	83
query62	621	419	448	419
query63	316	297	297	297
query64	8749	2199	1767	1767
query65	3571	3145	3079	3079
query66	939	326	340	326
query67	15200	14971	14809	14809
query68	4592	534	551	534
query69	667	460	391	391
query70	1171	1152	1099	1099
query71	436	277	276	276
query72	7048	5484	5608	5484
query73	762	321	319	319
query74	5924	5416	5393	5393
query75	3475	2669	2701	2669
query76	2886	984	892	892
query77	628	306	290	290
query78	10218	9903	9742	9742
query79	2136	507	518	507
query80	869	475	461	461
query81	576	219	219	219
query82	741	100	102	100
query83	270	168	171	168
query84	233	88	87	87
query85	1622	272	283	272
query86	467	320	279	279
query87	3258	3057	3021	3021
query88	4051	2394	2308	2308
query89	487	384	374	374
query90	1724	196	190	190
query91	124	98	95	95
query92	60	50	50	50
query93	2320	509	508	508
query94	1090	187	184	184
query95	407	327	320	320
query96	594	271	264	264
query97	3171	3021	3011	3011
query98	220	202	201	201
query99	1300	821	826	821
Total cold run time: 272818 ms
Total hot run time: 173546 ms

dataroaring

LGTM

github-actions · 2024-06-07T13:04:19Z

PR approved by at least one committer and no changes requested.

doris-robot · 2024-06-07T13:06:28Z

ClickBench: Total hot run time: 30.66 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0e639e78d9d94ea8fd8a1245c84717525e87e5fd, data reload: false
query1	0.04	0.03	0.04
query2	0.08	0.03	0.04
query3	0.24	0.04	0.05
query4	1.67	0.07	0.07
query5	0.50	0.49	0.49
query6	1.12	0.72	0.72
query7	0.02	0.01	0.01
query8	0.04	0.04	0.04
query9	0.54	0.48	0.48
query10	0.54	0.55	0.54
query11	0.16	0.12	0.12
query12	0.14	0.12	0.12
query13	0.60	0.59	0.61
query14	0.80	0.79	0.78
query15	0.83	0.82	0.82
query16	0.34	0.37	0.37
query17	0.98	1.03	0.95
query18	0.21	0.25	0.22
query19	1.77	1.70	1.67
query20	0.01	0.01	0.01
query21	15.40	0.66	0.65
query22	4.45	7.30	2.08
query23	18.22	1.44	1.27
query24	2.10	0.21	0.22
query25	0.15	0.09	0.09
query26	0.26	0.18	0.17
query27	0.09	0.09	0.09
query28	13.23	1.02	1.00
query29	12.63	3.29	3.24
query30	0.27	0.06	0.06
query31	2.86	0.40	0.38
query32	3.27	0.47	0.48
query33	2.92	2.90	2.94
query34	17.18	4.42	4.40
query35	4.49	4.47	4.47
query36	0.66	0.46	0.48
query37	0.18	0.15	0.15
query38	0.15	0.15	0.15
query39	0.04	0.04	0.03
query40	0.17	0.16	0.15
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.05
Total cold run time: 109.55 s
Total hot run time: 30.66 s

…35266) ## Proposed changes When some exception occur, such as `get offset failed ` for network isolation or be node down when upgrade, job will pause unexpectedly. Therefore, Doris introduce auto resume to keep job stable in #2958. **But currently there are some issues with this policy:** - If the fault recovery time is relatively long, such as the downtime of BE, then after auto resume three times, it will be completely unable to automatically resume. But in reality, it can be restored by automatically resume. **Improvement measures:** - Unrestricted retry attempts to solve the problem of automatic resume caused by long recovery time. - Add the backoff algorithm to solve the problem of some resume being invalid, such as Kafka having problems, but Doris still having a high resume frequency. The maximum backoff time is five minutes to avoid the difficulty of recovery caused by too much backoff time.

…cy (#37373) pick #35266

…35266) (#37453) pick (#35266)

…pache#35266) (apache#37453) pick (apache#35266)

…pache#35266) ## Proposed changes When some exception occur, such as `get offset failed ` for network isolation or be node down when upgrade, job will pause unexpectedly. Therefore, Doris introduce auto resume to keep job stable in apache#2958. **But currently there are some issues with this policy:** - If the fault recovery time is relatively long, such as the downtime of BE, then after auto resume three times, it will be completely unable to automatically resume. But in reality, it can be restored by automatically resume. **Improvement measures:** - Unrestricted retry attempts to solve the problem of automatic resume caused by long recovery time. - Add the backoff algorithm to solve the problem of some resume being invalid, such as Kafka having problems, but Doris still having a high resume frequency. The maximum backoff time is five minutes to avoid the difficulty of recovery caused by too much backoff time.

sollhui changed the title ~~[improve](routine-load) use backoff algorithm in auto resume~~[improve](routine-load) optimize auto resume policy to make job stableJun 5, 2024

sollhui changed the title ~~[improve](routine-load) optimize auto resume policy to make job stable~~[improve](routine-load) optimize routine load job auto resume policyJun 5, 2024

liaoxin01 previously approved these changes Jun 7, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jun 7, 2024

optimize routine load job auto resume policy
0e639e7

sollhui dismissed liaoxin01’s stale review via 0e639e7June 7, 2024 12:12

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Jun 7, 2024

dataroaring approved these changes Jun 7, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jun 7, 2024

dataroaring added the dev/2.1.x label Jun 7, 2024

liaoxin01 approved these changes Jun 7, 2024
View reviewed changes

dataroaring merged commit 1492a55 into apache:masterJun 7, 2024

dataroaring added the dev/2.0.x label Jun 11, 2024

sollhui mentioned this pull request Jul 6, 2024
[branch-2.1](routine-load) optimize routine load job auto resume policy #37373
Merged

dataroaring pushed a commit that referenced this pull request Jul 7, 2024
[branch-2.1](routine-load) optimize routine load job auto resume poli…
d08a418
…cy (#37373) pick #35266

dataroaring added dev/2.1.5-merged and removed dev/2.1.x labels Jul 7, 2024

sollhui mentioned this pull request Jul 8, 2024
[improve](routine-load) optimize routine load job auto resume policy (#35266) #37453
Merged

dataroaring pushed a commit that referenced this pull request Jul 8, 2024
[improve](routine-load) optimize routine load job auto resume policy (#…
bcb6eac
…35266) (#37453) pick (#35266)

dataroaring added dev/2.0.13-merged and removed dev/2.0.x labels Jul 8, 2024

xiaokang mentioned this pull request Jul 14, 2024
Release Note 2.0.13 #37753
Closed

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
[improve](routine-load) optimize routine load job auto resume policy (a…
b7c3b1e
…pache#35266) (apache#37453) pick (apache#35266)

Uh oh!

Conversation

sollhui commented May 23, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

sollhui commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

sollhui commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

sollhui commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

sollhui commented May 24, 2024

Uh oh!

sollhui commented May 24, 2024

Uh oh!

doris-robot commented May 24, 2024

Uh oh!

doris-robot commented May 24, 2024

Uh oh!

doris-robot commented May 24, 2024

Uh oh!

sollhui commented May 24, 2024

Uh oh!

doris-robot commented May 24, 2024

Uh oh!

doris-robot commented May 24, 2024

Uh oh!

doris-robot commented May 24, 2024

Uh oh!

dataroaring commented Jun 5, 2024

Uh oh!

doris-robot commented Jun 5, 2024

Uh oh!

doris-robot commented Jun 5, 2024

Uh oh!

doris-robot commented Jun 5, 2024

Uh oh!

sollhui commented Jun 5, 2024

Uh oh!

doris-robot commented Jun 5, 2024

Uh oh!

liaoxin01 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jun 7, 2024

Uh oh!

sollhui commented Jun 7, 2024

Uh oh!

doris-robot commented Jun 7, 2024

Uh oh!

doris-robot commented Jun 7, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jun 7, 2024

Uh oh!

doris-robot commented Jun 7, 2024

Uh oh!

sollhui commented May 23, 2024•
edited
Loading