[enhancement](export) filter empty partition before export table to remote storage by caiconghui · Pull Request #35389 · apache/doris

caiconghui · 2024-05-25T03:45:09Z

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-05-25T03:45:15Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

caiconghui · 2024-05-25T03:45:22Z

run buildall

doris-robot · 2024-05-25T04:23:11Z

TPC-H: Total hot run time: 41729 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 18fd1f5219b8e325c39637e15ac0f121efeadbf8, data reload: false
------ Round 1 ----------------------------------
q1	17608	4311	4253	4253
q2	2001	193	189	189
q3	10454	1248	1226	1226
q4	10207	762	777	762
q5	7466	2753	2596	2596
q6	216	133	135	133
q7	972	627	600	600
q8	9219	2130	2081	2081
q9	9695	6682	6748	6682
q10	9876	3922	3988	3922
q11	448	248	247	247
q12	512	227	224	224
q13	18419	3232	3123	3123
q14	251	223	218	218
q15	506	482	464	464
q16	513	399	392	392
q17	976	682	711	682
q18	8419	7949	7841	7841
q19	5595	1596	1563	1563
q20	646	345	327	327
q21	5182	3933	3924	3924
q22	354	280	285	280
Total cold run time: 119535 ms
Total hot run time: 41729 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4637	4401	4456	4401
q2	385	271	271	271
q3	3155	2888	2817	2817
q4	1933	1708	1730	1708
q5	5493	5501	5468	5468
q6	223	126	125	125
q7	2221	1832	1836	1832
q8	3286	3381	3363	3363
q9	8701	8707	8491	8491
q10	4065	3842	3822	3822
q11	586	479	482	479
q12	797	647	656	647
q13	16076	3209	3130	3130
q14	285	261	274	261
q15	528	462	477	462
q16	453	417	410	410
q17	1742	1481	1473	1473
q18	7939	7598	7545	7545
q19	1658	1554	1551	1551
q20	2006	1759	1747	1747
q21	9882	4678	4720	4678
q22	576	482	460	460
Total cold run time: 76627 ms
Total hot run time: 55141 ms

doris-robot · 2024-05-25T04:33:36Z

TPC-DS: Total hot run time: 168379 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 18fd1f5219b8e325c39637e15ac0f121efeadbf8, data reload: false
query1	919	384	364	364
query2	6427	2487	2185	2185
query3	6659	211	212	211
query4	19659	17362	17291	17291
query5	4126	427	407	407
query6	246	165	170	165
query7	4596	290	290	290
query8	235	183	183	183
query9	8357	2385	2364	2364
query10	451	270	260	260
query11	10715	10105	10090	10090
query12	137	90	84	84
query13	1632	356	374	356
query14	9243	6870	6664	6664
query15	208	174	177	174
query16	7128	263	253	253
query17	1036	514	506	506
query18	1905	279	262	262
query19	207	153	151	151
query20	89	90	88	88
query21	194	127	128	127
query22	4160	4079	3966	3966
query23	33411	33054	32844	32844
query24	5036	2795	2870	2795
query25	459	353	362	353
query26	686	151	157	151
query27	1856	325	321	321
query28	3669	2061	2082	2061
query29	850	602	641	602
query30	251	169	182	169
query31	893	760	749	749
query32	59	55	53	53
query33	427	270	262	262
query34	852	471	476	471
query35	690	588	579	579
query36	1052	956	960	956
query37	109	70	70	70
query38	2838	2771	2750	2750
query39	875	805	788	788
query40	194	125	123	123
query41	45	45	44	44
query42	101	96	96	96
query43	584	558	527	527
query44	1089	733	749	733
query45	181	163	163	163
query46	1058	710	743	710
query47	1874	1736	1776	1736
query48	363	303	293	293
query49	758	417	384	384
query50	767	374	393	374
query51	7004	6748	6705	6705
query52	104	89	90	89
query53	360	287	278	278
query54	525	446	421	421
query55	69	70	73	70
query56	254	246	243	243
query57	1084	1033	1046	1033
query58	216	206	226	206
query59	3482	3108	3045	3045
query60	263	243	249	243
query61	87	87	118	87
query62	563	436	467	436
query63	307	284	281	281
query64	2552	1701	1707	1701
query65	3163	3070	3108	3070
query66	652	320	319	319
query67	15308	14691	14663	14663
query68	4560	536	547	536
query69	459	286	269	269
query70	1169	1130	1133	1130
query71	413	270	268	268
query72	7863	5506	2737	2737
query73	723	317	314	314
query74	6048	5598	5643	5598
query75	3310	2613	2634	2613
query76	2306	985	927	927
query77	399	265	264	264
query78	10389	9922	9643	9643
query79	2349	512	519	512
query80	913	430	435	430
query81	508	244	241	241
query82	728	97	94	94
query83	245	173	169	169
query84	266	85	83	83
query85	993	268	263	263
query86	440	300	287	287
query87	3268	3193	3112	3112
query88	4245	2399	2347	2347
query89	477	391	373	373
query90	2041	187	196	187
query91	121	94	98	94
query92	61	50	47	47
query93	2031	512	497	497
query94	1150	188	180	180
query95	401	301	340	301
query96	590	274	267	267
query97	3150	3036	3039	3036
query98	246	227	215	215
query99	1188	857	858	857
Total cold run time: 248399 ms
Total hot run time: 168379 ms

doris-robot · 2024-05-25T04:39:06Z

ClickBench: Total hot run time: 30.39 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 18fd1f5219b8e325c39637e15ac0f121efeadbf8, data reload: false
query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.23	0.05	0.04
query4	1.68	0.06	0.06
query5	0.49	0.49	0.50
query6	1.13	0.72	0.71
query7	0.01	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.50	0.48
query10	0.55	0.56	0.55
query11	0.16	0.10	0.11
query12	0.15	0.12	0.12
query13	0.60	0.58	0.60
query14	0.76	0.79	0.77
query15	0.83	0.81	0.81
query16	0.37	0.37	0.37
query17	0.94	0.94	0.93
query18	0.20	0.28	0.26
query19	1.72	1.64	1.69
query20	0.02	0.01	0.01
query21	15.74	0.64	0.64
query22	4.28	7.73	1.85
query23	18.27	1.45	1.25
query24	1.26	0.31	0.28
query25	0.15	0.07	0.08
query26	0.26	0.17	0.17
query27	0.07	0.08	0.09
query28	13.51	1.01	1.01
query29	13.79	3.29	3.28
query30	0.25	0.05	0.06
query31	2.88	0.38	0.37
query32	3.29	0.47	0.46
query33	2.87	2.83	2.96
query34	17.21	4.44	4.42
query35	4.57	4.56	4.58
query36	0.65	0.47	0.46
query37	0.18	0.16	0.16
query38	0.15	0.14	0.14
query39	0.04	0.03	0.04
query40	0.16	0.13	0.15
query41	0.09	0.05	0.04
query42	0.06	0.05	0.04
query43	0.03	0.04	0.03
Total cold run time: 110.3 s
Total hot run time: 30.39 s

morningman · 2024-05-26T15:24:46Z

run buildall

doris-robot · 2024-05-26T16:45:12Z

TPC-H: Total hot run time: 41472 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 22ec3309c91839c38918f90c9f101ea9b90f9e13, data reload: false
------ Round 1 ----------------------------------
q1	17614	4456	4349	4349
q2	2022	202	197	197
q3	10477	1264	1210	1210
q4	10191	858	828	828
q5	7478	2736	2735	2735
q6	218	134	141	134
q7	969	625	623	623
q8	9241	2104	2114	2104
q9	9498	6652	6714	6652
q10	9185	3943	3894	3894
q11	459	249	240	240
q12	499	230	224	224
q13	18045	3268	3218	3218
q14	273	232	233	232
q15	522	484	468	468
q16	507	379	377	377
q17	983	683	771	683
q18	8607	8049	7801	7801
q19	2871	1567	1565	1565
q20	627	339	312	312
q21	5232	3356	4160	3356
q22	353	277	270	270
Total cold run time: 115871 ms
Total hot run time: 41472 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4535	4507	4455	4455
q2	372	267	283	267
q3	3197	2991	2986	2986
q4	1945	1600	1557	1557
q5	5412	5536	5530	5530
q6	229	130	127	127
q7	2278	1812	1869	1812
q8	3272	3415	3388	3388
q9	8617	8678	8664	8664
q10	4066	3701	3787	3701
q11	580	507	506	506
q12	810	616	639	616
q13	17197	3143	3169	3143
q14	295	280	287	280
q15	527	493	502	493
q16	505	439	444	439
q17	1838	1481	1469	1469
q18	7733	7885	7562	7562
q19	1673	1582	1584	1582
q20	1977	1776	1819	1776
q21	9461	4697	4741	4697
q22	533	473	503	473
Total cold run time: 77052 ms
Total hot run time: 55523 ms

doris-robot · 2024-05-26T16:55:53Z

TPC-DS: Total hot run time: 169230 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 22ec3309c91839c38918f90c9f101ea9b90f9e13, data reload: false
query1	909	387	372	372
query2	7857	2577	2492	2492
query3	6648	209	207	207
query4	21429	17341	17298	17298
query5	4152	426	421	421
query6	244	154	146	146
query7	4586	305	284	284
query8	243	192	183	183
query9	8603	2389	2346	2346
query10	462	292	280	280
query11	10715	10090	10199	10090
query12	138	93	89	89
query13	1633	366	355	355
query14	9623	6492	7000	6492
query15	216	173	171	171
query16	7528	277	260	260
query17	1296	529	520	520
query18	1750	268	269	268
query19	193	155	152	152
query20	92	85	85	85
query21	194	136	129	129
query22	4377	4022	3874	3874
query23	33810	33150	33096	33096
query24	7085	2884	2798	2798
query25	578	361	362	361
query26	706	155	156	155
query27	2060	320	319	319
query28	3837	2041	2052	2041
query29	918	609	598	598
query30	219	171	175	171
query31	965	757	749	749
query32	92	55	53	53
query33	488	266	263	263
query34	859	483	476	476
query35	711	616	602	602
query36	1068	954	910	910
query37	104	70	70	70
query38	2903	2777	2777	2777
query39	879	830	805	805
query40	193	124	124	124
query41	46	44	44	44
query42	102	96	97	96
query43	651	569	549	549
query44	1067	725	743	725
query45	180	167	165	165
query46	1059	728	736	728
query47	1844	1737	1779	1737
query48	362	292	302	292
query49	830	380	388	380
query50	775	397	397	397
query51	6913	6678	6739	6678
query52	103	96	94	94
query53	348	286	288	286
query54	532	437	452	437
query55	76	75	72	72
query56	261	247	252	247
query57	1106	1032	1060	1032
query58	258	216	229	216
query59	3582	3236	3281	3236
query60	267	253	261	253
query61	92	89	83	83
query62	596	438	464	438
query63	306	282	288	282
query64	8411	2207	1775	1775
query65	3170	3110	3100	3100
query66	822	322	325	322
query67	15259	14689	14931	14689
query68	4456	546	550	546
query69	442	275	273	273
query70	1108	1134	1153	1134
query71	370	271	273	271
query72	7622	5087	2715	2715
query73	721	320	323	320
query74	6083	5675	5613	5613
query75	3345	2662	2648	2648
query76	2201	916	1021	916
query77	375	270	269	269
query78	10412	9833	9743	9743
query79	2586	525	524	524
query80	1039	428	432	428
query81	517	251	245	245
query82	751	99	100	99
query83	247	168	176	168
query84	247	94	83	83
query85	1103	276	266	266
query86	459	291	315	291
query87	3276	3109	3084	3084
query88	4141	2336	2335	2335
query89	468	401	389	389
query90	2023	192	192	192
query91	123	96	148	96
query92	57	49	48	48
query93	2059	510	501	501
query94	1115	191	186	186
query95	406	317	313	313
query96	602	271	264	264
query97	3189	2982	3038	2982
query98	240	227	217	217
query99	1144	836	841	836
Total cold run time: 262308 ms
Total hot run time: 169230 ms

doris-robot · 2024-05-26T17:01:24Z

ClickBench: Total hot run time: 30.5 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 22ec3309c91839c38918f90c9f101ea9b90f9e13, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.03	0.04
query3	0.23	0.05	0.06
query4	1.67	0.09	0.10
query5	0.53	0.47	0.52
query6	1.13	0.74	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.48	0.49
query10	0.54	0.54	0.55
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.63	0.60	0.60
query14	0.77	0.78	0.79
query15	0.82	0.82	0.81
query16	0.37	0.34	0.37
query17	1.01	0.94	0.97
query18	0.21	0.25	0.25
query19	1.83	1.70	1.70
query20	0.02	0.01	0.01
query21	15.49	0.70	0.67
query22	4.50	7.27	1.78
query23	18.29	1.34	1.29
query24	1.78	0.26	0.22
query25	0.14	0.09	0.08
query26	0.26	0.16	0.17
query27	0.08	0.08	0.08
query28	13.29	1.05	1.03
query29	13.36	3.30	3.28
query30	0.25	0.06	0.06
query31	2.87	0.39	0.38
query32	3.28	0.48	0.47
query33	2.92	2.89	2.89
query34	17.17	4.43	4.41
query35	4.51	4.50	4.68
query36	0.66	0.46	0.47
query37	0.18	0.16	0.16
query38	0.16	0.15	0.15
query39	0.05	0.03	0.03
query40	0.16	0.14	0.17
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.41 s
Total hot run time: 30.5 s

doris-robot · 2024-05-27T04:23:31Z

TPC-H: Total hot run time: 40077 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4ea97bf79243d57db892a41dc0453e551140368f, data reload: false
------ Round 1 ----------------------------------
q1	17770	5137	4382	4382
q2	2767	197	201	197
q3	11116	1205	1224	1205
q4	10866	866	815	815
q5	8484	2747	2687	2687
q6	221	132	134	132
q7	973	621	600	600
q8	9534	2058	2082	2058
q9	8855	6501	6489	6489
q10	8951	3722	3687	3687
q11	454	235	232	232
q12	430	219	217	217
q13	18634	2987	2970	2970
q14	272	220	232	220
q15	514	484	484	484
q16	517	374	382	374
q17	987	696	794	696
q18	8100	7561	7349	7349
q19	5733	1556	1518	1518
q20	656	313	307	307
q21	4969	3169	3241	3169
q22	337	289	297	289
Total cold run time: 121140 ms
Total hot run time: 40077 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4372	4205	4242	4205
q2	355	277	259	259
q3	2981	2749	2765	2749
q4	1836	1605	1596	1596
q5	5237	5279	5286	5279
q6	212	121	124	121
q7	2136	1669	1751	1669
q8	3154	3304	3312	3304
q9	8402	8353	8352	8352
q10	3896	3708	3638	3638
q11	595	488	483	483
q12	772	621	641	621
q13	17264	2958	3003	2958
q14	298	267	270	267
q15	512	478	487	478
q16	483	410	408	408
q17	1784	1476	1496	1476
q18	7639	7518	7337	7337
q19	1661	1537	1522	1522
q20	1972	1782	1799	1782
q21	4873	4636	4859	4636
q22	547	488	503	488
Total cold run time: 70981 ms
Total hot run time: 53628 ms

doris-robot · 2024-05-27T04:34:23Z

TPC-DS: Total hot run time: 168684 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4ea97bf79243d57db892a41dc0453e551140368f, data reload: false
query1	912	377	379	377
query2	7168	2476	2382	2382
query3	6663	203	204	203
query4	19218	17281	17127	17127
query5	4135	414	428	414
query6	253	157	150	150
query7	4580	296	283	283
query8	235	172	183	172
query9	8368	2428	2409	2409
query10	463	274	262	262
query11	10604	9958	9917	9917
query12	130	99	90	90
query13	1632	370	364	364
query14	9332	6817	7572	6817
query15	232	169	169	169
query16	8005	264	263	263
query17	1859	517	500	500
query18	2032	286	273	273
query19	197	149	144	144
query20	89	83	83	83
query21	194	132	133	132
query22	4321	3978	3865	3865
query23	33622	33024	34385	33024
query24	11974	2898	2780	2780
query25	512	371	368	368
query26	1092	157	158	157
query27	2886	320	319	319
query28	7381	2081	2066	2066
query29	841	596	603	596
query30	294	150	150	150
query31	966	747	745	745
query32	97	51	55	51
query33	769	303	255	255
query34	1010	471	493	471
query35	718	623	606	606
query36	1132	916	930	916
query37	191	69	66	66
query38	2928	2818	2744	2744
query39	856	783	781	781
query40	276	121	125	121
query41	45	44	45	44
query42	100	99	92	92
query43	606	540	543	540
query44	1246	712	735	712
query45	176	158	164	158
query46	1077	693	724	693
query47	1870	1757	1806	1757
query48	373	292	301	292
query49	1161	380	407	380
query50	772	376	381	376
query51	6750	6652	6637	6637
query52	101	96	87	87
query53	355	294	280	280
query54	920	427	429	427
query55	72	70	72	70
query56	257	237	246	237
query57	1100	1064	1080	1064
query58	226	213	206	206
query59	3565	3300	3279	3279
query60	274	265	264	264
query61	92	91	127	91
query62	642	447	447	447
query63	310	281	288	281
query64	9659	2215	1764	1764
query65	3210	3104	3131	3104
query66	1238	324	319	319
query67	15433	15301	14839	14839
query68	4819	522	528	522
query69	447	323	260	260
query70	1192	1157	1129	1129
query71	429	267	266	266
query72	7367	2760	2543	2543
query73	714	321	327	321
query74	5990	5577	5546	5546
query75	3420	2582	2626	2582
query76	2976	910	971	910
query77	386	258	256	256
query78	10141	9797	9959	9797
query79	1970	506	510	506
query80	995	432	421	421
query81	539	231	224	224
query82	704	89	89	89
query83	240	170	167	167
query84	230	87	85	85
query85	1969	332	315	315
query86	479	290	304	290
query87	3352	3085	3098	3085
query88	4431	2404	2419	2404
query89	455	382	370	370
query90	2070	187	181	181
query91	123	96	94	94
query92	59	47	49	47
query93	2244	500	484	484
query94	1222	182	181	181
query95	392	300	300	300
query96	589	266	260	260
query97	3177	2958	3015	2958
query98	236	215	213	213
query99	1176	871	826	826
Total cold run time: 274330 ms
Total hot run time: 168684 ms

doris-robot · 2024-05-27T04:39:51Z

ClickBench: Total hot run time: 30.53 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4ea97bf79243d57db892a41dc0453e551140368f, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.04
query4	1.69	0.07	0.07
query5	0.48	0.49	0.52
query6	1.12	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.53	0.48	0.48
query10	0.55	0.55	0.55
query11	0.17	0.11	0.12
query12	0.15	0.12	0.12
query13	0.59	0.59	0.60
query14	0.75	0.77	0.77
query15	0.82	0.80	0.80
query16	0.37	0.35	0.39
query17	0.98	0.96	1.03
query18	0.21	0.24	0.25
query19	1.81	1.68	1.69
query20	0.01	0.01	0.01
query21	15.62	0.70	0.67
query22	4.70	6.59	2.01
query23	18.30	1.32	1.26
query24	1.56	0.25	0.32
query25	0.15	0.09	0.08
query26	0.27	0.16	0.15
query27	0.07	0.08	0.08
query28	13.24	1.00	0.99
query29	13.00	3.31	3.25
query30	0.25	0.06	0.06
query31	2.88	0.37	0.38
query32	3.28	0.47	0.47
query33	2.91	2.86	2.87
query34	17.18	4.40	4.51
query35	4.47	4.54	4.50
query36	0.65	0.48	0.46
query37	0.18	0.16	0.15
query38	0.14	0.14	0.15
query39	0.05	0.03	0.04
query40	0.15	0.15	0.14
query41	0.09	0.05	0.04
query42	0.06	0.05	0.04
query43	0.04	0.03	0.04
Total cold run time: 109.89 s
Total hot run time: 30.53 s

caiconghui · 2024-05-27T06:18:25Z

run buildall

doris-robot · 2024-05-27T07:33:20Z

TPC-H: Total hot run time: 42279 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d920815f65d925b0724274bd77e0fde4161bdd65, data reload: false
------ Round 1 ----------------------------------
q1	17593	4949	4278	4278
q2	2023	184	188	184
q3	10473	1237	1322	1237
q4	10200	854	922	854
q5	7469	2708	2782	2708
q6	225	133	141	133
q7	973	606	622	606
q8	9209	2160	2106	2106
q9	9526	6717	6724	6717
q10	9330	3955	3798	3798
q11	435	241	250	241
q12	424	218	243	218
q13	18290	3242	3187	3187
q14	265	210	232	210
q15	504	474	479	474
q16	532	403	395	395
q17	1005	781	780	780
q18	8449	7900	7915	7900
q19	5021	1592	1558	1558
q20	638	308	312	308
q21	5200	4106	4155	4106
q22	355	281	283	281
Total cold run time: 118139 ms
Total hot run time: 42279 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4613	4388	4427	4388
q2	369	282	278	278
q3	3112	3028	2758	2758
q4	1892	1597	1639	1597
q5	5489	5514	5509	5509
q6	218	128	132	128
q7	2207	1825	1859	1825
q8	3273	3400	3409	3400
q9	8736	8716	8714	8714
q10	3964	3816	3774	3774
q11	592	511	509	509
q12	786	630	648	630
q13	15933	3190	3130	3130
q14	321	280	261	261
q15	535	485	480	480
q16	498	441	444	441
q17	1795	1511	1499	1499
q18	7768	7621	7373	7373
q19	1675	1597	1563	1563
q20	2017	1775	1795	1775
q21	15096	4878	4789	4789
q22	556	512	504	504
Total cold run time: 81445 ms
Total hot run time: 55325 ms

doris-robot · 2024-05-27T07:43:47Z

TPC-DS: Total hot run time: 169518 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d920815f65d925b0724274bd77e0fde4161bdd65, data reload: false
query1	934	387	365	365
query2	6457	2509	2344	2344
query3	6666	203	201	201
query4	19764	17384	17441	17384
query5	4115	418	415	415
query6	240	156	155	155
query7	4583	295	292	292
query8	241	185	185	185
query9	8426	2364	2337	2337
query10	456	289	290	289
query11	10561	10238	10065	10065
query12	142	89	95	89
query13	1656	375	360	360
query14	9164	6848	7876	6848
query15	214	173	165	165
query16	7520	265	269	265
query17	1699	539	536	536
query18	1269	268	278	268
query19	198	155	161	155
query20	91	87	85	85
query21	192	140	133	133
query22	4143	3965	4072	3965
query23	33857	33079	33154	33079
query24	6716	2879	2852	2852
query25	513	353	382	353
query26	697	155	159	155
query27	1971	332	326	326
query28	3700	2038	2023	2023
query29	852	630	608	608
query30	215	148	152	148
query31	959	784	732	732
query32	77	54	53	53
query33	498	288	259	259
query34	851	484	480	480
query35	699	634	620	620
query36	1054	921	918	918
query37	103	70	67	67
query38	2876	2801	2787	2787
query39	833	801	802	801
query40	201	125	129	125
query41	46	45	51	45
query42	102	103	94	94
query43	613	581	585	581
query44	1086	739	754	739
query45	181	163	163	163
query46	1060	740	705	705
query47	1859	1758	1784	1758
query48	391	299	299	299
query49	769	377	394	377
query50	780	382	388	382
query51	6987	6697	6661	6661
query52	103	90	95	90
query53	364	292	283	283
query54	547	442	425	425
query55	75	72	73	72
query56	264	247	248	247
query57	1129	1054	1025	1025
query58	262	225	204	204
query59	3586	3275	3343	3275
query60	276	252	261	252
query61	89	85	88	85
query62	571	460	438	438
query63	315	291	287	287
query64	3176	1776	1727	1727
query65	3227	3109	3112	3109
query66	806	333	319	319
query67	15432	14695	14896	14695
query68	4558	529	518	518
query69	449	276	269	269
query70	1204	1185	1125	1125
query71	340	277	264	264
query72	7725	2709	2549	2549
query73	718	332	327	327
query74	6112	5574	5529	5529
query75	3224	2649	2630	2630
query76	2227	977	1029	977
query77	387	269	258	258
query78	10255	9797	10056	9797
query79	1458	510	514	510
query80	964	435	422	422
query81	496	218	219	218
query82	731	92	92	92
query83	250	171	168	168
query84	233	83	84	83
query85	988	270	262	262
query86	406	310	284	284
query87	3301	3129	3118	3118
query88	3161	2454	2469	2454
query89	473	391	382	382
query90	1959	191	198	191
query91	137	109	107	107
query92	59	52	51	51
query93	1032	508	500	500
query94	1089	262	186	186
query95	418	308	311	308
query96	582	270	274	270
query97	3196	3013	2980	2980
query98	241	215	218	215
query99	1171	872	850	850
Total cold run time: 249043 ms
Total hot run time: 169518 ms

doris-robot · 2024-05-27T07:49:16Z

ClickBench: Total hot run time: 30.13 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d920815f65d925b0724274bd77e0fde4161bdd65, data reload: false
query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.65	0.09	0.09
query5	0.55	0.49	0.50
query6	1.12	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.53	0.49	0.49
query10	0.53	0.56	0.54
query11	0.15	0.11	0.11
query12	0.14	0.12	0.12
query13	0.59	0.59	0.58
query14	0.80	0.81	0.78
query15	0.83	0.82	0.82
query16	0.37	0.36	0.37
query17	1.02	0.96	0.97
query18	0.23	0.25	0.24
query19	1.80	1.67	1.70
query20	0.02	0.01	0.01
query21	15.60	0.66	0.63
query22	3.85	8.02	1.57
query23	18.34	1.43	1.26
query24	1.34	0.38	0.23
query25	0.15	0.08	0.08
query26	0.26	0.17	0.17
query27	0.08	0.08	0.08
query28	13.38	1.03	0.99
query29	13.78	3.32	3.27
query30	0.24	0.06	0.05
query31	2.88	0.39	0.39
query32	3.31	0.48	0.47
query33	2.93	2.84	2.87
query34	16.86	4.44	4.38
query35	4.51	4.51	4.63
query36	0.65	0.46	0.49
query37	0.17	0.16	0.15
query38	0.15	0.15	0.15
query39	0.05	0.04	0.03
query40	0.17	0.14	0.15
query41	0.10	0.05	0.04
query42	0.05	0.04	0.04
query43	0.04	0.04	0.03
Total cold run time: 109.64 s
Total hot run time: 30.13 s

caiconghui · 2024-05-27T10:24:18Z

run buildall

caiconghui · 2024-05-27T10:45:47Z

run p0

doris-robot · 2024-05-27T11:20:59Z

TPC-H: Total hot run time: 41446 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit af32621e8912f4bc788faf075578e20182da219e, data reload: false
------ Round 1 ----------------------------------
q1	17690	4359	4273	4273
q2	2032	195	196	195
q3	10512	1306	1258	1258
q4	10216	798	811	798
q5	7563	2761	2700	2700
q6	221	137	134	134
q7	980	602	624	602
q8	9245	2168	2133	2133
q9	9757	6748	6692	6692
q10	9214	3928	3921	3921
q11	478	242	237	237
q12	442	222	231	222
q13	17359	3200	3286	3200
q14	269	232	237	232
q15	507	470	481	470
q16	517	392	417	392
q17	994	690	742	690
q18	8323	7781	7794	7781
q19	8966	1594	1556	1556
q20	649	316	318	316
q21	5163	3941	3371	3371
q22	347	300	273	273
Total cold run time: 121444 ms
Total hot run time: 41446 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4511	4314	4408	4314
q2	381	271	274	271
q3	3163	2965	2898	2898
q4	1952	1725	1705	1705
q5	5362	5378	5517	5378
q6	215	120	121	120
q7	2225	1842	1806	1806
q8	3249	3411	3389	3389
q9	8579	8667	8642	8642
q10	4124	3873	3803	3803
q11	591	484	482	482
q12	750	578	620	578
q13	15926	3083	3174	3083
q14	300	254	263	254
q15	541	489	490	489
q16	498	421	430	421
q17	1807	1540	1493	1493
q18	8194	7522	7473	7473
q19	1676	1586	1596	1586
q20	2018	1787	1801	1787
q21	4878	4843	4730	4730
q22	579	484	492	484
Total cold run time: 71519 ms
Total hot run time: 55186 ms

doris-robot · 2024-05-27T11:31:36Z

TPC-DS: Total hot run time: 168858 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit af32621e8912f4bc788faf075578e20182da219e, data reload: false
query1	915	384	365	365
query2	6460	2245	2213	2213
query3	6646	213	207	207
query4	19676	17402	17351	17351
query5	4159	441	436	436
query6	244	158	149	149
query7	4582	297	295	295
query8	237	182	181	181
query9	8627	2459	2429	2429
query10	440	278	264	264
query11	10661	10117	10073	10073
query12	137	92	87	87
query13	1650	372	371	371
query14	9262	6122	7127	6122
query15	217	168	171	168
query16	7730	276	288	276
query17	1685	545	529	529
query18	1886	286	286	286
query19	205	162	159	159
query20	93	89	88	88
query21	198	133	127	127
query22	4245	4019	3993	3993
query23	33570	33206	32962	32962
query24	6688	2896	2733	2733
query25	532	350	390	350
query26	703	157	157	157
query27	1906	319	327	319
query28	3752	2107	2085	2085
query29	851	625	589	589
query30	229	148	152	148
query31	939	778	753	753
query32	93	53	52	52
query33	497	287	260	260
query34	866	472	474	472
query35	726	605	584	584
query36	1086	930	879	879
query37	101	69	65	65
query38	2873	2826	2768	2768
query39	821	795	790	790
query40	197	123	122	122
query41	50	43	46	43
query42	102	98	97	97
query43	577	548	526	526
query44	1044	721	737	721
query45	175	165	168	165
query46	1057	755	741	741
query47	1853	1777	1766	1766
query48	362	300	294	294
query49	788	377	386	377
query50	773	399	386	386
query51	6824	6804	6765	6765
query52	99	90	87	87
query53	353	282	280	280
query54	545	440	423	423
query55	75	70	72	70
query56	261	236	274	236
query57	1099	1104	1030	1030
query58	225	204	209	204
query59	3308	3228	3228	3228
query60	284	258	254	254
query61	89	114	87	87
query62	540	461	459	459
query63	315	283	287	283
query64	8305	2202	1705	1705
query65	3176	3096	3126	3096
query66	835	330	319	319
query67	15193	15080	14957	14957
query68	4691	538	547	538
query69	499	288	278	278
query70	1167	1130	1099	1099
query71	399	276	268	268
query72	8176	2694	2555	2555
query73	728	324	326	324
query74	6027	5655	5635	5635
query75	3473	2622	2658	2622
query76	3147	983	1033	983
query77	581	271	262	262
query78	10341	9746	9743	9743
query79	2031	518	506	506
query80	1230	442	439	439
query81	474	218	219	218
query82	1283	92	100	92
query83	206	221	193	193
query84	274	91	85	85
query85	1135	261	262	261
query86	461	324	287	287
query87	3301	3110	3109	3109
query88	4136	2442	2458	2442
query89	473	404	383	383
query90	2075	192	192	192
query91	125	94	96	94
query92	58	48	48	48
query93	2581	513	494	494
query94	1152	188	188	188
query95	402	318	306	306
query96	597	285	273	273
query97	3188	3016	3015	3015
query98	250	218	223	218
query99	1150	852	863	852
Total cold run time: 260503 ms
Total hot run time: 168858 ms

doris-robot · 2024-05-27T11:37:07Z

ClickBench: Total hot run time: 30.55 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit af32621e8912f4bc788faf075578e20182da219e, data reload: false
query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.04
query4	1.68	0.07	0.09
query5	0.53	0.50	0.52
query6	1.12	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.49	0.47
query10	0.56	0.55	0.55
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.58	0.58	0.59
query14	0.77	0.78	0.77
query15	0.82	0.82	0.80
query16	0.37	0.36	0.36
query17	1.02	1.01	0.95
query18	0.22	0.24	0.25
query19	1.72	1.72	1.66
query20	0.01	0.01	0.01
query21	15.42	0.70	0.69
query22	4.92	6.64	1.88
query23	18.29	1.36	1.31
query24	1.49	0.29	0.29
query25	0.13	0.09	0.08
query26	0.25	0.16	0.16
query27	0.08	0.08	0.08
query28	13.29	1.01	1.00
query29	13.33	3.29	3.25
query30	0.24	0.06	0.05
query31	2.88	0.38	0.38
query32	3.28	0.46	0.47
query33	2.89	2.91	2.91
query34	17.13	4.42	4.52
query35	4.57	4.49	4.75
query36	0.66	0.45	0.45
query37	0.18	0.15	0.16
query38	0.16	0.14	0.14
query39	0.04	0.03	0.03
query40	0.17	0.14	0.14
query41	0.09	0.05	0.04
query42	0.06	0.04	0.05
query43	0.04	0.03	0.03
Total cold run time: 110.26 s
Total hot run time: 30.55 s

…emote storage

caiconghui · 2024-05-27T13:09:41Z

run buildall

doris-robot · 2024-05-27T13:35:01Z

TPC-H: Total hot run time: 40206 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3cdccf6cc51ea823847aac1e46fa3539f1d7ed5a, data reload: false
------ Round 1 ----------------------------------
q1	18047	4345	4225	4225
q2	2022	195	194	194
q3	10491	1188	1294	1188
q4	10144	708	742	708
q5	7472	2648	2572	2572
q6	213	133	130	130
q7	938	594	586	586
q8	9224	2065	2057	2057
q9	8608	6461	6463	6461
q10	8992	3696	3717	3696
q11	466	242	230	230
q12	443	219	222	219
q13	17840	2955	2989	2955
q14	265	225	214	214
q15	524	486	466	466
q16	536	382	380	380
q17	960	742	723	723
q18	8065	7385	7341	7341
q19	6953	1522	1485	1485
q20	681	309	301	301
q21	5020	3799	4116	3799
q22	342	279	276	276
Total cold run time: 118246 ms
Total hot run time: 40206 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4343	4191	4153	4153
q2	355	266	268	266
q3	2992	2713	2737	2713
q4	1851	1617	1639	1617
q5	5254	5271	5273	5271
q6	213	124	124	124
q7	2132	1751	1754	1751
q8	3187	3319	3282	3282
q9	8334	8387	8331	8331
q10	3898	3631	3690	3631
q11	579	491	493	491
q12	803	574	590	574
q13	17413	2983	2982	2982
q14	292	259	263	259
q15	506	493	478	478
q16	464	425	425	425
q17	1774	1468	1455	1455
q18	7588	7648	7340	7340
q19	1657	1551	1581	1551
q20	1987	1782	1789	1782
q21	4803	4772	4759	4759
q22	552	472	469	469
Total cold run time: 70977 ms
Total hot run time: 53704 ms

doris-robot · 2024-05-27T13:45:57Z

TPC-DS: Total hot run time: 171665 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3cdccf6cc51ea823847aac1e46fa3539f1d7ed5a, data reload: false
query1	906	376	367	367
query2	6453	2403	2379	2379
query3	6660	205	206	205
query4	19410	17330	17370	17330
query5	4121	413	416	413
query6	256	154	145	145
query7	4586	297	286	286
query8	248	192	186	186
query9	8625	2335	2319	2319
query10	451	284	274	274
query11	10543	10088	10075	10075
query12	137	93	87	87
query13	1645	371	395	371
query14	10106	7157	6532	6532
query15	228	164	157	157
query16	8064	274	257	257
query17	1875	527	515	515
query18	2073	271	273	271
query19	197	159	165	159
query20	94	85	84	84
query21	199	129	127	127
query22	4107	3931	4036	3931
query23	33591	32977	33060	32977
query24	11838	2921	2797	2797
query25	644	352	358	352
query26	1777	158	155	155
query27	2998	314	317	314
query28	7653	1996	1995	1995
query29	1031	610	586	586
query30	291	151	151	151
query31	949	749	752	749
query32	93	55	55	55
query33	797	269	271	269
query34	1021	472	480	472
query35	724	604	606	604
query36	1075	886	905	886
query37	204	65	69	65
query38	2887	2788	2752	2752
query39	853	786	787	786
query40	276	122	126	122
query41	48	44	50	44
query42	102	95	99	95
query43	581	557	559	557
query44	1151	732	745	732
query45	187	167	160	160
query46	1061	707	722	707
query47	1847	1755	1776	1755
query48	378	306	288	288
query49	1202	378	392	378
query50	766	385	399	385
query51	6850	6794	6843	6794
query52	106	120	85	85
query53	348	284	282	282
query54	994	446	438	438
query55	75	71	72	71
query56	264	242	247	242
query57	1097	1088	1082	1082
query58	234	204	210	204
query59	3420	3092	3326	3092
query60	283	254	262	254
query61	95	94	90	90
query62	650	463	465	463
query63	308	286	285	285
query64	9853	2198	1729	1729
query65	3179	3097	3144	3097
query66	1365	331	329	329
query67	15153	14908	14866	14866
query68	4609	517	528	517
query69	440	279	269	269
query70	1079	1091	1075	1075
query71	400	268	269	268
query72	7393	5541	5403	5403
query73	725	324	321	321
query74	5948	5560	5594	5560
query75	3415	2621	2634	2621
query76	2825	1027	935	935
query77	417	265	265	265
query78	10126	9785	9808	9785
query79	2517	498	513	498
query80	1032	445	428	428
query81	511	215	218	215
query82	726	90	91	90
query83	251	173	170	170
query84	238	91	86	86
query85	1448	288	269	269
query86	447	322	282	282
query87	3272	3103	3127	3103
query88	4074	2429	2418	2418
query89	476	388	375	375
query90	1999	195	191	191
query91	191	97	95	95
query92	62	51	52	51
query93	1552	508	487	487
query94	1187	188	189	188
query95	393	310	309	309
query96	582	267	267	267
query97	3161	3008	3018	3008
query98	236	209	203	203
query99	1078	873	852	852
Total cold run time: 274093 ms
Total hot run time: 171665 ms

doris-robot · 2024-05-27T13:51:28Z

ClickBench: Total hot run time: 31.03 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3cdccf6cc51ea823847aac1e46fa3539f1d7ed5a, data reload: false
query1	0.03	0.03	0.04
query2	0.08	0.04	0.04
query3	0.24	0.05	0.05
query4	1.68	0.07	0.06
query5	0.55	0.50	0.50
query6	1.12	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.53	0.48	0.48
query10	0.54	0.54	0.52
query11	0.16	0.12	0.11
query12	0.14	0.12	0.12
query13	0.60	0.59	0.59
query14	0.77	0.77	0.77
query15	0.82	0.80	0.80
query16	0.37	0.36	0.37
query17	1.03	1.02	0.95
query18	0.22	0.24	0.23
query19	1.90	1.79	1.71
query20	0.01	0.01	0.02
query21	15.72	0.65	0.64
query22	4.36	6.81	2.41
query23	18.31	1.32	1.39
query24	1.92	0.24	0.22
query25	0.14	0.09	0.09
query26	0.27	0.18	0.17
query27	0.09	0.07	0.08
query28	13.28	1.01	1.00
query29	13.02	3.29	3.29
query30	0.24	0.05	0.05
query31	2.87	0.38	0.38
query32	3.29	0.47	0.47
query33	2.86	2.87	2.89
query34	17.09	4.52	4.44
query35	4.50	4.47	4.51
query36	0.64	0.46	0.46
query37	0.17	0.16	0.15
query38	0.15	0.14	0.14
query39	0.04	0.03	0.04
query40	0.16	0.14	0.16
query41	0.09	0.04	0.04
query42	0.06	0.04	0.04
query43	0.03	0.04	0.04
Total cold run time: 110.16 s
Total hot run time: 31.03 s

morningman

LGTM

github-actions · 2024-05-27T15:48:34Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-05-27T15:48:36Z

PR approved by anyone and no changes requested.

GoGoWen

LGTM

…emote storage (#35389) filter empty partition before export table to remote storage Co-authored-by: caiconghui1 <caiconghui1@jd.com>

…emote storage (#35389) (#35542) ## Proposed changes Linked PR : #35389  ## Further comments If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...

…emote storage (#35389) filter empty partition before export table to remote storage Co-authored-by: caiconghui1 <caiconghui1@jd.com>

…emote storage (apache#35389) filter empty partition before export table to remote storage Co-authored-by: caiconghui1 <caiconghui1@jd.com>

morningman force-pushed the export branch from 18fd1f5 to 22ec330Compare May 26, 2024 15:24

caiconghui force-pushed the export branch 2 times, most recently from f44a511 to d920815Compare May 27, 2024 06:17

[enhancement](export) filter empty partition before export table to r…
3cdccf6
…emote storage

caiconghui force-pushed the export branch from af32621 to 3cdccf6Compare May 27, 2024 13:09

morningman approved these changes May 27, 2024
View reviewed changes

github-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels May 27, 2024

GoGoWen approved these changes May 28, 2024
View reviewed changes

morningman merged commit f0091a4 into masterMay 28, 2024

morningman added dev/2.1.x dev/2.0.x labels May 28, 2024

caiconghui added a commit that referenced this pull request May 28, 2024
[enhancement](export) filter empty partition before export table to r…
b806a49
…emote storage (#35389) filter empty partition before export table to remote storage Co-authored-by: caiconghui1 <caiconghui1@jd.com>

caiconghui mentioned this pull request May 28, 2024
[enhancement](export) filter empty partition before export table to remote storage (#35389) #35542
Merged

dataroaring pushed a commit that referenced this pull request May 31, 2024
[enhancement](export) filter empty partition before export table to r…
6d6c2dc
…emote storage (#35389) filter empty partition before export table to remote storage Co-authored-by: caiconghui1 <caiconghui1@jd.com>

wm1581066 added dev/2.1.4-merged and removed dev/2.1.x labels Jun 19, 2024

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

Uh oh!

Conversation

caiconghui commented May 25, 2024

Proposed changes

Further comments

Uh oh!

doris-robot commented May 25, 2024

Uh oh!

caiconghui commented May 25, 2024

Uh oh!

doris-robot commented May 25, 2024

Uh oh!

doris-robot commented May 25, 2024

Uh oh!

doris-robot commented May 25, 2024

Uh oh!

morningman commented May 26, 2024

Uh oh!

doris-robot commented May 26, 2024

Uh oh!

doris-robot commented May 26, 2024

Uh oh!

doris-robot commented May 26, 2024

Uh oh!

doris-robot commented May 27, 2024

Uh oh!

doris-robot commented May 27, 2024

Uh oh!

doris-robot commented May 27, 2024

Uh oh!

caiconghui commented May 27, 2024

Uh oh!

doris-robot commented May 27, 2024

Uh oh!

doris-robot commented May 27, 2024

Uh oh!

doris-robot commented May 27, 2024

Uh oh!

caiconghui commented May 27, 2024

Uh oh!

caiconghui commented May 27, 2024

Uh oh!

doris-robot commented May 27, 2024

Uh oh!

doris-robot commented May 27, 2024

Uh oh!

doris-robot commented May 27, 2024

Uh oh!

caiconghui commented May 27, 2024

Uh oh!

doris-robot commented May 27, 2024

Uh oh!

doris-robot commented May 27, 2024

Uh oh!

doris-robot commented May 27, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented May 27, 2024

Uh oh!

github-actionsBot commented May 27, 2024

Uh oh!

GoGoWen left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants