[fix](Export) Set the default value of the `data_consistence` property of export to `partition` by BePPPower · Pull Request #32830 · apache/doris

BePPPower · 2024-03-26T08:28:21Z

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-03-26T08:28:26Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

BePPPower · 2024-03-26T08:28:55Z

run buildall

doris-robot · 2024-03-26T09:00:56Z

TPC-H: Total hot run time: 38406 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1b2968c41ff1f47b996fecb38b1caf19198dacb2, data reload: false
------ Round 1 ----------------------------------
q1	18373	7094	4321	4321
q2	2909	165	161	161
q3	11365	1128	1184	1128
q4	10874	823	848	823
q5	7874	3120	3115	3115
q6	211	130	124	124
q7	1063	613	587	587
q8	9477	1991	2015	1991
q9	7325	6610	6554	6554
q10	8412	3441	3556	3441
q11	437	229	225	225
q12	372	213	197	197
q13	17793	2844	2878	2844
q14	238	209	204	204
q15	522	458	453	453
q16	489	375	380	375
q17	973	559	555	555
q18	7232	6543	6468	6468
q19	2911	1402	1410	1402
q20	551	263	252	252
q21	3532	2897	3033	2897
q22	349	297	289	289
Total cold run time: 113282 ms
Total hot run time: 38406 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4149	4099	4165	4099
q2	339	237	241	237
q3	2983	2888	2817	2817
q4	1851	1622	1531	1531
q5	5308	5367	5375	5367
q6	192	116	118	116
q7	2258	1865	1833	1833
q8	3153	3298	3275	3275
q9	8795	8694	8717	8694
q10	3779	3799	3779	3779
q11	547	451	460	451
q12	704	530	587	530
q13	16913	2862	2838	2838
q14	275	255	246	246
q15	500	468	456	456
q16	488	420	438	420
q17	1750	1512	1474	1474
q18	7414	7173	7082	7082
q19	1605	1504	1541	1504
q20	1879	1692	1707	1692
q21	4889	4656	4726	4656
q22	517	459	428	428
Total cold run time: 70288 ms
Total hot run time: 53525 ms

doris-robot · 2024-03-26T09:12:39Z

TPC-DS: Total hot run time: 180772 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1b2968c41ff1f47b996fecb38b1caf19198dacb2, data reload: false
query1	939	361	353	353
query2	7598	1942	1932	1932
query3	6704	219	222	219
query4	31823	21290	21293	21290
query5	4334	399	391	391
query6	276	178	182	178
query7	4632	297	291	291
query8	231	166	172	166
query9	9044	2288	2259	2259
query10	565	243	247	243
query11	15290	14125	14159	14125
query12	137	94	87	87
query13	1628	431	410	410
query14	10203	7144	7969	7144
query15	260	198	201	198
query16	8171	265	262	262
query17	1940	586	553	553
query18	2114	284	292	284
query19	356	153	155	153
query20	96	92	92	92
query21	204	126	130	126
query22	5046	4795	4724	4724
query23	33024	32848	32800	32800
query24	10741	2965	2858	2858
query25	616	366	373	366
query26	1230	155	151	151
query27	2645	353	360	353
query28	7316	1913	1878	1878
query29	866	644	601	601
query30	300	147	147	147
query31	988	757	737	737
query32	96	61	55	55
query33	750	247	246	246
query34	1075	483	484	483
query35	825	617	592	592
query36	1029	871	881	871
query37	122	63	62	62
query38	3547	3431	3411	3411
query39	1471	1440	1423	1423
query40	199	110	109	109
query41	50	44	45	44
query42	103	93	95	93
query43	491	464	442	442
query44	1088	717	742	717
query45	275	260	255	255
query46	1112	711	698	698
query47	1910	1824	1840	1824
query48	459	362	361	361
query49	1125	329	317	317
query50	751	378	363	363
query51	6690	6632	6549	6549
query52	114	89	87	87
query53	338	272	268	268
query54	301	237	224	224
query55	91	78	84	78
query56	240	225	215	215
query57	1206	1140	1118	1118
query58	224	198	203	198
query59	2667	2509	2442	2442
query60	261	237	251	237
query61	100	108	111	108
query62	676	462	441	441
query63	307	274	272	272
query64	5788	4113	4002	4002
query65	3050	3025	3012	3012
query66	875	398	382	382
query67	15344	14780	14645	14645
query68	5421	533	525	525
query69	567	372	375	372
query70	1178	1188	1125	1125
query71	425	265	264	264
query72	6522	2855	2686	2686
query73	695	326	322	322
query74	7094	6359	6353	6353
query75	2964	2236	2207	2207
query76	3531	905	902	902
query77	412	270	259	259
query78	10818	10239	10260	10239
query79	8352	537	530	530
query80	2027	391	381	381
query81	561	217	213	213
query82	1351	88	91	88
query83	326	175	142	142
query84	286	82	76	76
query85	2080	332	318	318
query86	494	290	298	290
query87	3707	3554	3530	3530
query88	5204	2396	2367	2367
query89	504	376	371	371
query90	1981	175	176	175
query91	173	135	145	135
query92	61	48	49	48
query93	5769	502	491	491
query94	1258	176	179	176
query95	430	339	326	326
query96	595	269	272	269
query97	2686	2491	2459	2459
query98	235	213	214	213
query99	1199	905	906	905
Total cold run time: 302973 ms
Total hot run time: 180772 ms

doris-robot · 2024-03-26T09:15:52Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 1b2968c41ff1f47b996fecb38b1caf19198dacb2 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 20.4 seconds inserted 10000000 Rows, about 490K ops/s

BePPPower · 2024-03-27T08:16:31Z

run buildall

doris-robot · 2024-03-27T08:33:58Z

TPC-H: Total hot run time: 38221 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ac7a009f83e24c6618094b955c56347cffda13c7, data reload: false
------ Round 1 ----------------------------------
q1	17641	4180	4097	4097
q2	2118	159	152	152
q3	10907	1171	1219	1171
q4	10981	827	885	827
q5	8163	3085	3043	3043
q6	207	130	129	129
q7	1069	595	600	595
q8	9656	2059	2027	2027
q9	7415	6644	6571	6571
q10	8467	3497	3569	3497
q11	436	224	213	213
q12	369	206	195	195
q13	17900	2870	2865	2865
q14	227	206	205	205
q15	507	467	471	467
q16	511	378	381	378
q17	956	539	591	539
q18	7198	6497	6419	6419
q19	1557	1482	1405	1405
q20	543	251	258	251
q21	3602	2935	2878	2878
q22	343	297	299	297
Total cold run time: 110773 ms
Total hot run time: 38221 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4090	4027	4095	4027
q2	336	238	234	234
q3	2986	2857	2801	2801
q4	1869	1539	1535	1535
q5	5318	5358	5333	5333
q6	198	117	120	117
q7	2252	1869	1845	1845
q8	3140	3294	3253	3253
q9	8770	8665	8711	8665
q10	3785	3787	3768	3768
q11	538	445	437	437
q12	701	560	514	514
q13	13391	2868	2870	2868
q14	278	248	252	248
q15	503	461	456	456
q16	460	427	429	427
q17	1727	1482	1453	1453
q18	7422	7178	7164	7164
q19	1667	1488	1538	1488
q20	1916	1741	1695	1695
q21	4869	4766	4681	4681
q22	531	472	457	457
Total cold run time: 66747 ms
Total hot run time: 53466 ms

doris-robot · 2024-03-27T08:45:45Z

TPC-DS: Total hot run time: 181824 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ac7a009f83e24c6618094b955c56347cffda13c7, data reload: false
query1	942	377	356	356
query2	6551	2169	1958	1958
query3	6705	207	209	207
query4	32217	21281	21270	21270
query5	4377	386	396	386
query6	272	172	197	172
query7	4632	297	290	290
query8	242	175	168	168
query9	9160	2347	2342	2342
query10	560	237	250	237
query11	17168	14186	14201	14186
query12	137	92	84	84
query13	1625	411	416	411
query14	10148	7966	7784	7784
query15	256	194	202	194
query16	8170	266	274	266
query17	1906	580	561	561
query18	2116	292	277	277
query19	314	152	160	152
query20	97	90	92	90
query21	209	133	129	129
query22	5053	4838	4821	4821
query23	33603	33038	32782	32782
query24	11830	2876	2856	2856
query25	640	387	389	387
query26	1804	159	158	158
query27	3096	353	355	353
query28	7684	1848	1878	1848
query29	1033	656	620	620
query30	309	149	147	147
query31	973	719	738	719
query32	95	62	56	56
query33	765	251	251	251
query34	1149	476	496	476
query35	851	620	611	611
query36	1038	933	891	891
query37	274	66	67	66
query38	3675	3461	3463	3461
query39	1507	1477	1436	1436
query40	307	117	115	115
query41	51	48	48	48
query42	108	94	95	94
query43	486	494	460	460
query44	1207	736	729	729
query45	280	256	253	253
query46	1126	686	697	686
query47	1912	1927	1867	1867
query48	452	354	360	354
query49	1248	352	335	335
query50	767	360	375	360
query51	6757	6672	6574	6574
query52	112	93	92	92
query53	346	274	278	274
query54	323	224	237	224
query55	87	77	78	77
query56	242	212	218	212
query57	1223	1145	1135	1135
query58	240	193	202	193
query59	2860	2697	2601	2601
query60	260	235	236	235
query61	93	92	96	92
query62	680	440	439	439
query63	301	278	278	278
query64	6380	4011	4048	4011
query65	3095	3020	3021	3020
query66	1411	383	359	359
query67	15274	14981	14813	14813
query68	5077	512	522	512
query69	547	363	373	363
query70	1203	1175	1139	1139
query71	413	271	255	255
query72	6152	2729	2562	2562
query73	701	314	310	310
query74	6809	6472	6354	6354
query75	2934	2195	2241	2195
query76	3477	894	935	894
query77	378	263	264	263
query78	10812	10205	10087	10087
query79	7113	517	522	517
query80	1710	398	388	388
query81	538	215	222	215
query82	1527	93	86	86
query83	333	150	153	150
query84	294	86	81	81
query85	1730	370	374	370
query86	499	318	293	293
query87	3709	3560	3548	3548
query88	5158	2314	2310	2310
query89	472	379	359	359
query90	1970	175	172	172
query91	168	139	159	139
query92	58	47	46	46
query93	5348	502	474	474
query94	1270	175	174	174
query95	437	329	319	319
query96	601	275	267	267
query97	2660	2473	2479	2473
query98	227	221	205	205
query99	1229	915	924	915
Total cold run time: 306365 ms
Total hot run time: 181824 ms

doris-robot · 2024-03-27T08:54:09Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit ac7a009f83e24c6618094b955c56347cffda13c7 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.7 seconds inserted 10000000 Rows, about 729K ops/s

BePPPower · 2024-03-27T12:10:37Z

run external

BePPPower · 2024-03-28T12:56:18Z

run buildall

BePPPower · 2024-03-29T02:54:14Z

run buildall

doris-robot · 2024-03-29T03:25:48Z

TPC-H: Total hot run time: 38062 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4d16e2b4c1dd8ea1ff54d43571395894b93bd868, data reload: false
------ Round 1 ----------------------------------
q1	17610	4181	4140	4140
q2	2113	164	164	164
q3	10693	1194	1201	1194
q4	10850	851	864	851
q5	7732	3004	2943	2943
q6	210	128	125	125
q7	1069	595	593	593
q8	9660	2100	2027	2027
q9	7139	6540	6428	6428
q10	10061	3499	3558	3499
q11	445	236	234	234
q12	415	203	202	202
q13	17799	2846	2893	2846
q14	241	213	195	195
q15	517	477	460	460
q16	497	365	379	365
q17	965	534	590	534
q18	7246	6545	6507	6507
q19	1542	1422	1508	1422
q20	545	245	254	245
q21	3547	2990	2794	2794
q22	347	294	309	294
Total cold run time: 111243 ms
Total hot run time: 38062 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4105	4125	4066	4066
q2	319	224	225	224
q3	2970	2832	2854	2832
q4	1866	1551	1555	1551
q5	5240	5246	5265	5246
q6	194	116	116	116
q7	2219	1836	1854	1836
q8	3196	3325	3326	3325
q9	8429	8424	8402	8402
q10	3780	3776	3776	3776
q11	542	443	462	443
q12	721	564	528	528
q13	15015	2885	2896	2885
q14	286	253	250	250
q15	495	457	461	457
q16	483	430	424	424
q17	1727	1485	1474	1474
q18	7497	7041	7409	7041
q19	1626	1577	1513	1513
q20	1930	1745	1700	1700
q21	4914	4735	4648	4648
q22	505	433	448	433
Total cold run time: 68059 ms
Total hot run time: 53170 ms

doris-robot · 2024-03-29T03:37:48Z

TPC-DS: Total hot run time: 181506 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4d16e2b4c1dd8ea1ff54d43571395894b93bd868, data reload: false
query1	928	364	342	342
query2	6532	1977	1855	1855
query3	6702	216	218	216
query4	31895	21323	21309	21309
query5	4228	401	395	395
query6	275	195	192	192
query7	4642	294	289	289
query8	230	176	178	176
query9	8986	2259	2246	2246
query10	578	272	286	272
query11	17071	14289	14337	14289
query12	140	93	86	86
query13	1647	426	425	425
query14	10305	7502	7767	7502
query15	261	190	189	189
query16	8223	269	261	261
query17	1993	597	552	552
query18	2100	294	287	287
query19	370	165	154	154
query20	101	86	89	86
query21	204	127	122	122
query22	4968	4842	4815	4815
query23	33615	32975	32911	32911
query24	11922	2865	2889	2865
query25	652	392	384	384
query26	1810	158	158	158
query27	2954	329	316	316
query28	7649	1848	1872	1848
query29	1014	646	618	618
query30	312	147	146	146
query31	967	719	750	719
query32	92	56	55	55
query33	753	266	278	266
query34	952	505	483	483
query35	905	701	673	673
query36	1031	880	886	880
query37	177	67	67	67
query38	3577	3400	3448	3400
query39	1597	1553	1527	1527
query40	289	109	110	109
query41	49	47	50	47
query42	102	97	96	96
query43	479	444	453	444
query44	1203	741	730	730
query45	285	254	261	254
query46	1109	695	721	695
query47	1950	1880	1838	1838
query48	436	375	370	370
query49	1233	344	353	344
query50	769	374	380	374
query51	6780	6637	6703	6637
query52	98	97	96	96
query53	348	272	275	272
query54	318	245	249	245
query55	82	78	82	78
query56	265	238	237	237
query57	1217	1119	1124	1119
query58	241	207	209	207
query59	2890	2710	2586	2586
query60	266	251	254	251
query61	116	116	112	112
query62	666	437	430	430
query63	306	277	278	277
query64	6848	4090	4102	4090
query65	3097	3022	3048	3022
query66	1412	373	349	349
query67	15518	15245	14770	14770
query68	9016	523	531	523
query69	652	382	372	372
query70	1398	1169	1149	1149
query71	508	259	260	259
query72	6561	2532	2363	2363
query73	1637	320	331	320
query74	7314	6465	6289	6289
query75	3805	2219	2167	2167
query76	5801	781	877	781
query77	633	263	262	262
query78	11088	10239	10156	10156
query79	10896	541	540	540
query80	1958	369	371	369
query81	515	214	211	211
query82	712	83	85	83
query83	217	152	158	152
query84	290	76	76	76
query85	1220	297	297	297
query86	364	286	311	286
query87	3757	3597	3535	3535
query88	4597	2408	2415	2408
query89	510	369	370	369
query90	2065	174	184	174
query91	169	144	133	133
query92	59	46	49	46
query93	6933	501	479	479
query94	1288	173	176	173
query95	412	310	305	305
query96	604	284	271	271
query97	2662	2485	2471	2471
query98	230	216	207	207
query99	1122	825	873	825
Total cold run time: 319723 ms
Total hot run time: 181506 ms

doris-robot · 2024-03-29T03:43:07Z

ClickBench: Total hot run time: 30.12 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4d16e2b4c1dd8ea1ff54d43571395894b93bd868, data reload: false
query1	0.04	0.04	0.03
query2	0.06	0.02	0.02
query3	0.23	0.06	0.06
query4	1.65	0.09	0.10
query5	0.50	0.50	0.49
query6	1.18	0.66	0.64
query7	0.02	0.01	0.01
query8	0.03	0.03	0.03
query9	0.54	0.50	0.49
query10	0.56	0.55	0.54
query11	0.12	0.08	0.10
query12	0.11	0.09	0.10
query13	0.59	0.58	0.57
query14	0.74	0.76	0.76
query15	0.81	0.78	0.79
query16	0.38	0.36	0.37
query17	1.04	0.98	1.00
query18	0.22	0.23	0.24
query19	1.83	1.74	1.69
query20	0.02	0.01	0.01
query21	15.55	0.58	0.56
query22	2.74	2.91	1.82
query23	17.23	0.94	0.65
query24	3.43	1.60	2.01
query25	0.20	0.25	0.24
query26	0.67	0.13	0.11
query27	0.03	0.03	0.05
query28	9.51	0.86	0.89
query29	12.65	3.17	3.21
query30	0.28	0.09	0.09
query31	2.81	0.36	0.34
query32	3.33	0.45	0.44
query33	2.78	2.76	2.86
query34	16.17	4.39	4.38
query35	4.41	4.39	4.48
query36	0.67	0.46	0.48
query37	0.08	0.06	0.05
query38	0.04	0.03	0.03
query39	0.03	0.01	0.02
query40	0.17	0.13	0.16
query41	0.08	0.02	0.01
query42	0.03	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 103.59 s
Total hot run time: 30.12 s

doris-robot · 2024-03-29T03:46:13Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 4d16e2b4c1dd8ea1ff54d43571395894b93bd868 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 13.8 seconds inserted 10000000 Rows, about 724K ops/s

morningman

LGTM

github-actions · 2024-03-31T13:58:34Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-03-31T13:58:36Z

PR approved by anyone and no changes requested.

morningman · 2024-03-31T13:58:40Z

Please add case for partition consistency

…y of export to `partition` (apache#32830)

…y of export to `partition` (#32830)

…y of export to `partition` (apache#32830)

fix
1b2968c

BePPPower added 2 commits March 27, 2024 16:03

fix feut
52b84c1

fix p0
ac7a009

BePPPower mentioned this pull request Mar 28, 2024
[fix](Export) fix description of data_consitencyapache/doris-website#482
Merged

morningman added the kind/behavior-changed label Mar 28, 2024

fix 3
5f9fc6f

fix 4
4d16e2b

xiaokang added the dev/2.0.x label Mar 29, 2024

morningman approved these changes Mar 31, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Mar 31, 2024

github-actionsBot added the reviewed label Mar 31, 2024

wuwenchi approved these changes Apr 3, 2024
View reviewed changes

Jibing-Li approved these changes Apr 3, 2024
View reviewed changes

morningman merged commit f978079 into apache:masterApr 3, 2024

morningman pushed a commit to morningman/doris that referenced this pull request Apr 7, 2024
[fix](Export) Set the default value of the data_consistence propert…
8186fbd
…y of export to `partition` (apache#32830)

morningman added the dev/2.1.2-merged label Apr 7, 2024

morningman pushed a commit that referenced this pull request Apr 7, 2024
[fix](Export) Set the default value of the data_consistence propert…
b882704
…y of export to `partition` (#32830)

yiguolei mentioned this pull request Apr 8, 2024
Release Note 2.1.2 #33307
Closed

morningman removed the dev/2.0.x label Apr 17, 2024

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[fix](Export) Set the default value of the data_consistence propert…
a8840d4
…y of export to `partition` (apache#32830)

Uh oh!

Conversation

BePPPower commented Mar 26, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented Mar 26, 2024

Uh oh!

BePPPower commented Mar 26, 2024

Uh oh!

doris-robot commented Mar 26, 2024

Uh oh!

doris-robot commented Mar 26, 2024

Uh oh!

doris-robot commented Mar 26, 2024

Uh oh!

BePPPower commented Mar 27, 2024

Uh oh!

doris-robot commented Mar 27, 2024

Uh oh!

doris-robot commented Mar 27, 2024

Uh oh!

doris-robot commented Mar 27, 2024

Uh oh!

BePPPower commented Mar 27, 2024

Uh oh!

BePPPower commented Mar 28, 2024

Uh oh!

BePPPower commented Mar 29, 2024

Uh oh!

doris-robot commented Mar 29, 2024

Uh oh!

doris-robot commented Mar 29, 2024

Uh oh!

doris-robot commented Mar 29, 2024

Uh oh!

doris-robot commented Mar 29, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Mar 31, 2024

Uh oh!

github-actionsBot commented Mar 31, 2024

Uh oh!

morningman commented Mar 31, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

BePPPower commented Mar 26, 2024•
edited
Loading