[bugfix](export command) Export exceptions when use legacy optimizer by liugddx · Pull Request #31020 · apache/doris

liugddx · 2024-02-12T02:52:01Z

Proposed changes

Issue Number: close#30645

Exception information will not be thrown when exporting exceptions, theck this #30645

The actual exception is Exporting results to local disk is not allowed.To enable this feature, you need to add enable_outfile_to_local=true in fe.conf and restart FE

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-02-12T02:52:05Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

liugddx · 2024-02-12T02:53:43Z

run buildall

liugddx · 2024-02-12T07:10:07Z

run buildall

doris-robot · 2024-02-12T07:41:39Z

TPC-H: Total hot run time: 41118 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit add5272568e543bf6ae5fddc3f0f323256027ab9, data reload: false
------ Round 1 ----------------------------------
q1	17692	4920	4870	4870
q2	2033	142	135	135
q3	10580	1042	992	992
q4	4648	960	972	960
q5	7680	3208	3250	3208
q6	193	133	130	130
q7	1274	779	749	749
q8	9231	2091	2051	2051
q9	7573	6657	6647	6647
q10	8295	2647	2639	2639
q11	405	199	205	199
q12	794	324	329	324
q13	17969	3642	3666	3642
q14	294	260	253	253
q15	585	518	509	509
q16	467	423	409	409
q17	921	861	764	764
q18	7511	6606	6580	6580
q19	1539	1506	1486	1486
q20	628	338	338	338
q21	6193	3891	3952	3891
q22	867	345	342	342
Total cold run time: 107372 ms
Total hot run time: 41118 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4848	4862	4846	4846
q2	291	181	186	181
q3	3594	3579	3582	3579
q4	2520	2512	2524	2512
q5	5777	5735	5739	5735
q6	213	126	121	121
q7	2211	1632	1665	1632
q8	2945	3076	3066	3066
q9	8632	8635	8722	8635
q10	6754	4234	4248	4234
q11	514	382	373	373
q12	769	547	545	545
q13	4292	3373	3387	3373
q14	264	249	243	243
q15	601	499	497	497
q16	469	430	441	430
q17	1640	1578	1580	1578
q18	8307	7583	7649	7583
q19	1619	1640	1632	1632
q20	2102	1822	1830	1822
q21	6497	6144	6094	6094
q22	547	511	517	511
Total cold run time: 65406 ms
Total hot run time: 59222 ms

doris-robot · 2024-02-12T07:53:08Z

TPC-DS: Total hot run time: 180804 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit add5272568e543bf6ae5fddc3f0f323256027ab9, data reload: false
query1	960	354	345	345
query2	6538	1703	1771	1703
query3	6691	220	208	208
query4	24557	22547	22434	22434
query5	4297	460	383	383
query6	266	169	184	169
query7	4597	304	295	295
query8	250	194	222	194
query9	8431	2788	2778	2778
query10	417	228	231	228
query11	16039	15621	15549	15549
query12	140	84	80	80
query13	1686	430	415	415
query14	9519	7692	7634	7634
query15	221	187	189	187
query16	7581	263	261	261
query17	1388	559	542	542
query18	1952	270	266	266
query19	187	149	150	149
query20	88	87	84	84
query21	194	128	114	114
query22	5029	4787	4857	4787
query23	32485	31608	31619	31608
query24	12649	3429	3367	3367
query25	650	367	369	367
query26	1868	152	159	152
query27	3054	319	314	314
query28	6578	1803	1799	1799
query29	1099	611	614	611
query30	279	136	146	136
query31	977	764	770	764
query32	99	58	54	54
query33	723	254	229	229
query34	1059	495	490	490
query35	931	799	832	799
query36	995	875	935	875
query37	166	62	61	61
query38	3366	3302	3337	3302
query39	1343	1323	1295	1295
query40	283	111	101	101
query41	37	35	35	35
query42	104	100	100	100
query43	485	457	453	453
query44	1082	674	698	674
query45	197	188	177	177
query46	1041	775	760	760
query47	1616	1562	1620	1562
query48	416	350	346	346
query49	1199	308	305	305
query50	775	388	391	388
query51	5313	5184	5197	5184
query52	101	94	99	94
query53	397	300	299	299
query54	290	225	224	224
query55	84	81	87	81
query56	217	197	204	197
query57	1071	919	947	919
query58	221	206	198	198
query59	2313	2120	2316	2120
query60	238	221	229	221
query61	88	85	84	84
query62	591	383	358	358
query63	320	288	282	282
query64	6227	3067	3153	3067
query65	3288	3242	3235	3235
query66	1339	332	321	321
query67	14803	14414	14451	14414
query68	5120	574	562	562
query69	511	348	366	348
query70	1332	1254	1281	1254
query71	432	249	256	249
query72	6278	2826	2664	2664
query73	700	319	319	319
query74	7100	6679	6739	6679
query75	3205	2550	2545	2545
query76	3304	1131	1189	1131
query77	352	242	237	237
query78	9477	8748	8739	8739
query79	964	516	508	508
query80	504	356	355	355
query81	439	201	205	201
query82	164	80	85	80
query83	144	124	125	124
query84	221	76	76	76
query85	1001	336	343	336
query86	294	309	305	305
query87	3598	3407	3429	3407
query88	2698	2291	2270	2270
query89	440	355	350	350
query90	1867	166	173	166
query91	161	124	128	124
query92	56	55	47	47
query93	1003	521	517	517
query94	1139	176	177	176
query95	8549	8417	370	370
query96	577	266	267	266
query97	4442	4285	4264	4264
query98	213	213	200	200
query99	1074	727	708	708
Total cold run time: 280844 ms
Total hot run time: 180804 ms

doris-robot · 2024-02-12T07:58:32Z

ClickBench: Total hot run time: 30.8 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit add5272568e543bf6ae5fddc3f0f323256027ab9, data reload: false
query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.07
query4	1.65	0.08	0.08
query5	0.49	0.48	0.49
query6	1.35	0.62	0.61
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.52	0.45	0.46
query10	0.50	0.49	0.49
query11	0.13	0.09	0.09
query12	0.13	0.10	0.10
query13	0.59	0.59	0.59
query14	0.79	0.78	0.78
query15	0.82	0.79	0.80
query16	0.33	0.33	0.35
query17	0.90	0.88	0.93
query18	0.18	0.20	0.19
query19	1.75	1.71	1.70
query20	0.02	0.01	0.02
query21	15.41	0.61	0.58
query22	3.63	3.62	2.07
query23	17.32	0.95	0.96
query24	2.00	0.56	0.58
query25	0.62	0.06	0.06
query26	0.17	0.13	0.13
query27	0.06	0.05	0.04
query28	11.83	0.82	0.82
query29	12.70	3.23	3.32
query30	0.51	0.48	0.48
query31	2.77	0.37	0.36
query32	3.35	0.48	0.48
query33	3.13	3.14	3.13
query34	15.39	4.46	4.52
query35	4.50	4.48	4.48
query36	1.08	0.96	0.96
query37	0.06	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.01	0.02
query40	0.16	0.15	0.15
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.02	0.02
Total cold run time: 105.4 s
Total hot run time: 30.8 s

doris-robot · 2024-02-12T08:01:36Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit add5272568e543bf6ae5fddc3f0f323256027ab9 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.5 seconds inserted 10000000 Rows, about 740K ops/s

liugddx · 2024-02-12T12:30:29Z

run buildall

liugddx · 2024-02-12T15:14:42Z

run buildall

doris-robot · 2024-02-12T15:40:28Z

TPC-H: Total hot run time: 41942 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 031b3b74d7c62587587c499e487a54ba410ef81a, data reload: false
------ Round 1 ----------------------------------
q1	17696	5147	5025	5025
q2	2061	148	133	133
q3	10586	1021	1015	1015
q4	4747	986	997	986
q5	7715	3263	3289	3263
q6	198	138	135	135
q7	1297	784	771	771
q8	9364	2080	2073	2073
q9	7489	6746	6714	6714
q10	8346	2653	2661	2653
q11	421	211	230	211
q12	763	345	347	345
q13	18046	3733	3745	3733
q14	298	262	254	254
q15	625	508	530	508
q16	475	418	429	418
q17	920	870	833	833
q18	7482	6679	6693	6679
q19	1551	1494	1493	1493
q20	601	370	357	357
q21	6441	3982	4055	3982
q22	910	361	363	361
Total cold run time: 108032 ms
Total hot run time: 41942 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4990	5004	5011	5004
q2	298	195	185	185
q3	3663	3648	3653	3648
q4	2627	2634	2634	2634
q5	5840	5810	5806	5806
q6	213	125	129	125
q7	2289	1724	1692	1692
q8	3099	3185	3149	3149
q9	8845	8883	8808	8808
q10	6815	4271	4264	4264
q11	528	400	397	397
q12	826	576	577	576
q13	7044	3444	3431	3431
q14	276	247	260	247
q15	603	507	505	505
q16	489	461	497	461
q17	1696	1651	1653	1651
q18	8394	7611	7705	7611
q19	1642	1638	1642	1638
q20	2130	1858	1844	1844
q21	6754	6432	6487	6432
q22	638	534	552	534
Total cold run time: 69699 ms
Total hot run time: 60642 ms

doris-robot · 2024-02-12T15:52:09Z

TPC-DS: Total hot run time: 184554 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 031b3b74d7c62587587c499e487a54ba410ef81a, data reload: false
query1	935	366	356	356
query2	6532	1997	1961	1961
query3	6706	229	222	222
query4	24505	22667	22545	22545
query5	4263	494	477	477
query6	275	177	192	177
query7	4618	319	306	306
query8	262	211	222	211
query9	8420	3007	2989	2989
query10	422	233	240	233
query11	16155	15762	15642	15642
query12	152	91	87	87
query13	1698	457	453	453
query14	9274	7974	8017	7974
query15	220	193	208	193
query16	7403	274	262	262
query17	1388	568	550	550
query18	1970	289	277	277
query19	207	163	155	155
query20	95	85	87	85
query21	198	125	127	125
query22	5025	4909	4804	4804
query23	32638	31782	31763	31763
query24	12785	3466	3554	3466
query25	662	381	376	376
query26	1891	165	167	165
query27	3040	325	332	325
query28	6588	1946	1953	1946
query29	1215	672	687	672
query30	290	141	153	141
query31	939	765	792	765
query32	100	61	56	56
query33	731	248	245	245
query34	1064	530	532	530
query35	955	865	856	856
query36	1049	965	926	926
query37	192	65	70	65
query38	3453	3353	3347	3347
query39	1411	1356	1361	1356
query40	287	109	111	109
query41	39	36	37	36
query42	114	110	110	110
query43	509	477	493	477
query44	1085	735	741	735
query45	210	192	184	184
query46	1072	842	788	788
query47	1726	1612	1648	1612
query48	473	394	391	391
query49	1247	327	327	327
query50	793	412	408	408
query51	5321	5216	5224	5216
query52	112	98	100	98
query53	427	324	329	324
query54	292	232	247	232
query55	87	84	94	84
query56	241	212	211	211
query57	1073	970	1004	970
query58	224	211	215	211
query59	2522	2433	2278	2278
query60	257	240	244	240
query61	89	88	86	86
query62	602	389	383	383
query63	352	305	313	305
query64	6387	3158	3200	3158
query65	3308	3291	3275	3275
query66	1361	334	336	334
query67	14826	14427	14687	14427
query68	5156	594	607	594
query69	550	400	404	400
query70	1359	1245	1293	1245
query71	454	268	270	268
query72	6377	2883	2661	2661
query73	732	340	336	336
query74	7106	6715	6715	6715
query75	3265	2579	2583	2579
query76	3284	1190	1213	1190
query77	375	262	253	253
query78	9534	8916	8846	8846
query79	1040	542	539	539
query80	546	369	369	369
query81	444	221	216	216
query82	163	92	96	92
query83	153	134	132	132
query84	226	84	87	84
query85	1070	384	363	363
query86	332	328	325	325
query87	3667	3526	3417	3417
query88	2884	2456	2460	2456
query89	485	382	376	376
query90	2086	177	168	168
query91	175	146	150	146
query92	58	50	54	50
query93	1044	558	545	545
query94	1238	188	195	188
query95	8901	390	8680	390
query96	619	277	292	277
query97	4454	4303	4317	4303
query98	232	210	201	201
query99	1084	737	762	737
Total cold run time: 283855 ms
Total hot run time: 184554 ms

doris-robot · 2024-02-12T15:57:36Z

ClickBench: Total hot run time: 31.5 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 031b3b74d7c62587587c499e487a54ba410ef81a, data reload: false
query1	0.03	0.02	0.02
query2	0.06	0.03	0.03
query3	0.23	0.08	0.08
query4	1.63	0.09	0.09
query5	0.51	0.49	0.49
query6	1.39	0.64	0.63
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.47	0.45
query10	0.51	0.49	0.50
query11	0.13	0.10	0.10
query12	0.12	0.10	0.11
query13	0.60	0.59	0.59
query14	0.77	0.81	0.80
query15	0.82	0.83	0.80
query16	0.33	0.34	0.35
query17	0.94	0.94	0.96
query18	0.19	0.18	0.18
query19	1.81	1.77	1.75
query20	0.01	0.02	0.01
query21	15.41	0.62	0.59
query22	3.04	4.64	2.17
query23	17.54	1.06	1.01
query24	2.00	0.45	0.49
query25	0.57	0.11	0.06
query26	0.18	0.15	0.14
query27	0.06	0.05	0.05
query28	11.98	0.87	0.85
query29	12.61	3.37	3.36
query30	0.61	0.53	0.51
query31	2.79	0.36	0.37
query32	3.32	0.49	0.49
query33	3.19	3.17	3.21
query34	15.36	4.55	4.56
query35	4.55	4.54	4.53
query36	1.13	1.01	0.99
query37	0.08	0.05	0.05
query38	0.05	0.03	0.03
query39	0.02	0.02	0.02
query40	0.18	0.14	0.14
query41	0.08	0.01	0.01
query42	0.03	0.01	0.01
query43	0.03	0.02	0.02
Total cold run time: 105.52 s
Total hot run time: 31.5 s

doris-robot · 2024-02-12T16:00:48Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 031b3b74d7c62587587c499e487a54ba410ef81a with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 62 seconds loaded 1101869774 Bytes, about 16 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 13.5 seconds inserted 10000000 Rows, about 740K ops/s

liugddx · 2024-02-13T00:20:13Z

run feut

github-actions · 2024-02-18T03:09:15Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-02-18T03:09:18Z

PR approved by anyone and no changes requested.

…31020)

…pache#31020)

1. Forbid rollback to the old optimizer in `Export` task. Since originStmt is empty, reverting to the old optimizer when the new optimizer is enabled is meaningless. 2. Display `parallelism` in 'show export'. 3. Create an initial Map for resultAttachedInfo to avoid NullPointerException. 4. Remove the originStmt added in this PR #31020, because the `Export` statement is underlying the Outfile statement, it cannot be treated as `OriginStmt`.

…pache#31020)

1. Forbid rollback to the old optimizer in `Export` task. Since originStmt is empty, reverting to the old optimizer when the new optimizer is enabled is meaningless. 2. Display `parallelism` in 'show export'. 3. Create an initial Map for resultAttachedInfo to avoid NullPointerException. 4. Remove the originStmt added in this PR apache#31020, because the `Export` statement is underlying the Outfile statement, it cannot be treated as `OriginStmt`.

liugddx force-pushed the bugfix-22 branch from 4a199a7 to d1939b7Compare February 12, 2024 10:06

1
50351d2

liugddx force-pushed the bugfix-22 branch from d1939b7 to 50351d2Compare February 12, 2024 10:23

1
1bd5202

1
031b3b7

liugddx changed the title ~~[bugfix](export command) Export exceptions are thrown directly~~[bugfix](export command) Export exceptions when use legacy optimizerFeb 13, 2024

morrySnow approved these changes Feb 18, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Feb 18, 2024

github-actionsBot added the reviewed label Feb 18, 2024

yiguolei approved these changes Feb 18, 2024
View reviewed changes

yiguolei merged commit 8416f69 into apache:masterFeb 18, 2024

yiguolei pushed a commit that referenced this pull request Feb 18, 2024
[bugfix](export command) Export exceptions when use legacy optimizer (#…
753ff10
…31020)

xiaokang added the dev/2.0.5 label Feb 18, 2024

mymeiyi pushed a commit to mymeiyi/doris that referenced this pull request Feb 19, 2024
[bugfix](export command) Export exceptions when use legacy optimizer (a…
e99ff6f
…pache#31020)

BePPPower mentioned this pull request Apr 30, 2024
[fix](Export) fix some issues of Export #34345
Merged

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[bugfix](export command) Export exceptions when use legacy optimizer (a…
0f9f42c
…pache#31020)

Uh oh!

Conversation

liugddx commented Feb 12, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented Feb 12, 2024

Uh oh!

liugddx commented Feb 12, 2024

Uh oh!

liugddx commented Feb 12, 2024

Uh oh!

doris-robot commented Feb 12, 2024

Uh oh!

doris-robot commented Feb 12, 2024

Uh oh!

doris-robot commented Feb 12, 2024

Uh oh!

doris-robot commented Feb 12, 2024

Uh oh!

liugddx commented Feb 12, 2024

Uh oh!

liugddx commented Feb 12, 2024

Uh oh!

doris-robot commented Feb 12, 2024

Uh oh!

doris-robot commented Feb 12, 2024

Uh oh!

doris-robot commented Feb 12, 2024

Uh oh!

doris-robot commented Feb 12, 2024

Uh oh!

liugddx commented Feb 13, 2024

Uh oh!

github-actionsBot commented Feb 18, 2024

Uh oh!

github-actionsBot commented Feb 18, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

liugddx commented Feb 12, 2024•
edited
Loading