[Fix](Export) Export delete multiple times when specify the `delete_existing_files` property by BePPPower · Pull Request #38400 · apache/doris

BePPPower · 2024-07-26T03:42:44Z

When the Export statement specifies the delete_existing_files property, each Outfile statement generated by the Export will carry this property. This causes each Outfile statement to delete existing files, so only the result of the last Outfile statement will be retained.

So, we add a rpc method which can delete existing files for Export statement and the Outfile statements generated by the Export will not carry delete_existing_files property any more.

doris-robot · 2024-07-26T03:42:48Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions · 2024-07-26T03:49:08Z

clang-tidy review says "All clean, LGTM! 👍"

BePPPower · 2024-07-26T03:52:08Z

run buildall

github-actions · 2024-07-26T07:06:53Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-08-01T07:54:34Z

clang-tidy review says "All clean, LGTM! 👍"

BePPPower · 2024-08-01T12:42:37Z

run buildall

github-actions · 2024-08-01T12:49:05Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-08-01T13:58:31Z

TPC-H: Total hot run time: 42039 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3114850b95c79a8985c8c00b8df24e384228b0f9, data reload: false
------ Round 1 ----------------------------------
q1	17626	4535	4093	4093
q2	2033	202	200	200
q3	10579	1262	1377	1262
q4	10217	835	949	835
q5	7662	3061	3003	3003
q6	223	138	145	138
q7	1044	621	613	613
q8	9437	1953	1983	1953
q9	8449	6638	6678	6638
q10	8754	3934	3887	3887
q11	430	243	248	243
q12	408	236	236	236
q13	17762	2973	2984	2973
q14	268	254	248	248
q15	524	482	488	482
q16	512	400	384	384
q17	985	927	903	903
q18	8067	7310	7510	7310
q19	1539	1228	1231	1228
q20	566	329	329	329
q21	5397	4919	4802	4802
q22	351	286	279	279
Total cold run time: 112833 ms
Total hot run time: 42039 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4114	4053	4024	4024
q2	329	222	227	222
q3	2984	3103	3171	3103
q4	2067	2064	2018	2018
q5	5653	5515	5437	5437
q6	221	143	134	134
q7	2155	1847	1835	1835
q8	3404	3415	3395	3395
q9	8651	8759	8768	8759
q10	4028	4052	4029	4029
q11	563	483	469	469
q12	783	582	609	582
q13	13144	3154	3139	3139
q14	304	289	265	265
q15	517	487	480	480
q16	467	405	410	405
q17	1796	1752	1790	1752
q18	8311	7755	7800	7755
q19	1765	1724	1774	1724
q20	2075	1860	1819	1819
q21	5801	5613	5350	5350
q22	547	466	466	466
Total cold run time: 69679 ms
Total hot run time: 57162 ms

doris-robot · 2024-08-01T14:09:15Z

TPC-DS: Total hot run time: 170728 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3114850b95c79a8985c8c00b8df24e384228b0f9, data reload: false
query1	916	388	370	370
query2	6460	1720	1720	1720
query3	6678	225	228	225
query4	20351	17482	17411	17411
query5	3612	517	517	517
query6	281	173	161	161
query7	4594	293	296	293
query8	260	202	189	189
query9	8512	2382	2376	2376
query10	422	284	271	271
query11	10651	10142	10065	10065
query12	120	89	90	89
query13	1622	377	392	377
query14	8376	7249	7680	7249
query15	197	172	174	172
query16	6871	483	489	483
query17	956	562	568	562
query18	1777	299	284	284
query19	196	155	151	151
query20	93	86	90	86
query21	203	103	97	97
query22	4212	4050	3992	3992
query23	33832	33795	33800	33795
query24	10270	3156	3074	3074
query25	715	429	404	404
query26	1640	150	150	150
query27	2862	289	301	289
query28	7401	2015	2033	2015
query29	1145	422	447	422
query30	243	155	155	155
query31	926	800	789	789
query32	103	54	62	54
query33	700	313	342	313
query34	918	497	498	497
query35	880	772	765	765
query36	1036	908	913	908
query37	278	82	88	82
query38	2950	2857	2925	2857
query39	896	809	824	809
query40	258	114	111	111
query41	48	44	49	44
query42	122	97	106	97
query43	476	419	420	419
query44	1184	739	740	739
query45	208	179	176	176
query46	1094	821	795	795
query47	1838	1710	1744	1710
query48	366	295	290	290
query49	955	446	427	427
query50	894	466	433	433
query51	6808	6746	6617	6617
query52	102	92	93	92
query53	260	183	184	183
query54	620	453	459	453
query55	78	78	74	74
query56	267	261	265	261
query57	1141	1040	1051	1040
query58	274	276	261	261
query59	2627	2548	2481	2481
query60	295	266	288	266
query61	95	91	95	91
query62	887	657	649	649
query63	211	180	185	180
query64	5677	1909	1878	1878
query65	3171	3107	3101	3101
query66	1292	342	340	340
query67	15342	14848	14752	14752
query68	4602	570	559	559
query69	452	305	306	305
query70	1204	1075	1060	1060
query71	462	286	283	283
query72	7050	2666	2509	2509
query73	765	335	337	335
query74	6064	5707	5696	5696
query75	3384	2755	2706	2706
query76	2581	1225	1288	1225
query77	464	322	336	322
query78	9565	8951	8844	8844
query79	2666	543	536	536
query80	1154	506	504	504
query81	587	228	233	228
query82	1015	137	137	137
query83	254	176	176	176
query84	274	79	86	79
query85	1318	369	292	292
query86	462	293	290	290
query87	3280	3138	3127	3127
query88	3776	2417	2432	2417
query89	398	292	289	289
query90	1767	198	202	198
query91	127	104	99	99
query92	66	50	51	50
query93	2011	635	621	621
query94	850	296	315	296
query95	383	276	272	272
query96	608	298	287	287
query97	3261	3034	3073	3034
query98	224	205	203	203
query99	1609	1289	1271	1271
Total cold run time: 263428 ms
Total hot run time: 170728 ms

doris-robot · 2024-08-01T14:14:46Z

ClickBench: Total hot run time: 30.09 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3114850b95c79a8985c8c00b8df24e384228b0f9, data reload: false
query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.05	0.04
query4	1.68	0.07	0.07
query5	0.50	0.49	0.48
query6	1.14	0.71	0.71
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.58	0.51	0.51
query10	0.57	0.55	0.56
query11	0.16	0.12	0.12
query12	0.15	0.13	0.12
query13	0.61	0.61	0.60
query14	0.77	0.82	0.80
query15	0.90	0.87	0.86
query16	0.35	0.36	0.36
query17	1.01	1.02	1.01
query18	0.22	0.21	0.21
query19	1.86	1.77	1.74
query20	0.02	0.01	0.01
query21	15.42	0.75	0.65
query22	3.70	8.59	1.29
query23	17.90	1.28	1.23
query24	2.14	0.25	0.22
query25	0.18	0.08	0.08
query26	0.33	0.22	0.21
query27	0.46	0.24	0.23
query28	13.17	1.00	0.98
query29	12.56	3.34	3.31
query30	0.25	0.06	0.05
query31	2.87	0.42	0.40
query32	3.22	0.50	0.49
query33	2.95	2.97	2.97
query34	15.42	4.31	4.27
query35	4.32	4.31	4.30
query36	0.68	0.48	0.49
query37	0.19	0.16	0.16
query38	0.17	0.15	0.16
query39	0.04	0.04	0.03
query40	0.16	0.13	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.06
query43	0.05	0.04	0.04
Total cold run time: 107.25 s
Total hot run time: 30.09 s

github-actions · 2024-08-02T03:40:07Z

clang-tidy review says "All clean, LGTM! 👍"

BePPPower · 2024-08-02T03:56:12Z

run buildall

github-actions · 2024-08-02T04:00:31Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-08-02T04:02:20Z

clang-tidy review says "All clean, LGTM! 👍"

morningman

LGTM

github-actions · 2024-08-02T04:32:04Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-08-02T04:32:06Z

PR approved by anyone and no changes requested.

doris-robot · 2024-08-02T04:58:04Z

TPC-H: Total hot run time: 41677 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b9ad5eea731fd32053370a79bc23861279a2d3bb, data reload: false
------ Round 1 ----------------------------------
q1	17649	4347	4086	4086
q2	2026	200	199	199
q3	10546	1327	1332	1327
q4	10205	846	902	846
q5	7676	3023	2962	2962
q6	225	142	138	138
q7	1053	614	614	614
q8	9429	1924	1939	1924
q9	8434	6591	6613	6591
q10	8752	3821	3846	3821
q11	438	252	253	252
q12	406	229	226	226
q13	17766	2933	2970	2933
q14	266	243	240	240
q15	518	481	491	481
q16	530	405	386	386
q17	977	933	889	889
q18	8052	7237	7255	7237
q19	1556	1219	1210	1210
q20	546	334	336	334
q21	5245	4700	4741	4700
q22	353	281	287	281
Total cold run time: 112648 ms
Total hot run time: 41677 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4109	4069	4099	4069
q2	323	227	238	227
q3	2994	3002	3152	3002
q4	1973	2030	1967	1967
q5	5612	5500	5451	5451
q6	222	135	139	135
q7	2111	1758	1846	1758
q8	3315	3352	3334	3334
q9	8635	8625	8779	8625
q10	3939	4034	3937	3937
q11	550	452	449	449
q12	775	600	605	600
q13	16432	3129	3109	3109
q14	301	273	268	268
q15	531	486	498	486
q16	470	416	429	416
q17	1756	1714	1730	1714
q18	8123	7709	7745	7709
q19	1724	1714	1719	1714
q20	2067	1820	1883	1820
q21	5739	5408	5341	5341
q22	530	461	460	460
Total cold run time: 72231 ms
Total hot run time: 56591 ms

doris-robot · 2024-08-02T05:08:43Z

TPC-DS: Total hot run time: 169454 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b9ad5eea731fd32053370a79bc23861279a2d3bb, data reload: false
query1	924	384	374	374
query2	6509	1641	1668	1641
query3	6659	210	220	210
query4	19046	17437	17313	17313
query5	3664	519	499	499
query6	275	175	157	157
query7	4847	303	296	296
query8	266	205	214	205
query9	8512	2377	2361	2361
query10	433	285	264	264
query11	10530	9947	10045	9947
query12	124	92	90	90
query13	1639	384	384	384
query14	9245	6627	7246	6627
query15	209	171	167	167
query16	6447	450	403	403
query17	938	557	536	536
query18	1102	279	295	279
query19	189	142	142	142
query20	90	88	84	84
query21	209	99	98	98
query22	4306	4139	4041	4041
query23	33731	33539	33833	33539
query24	10303	3068	3051	3051
query25	671	414	389	389
query26	1810	152	154	152
query27	2819	287	285	285
query28	7323	2029	2013	2013
query29	1023	436	465	436
query30	234	153	154	153
query31	931	758	790	758
query32	99	56	55	55
query33	664	316	331	316
query34	888	528	500	500
query35	867	765	770	765
query36	1059	896	887	887
query37	262	88	85	85
query38	2956	2861	2854	2854
query39	928	828	797	797
query40	254	113	111	111
query41	46	44	49	44
query42	124	105	101	101
query43	488	427	438	427
query44	1187	743	748	743
query45	205	193	187	187
query46	1104	816	815	815
query47	1800	1700	1696	1696
query48	372	301	301	301
query49	950	423	432	423
query50	885	437	428	428
query51	6785	6868	6732	6732
query52	100	106	89	89
query53	256	190	191	190
query54	633	475	455	455
query55	80	74	74	74
query56	275	254	250	250
query57	1129	1040	1064	1040
query58	278	283	279	279
query59	2468	2385	2261	2261
query60	283	285	273	273
query61	97	91	95	91
query62	882	665	659	659
query63	217	188	181	181
query64	5840	1928	1893	1893
query65	3147	3091	3097	3091
query66	1194	329	332	329
query67	15459	14927	14862	14862
query68	4385	560	584	560
query69	459	306	311	306
query70	1130	1063	1083	1063
query71	438	283	280	280
query72	7175	2682	2495	2495
query73	762	330	333	330
query74	6023	5625	5571	5571
query75	3394	2711	2820	2711
query76	2657	1218	1288	1218
query77	440	311	313	311
query78	9371	8960	8929	8929
query79	1939	548	533	533
query80	1186	510	513	510
query81	581	228	227	227
query82	1028	133	129	129
query83	239	169	170	169
query84	273	80	84	80
query85	1347	318	302	302
query86	462	304	310	304
query87	3248	3131	3093	3093
query88	2995	2537	2511	2511
query89	383	303	289	289
query90	1732	189	191	189
query91	127	99	100	99
query92	62	49	48	48
query93	2012	626	636	626
query94	839	306	374	306
query95	375	260	265	260
query96	607	288	280	280
query97	3228	3053	3076	3053
query98	222	201	192	192
query99	1672	1302	1305	1302
Total cold run time: 260060 ms
Total hot run time: 169454 ms

doris-robot · 2024-08-02T05:14:13Z

ClickBench: Total hot run time: 29.91 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b9ad5eea731fd32053370a79bc23861279a2d3bb, data reload: false
query1	0.05	0.04	0.04
query2	0.07	0.04	0.04
query3	0.23	0.05	0.04
query4	1.68	0.09	0.08
query5	0.49	0.47	0.48
query6	1.14	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.58	0.51	0.50
query10	0.55	0.56	0.56
query11	0.17	0.11	0.12
query12	0.15	0.11	0.13
query13	0.60	0.60	0.59
query14	0.76	0.77	0.79
query15	0.93	0.87	0.85
query16	0.36	0.35	0.36
query17	1.00	1.00	0.97
query18	0.23	0.22	0.22
query19	1.80	1.75	1.77
query20	0.02	0.01	0.01
query21	15.40	0.77	0.67
query22	4.07	7.41	1.22
query23	17.89	1.34	1.25
query24	2.25	0.22	0.22
query25	0.18	0.07	0.08
query26	0.31	0.21	0.21
query27	0.46	0.23	0.23
query28	13.16	1.00	0.98
query29	12.57	3.31	3.30
query30	0.25	0.06	0.05
query31	2.86	0.40	0.41
query32	3.27	0.48	0.49
query33	2.98	2.99	3.02
query34	15.42	4.23	4.26
query35	4.27	4.30	4.28
query36	0.68	0.48	0.48
query37	0.19	0.17	0.17
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.16	0.13	0.13
query41	0.09	0.04	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 107.65 s
Total hot run time: 29.91 s

…xisting_files` property (#38400) When the `Export` statement specifies the `delete_existing_files` property, each `Outfile` statement generated by the `Export` will carry this property. This causes each `Outfile` statement to delete existing files, so only the result of the last Outfile statement will be retained. So, we add a rpc method which can delete existing files for `Export` statement and the `Outfile` statements generated by the `Export` will not carry `delete_existing_files` property any more.

…xisting_files` property (apache#38400) When the `Export` statement specifies the `delete_existing_files` property, each `Outfile` statement generated by the `Export` will carry this property. This causes each `Outfile` statement to delete existing files, so only the result of the last Outfile statement will be retained. So, we add a rpc method which can delete existing files for `Export` statement and the `Outfile` statements generated by the `Export` will not carry `delete_existing_files` property any more.

…xisting_files` property () (#39304) bp: #38400 When the `Export` statement specifies the `delete_existing_files` property, each `Outfile` statement generated by the `Export` will carry this property. This causes each `Outfile` statement to delete existing files, so only the result of the last Outfile statement will be retained. So, we add a rpc method which can delete existing files for `Export` statement and the `Outfile` statements generated by the `Export` will not carry `delete_existing_files` property any more. ## Proposed changes Issue Number: close #xxx

…xisting_files` property (#38400) When the `Export` statement specifies the `delete_existing_files` property, each `Outfile` statement generated by the `Export` will carry this property. This causes each `Outfile` statement to delete existing files, so only the result of the last Outfile statement will be retained. So, we add a rpc method which can delete existing files for `Export` statement and the `Outfile` statements generated by the `Export` will not carry `delete_existing_files` property any more.

…61223) ### What problem does this PR solve? Issue Number: N/A Related PR: #38400 Problem Summary: When `select ... into outfile` uses `delete_existing_files=true`, parallel outfile writers can race on directory cleanup and delete files uploaded by other writers. This PR follows the same FE-side cleanup pattern used by export in #38400: remote outfile cleanup is executed once in FE before query execution, and the delete flag is cleared before sink options are sent to BE. This PR also aligns local outfile behavior with export: `file:///` does not support `delete_existing_files=true`, so FE rejects that combination during analysis instead of relying on BE-side cleanup. To reduce duplicated logic, the FE-side parent-directory cleanup used by export/outfile/TVF is refactored into shared `BrokerUtil` helpers.

…pache#61223) Issue Number: N/A Related PR: apache#38400 Problem Summary: When `select ... into outfile` uses `delete_existing_files=true`, parallel outfile writers can race on directory cleanup and delete files uploaded by other writers. This PR follows the same FE-side cleanup pattern used by export in apache#38400: remote outfile cleanup is executed once in FE before query execution, and the delete flag is cleared before sink options are sent to BE. This PR also aligns local outfile behavior with export: `file:///` does not support `delete_existing_files=true`, so FE rejects that combination during analysis instead of relying on BE-side cleanup. To reduce duplicated logic, the FE-side parent-directory cleanup used by export/outfile/TVF is refactored into shared `BrokerUtil` helpers. (cherry picked from commit 1576653)

…pache#61223) ### What problem does this PR solve? Issue Number: N/A Related PR: apache#38400 Problem Summary: When `select ... into outfile` uses `delete_existing_files=true`, parallel outfile writers can race on directory cleanup and delete files uploaded by other writers. This PR follows the same FE-side cleanup pattern used by export in apache#38400: remote outfile cleanup is executed once in FE before query execution, and the delete flag is cleared before sink options are sent to BE. This PR also aligns local outfile behavior with export: `file:///` does not support `delete_existing_files=true`, so FE rejects that combination during analysis instead of relying on BE-side cleanup. To reduce duplicated logic, the FE-side parent-directory cleanup used by export/outfile/TVF is refactored into shared `BrokerUtil` helpers.

…xisting_files` property (apache#38400) When the `Export` statement specifies the `delete_existing_files` property, each `Outfile` statement generated by the `Export` will carry this property. This causes each `Outfile` statement to delete existing files, so only the result of the last Outfile statement will be retained. So, we add a rpc method which can delete existing files for `Export` statement and the `Outfile` statements generated by the `Export` will not carry `delete_existing_files` property any more.

github-actionsBot added the meta-change label Jul 26, 2024

BePPPower added 5 commits August 1, 2024 15:41

fix 1
50dcce5

fix 2
de4b064

fix 3
c2c22b3

fix 4
23c1ffb

fix 5
e5c2aeb

BePPPower force-pushed the exportDeleteExistFilesRpc branch from ea5d257 to e5c2aebCompare August 1, 2024 07:47

github-actionsBot added the doing label Aug 1, 2024

fix 6
3114850

fix 7
425ef70

BePPPower added 2 commits August 2, 2024 11:53

fix 8
f1a091e

fix 9
b9ad5ee

morningman approved these changes Aug 2, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 2, 2024

github-actionsBot added the reviewed label Aug 2, 2024

morningman added the dev/2.1.x label Aug 2, 2024

CalvinKirs approved these changes Aug 5, 2024
View reviewed changes

morningman merged commit 07e13bb into apache:masterAug 6, 2024

dataroaring added the dev/3.0.2-merged label Aug 12, 2024

BePPPower mentioned this pull request Aug 13, 2024
[Fix](Export) Export delete multiple times when specify the delete_existing_files property () #39304
Merged

yiguolei added dev/2.1.6-merged and removed dev/2.1.x labels Aug 13, 2024

suxiaogang223 mentioned this pull request Mar 12, 2026
[fix](outfile) handle delete_existing_files before parallel export #61223
Merged
15 tasks

Uh oh!

Conversation

BePPPower commented Jul 26, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doris-robot commented Jul 26, 2024

Uh oh!

github-actionsBot commented Jul 26, 2024

Uh oh!

BePPPower commented Jul 26, 2024

Uh oh!

github-actionsBot commented Jul 26, 2024

Uh oh!

github-actionsBot commented Aug 1, 2024

Uh oh!

BePPPower commented Aug 1, 2024

Uh oh!

github-actionsBot commented Aug 1, 2024

Uh oh!

doris-robot commented Aug 1, 2024

Uh oh!

doris-robot commented Aug 1, 2024

Uh oh!

doris-robot commented Aug 1, 2024

Uh oh!

github-actionsBot commented Aug 2, 2024

Uh oh!

BePPPower commented Aug 2, 2024

Uh oh!

github-actionsBot commented Aug 2, 2024

Uh oh!

github-actionsBot commented Aug 2, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Aug 2, 2024

Uh oh!

github-actionsBot commented Aug 2, 2024

Uh oh!

doris-robot commented Aug 2, 2024

Uh oh!

doris-robot commented Aug 2, 2024

Uh oh!

doris-robot commented Aug 2, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

BePPPower commented Jul 26, 2024•
edited
Loading