[fix](oss) the write to hive table on oss-hdfs may fail by morningman · Pull Request #49754 · apache/doris

morningman · 2025-04-02T02:44:31Z

What problem does this PR solve?

Problem Summary:

When insert data to a hive table on oss-hdfs, it may fail with following error:

Failed to delete directories for files: [oss://xxx]

This is because for oss-hdfs, we should use hadoop filesystem to do the operation.

This PR fix it.
When calling getFSIdentity(), we should pass the properties so that LocationPath can identify
the right fs type.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-04-02T02:44:35Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

morningman · 2025-04-02T02:46:10Z

run buildall

github-actions · 2025-04-02T02:58:21Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-04-02T02:58:23Z

PR approved by anyone and no changes requested.

doris-robot · 2025-04-02T03:05:11Z

TPC-H: Total hot run time: 33943 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a9d74d959828debbf9f947f2a0b5fc05afac86ec, data reload: false
------ Round 1 ----------------------------------
q1	26158	5046	5065	5046
q2	2133	296	195	195
q3	10471	1243	674	674
q4	10219	981	538	538
q5	7540	2820	2324	2324
q6	193	161	132	132
q7	919	746	610	610
q8	9290	1225	1039	1039
q9	6723	5007	5041	5007
q10	6854	2282	1858	1858
q11	477	289	269	269
q12	356	345	223	223
q13	17785	3705	3135	3135
q14	233	225	213	213
q15	546	477	501	477
q16	613	610	593	593
q17	578	846	352	352
q18	7620	7090	7033	7033
q19	2258	1326	563	563
q20	321	341	218	218
q21	4172	2595	2483	2483
q22	1051	1020	961	961
Total cold run time: 116510 ms
Total hot run time: 33943 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5272	5165	5083	5083
q2	245	325	227	227
q3	2125	2667	2291	2291
q4	1426	1851	1530	1530
q5	4467	4429	4369	4369
q6	214	169	131	131
q7	1974	1948	1755	1755
q8	2569	2444	2505	2444
q9	7282	7076	6978	6978
q10	2988	3182	2734	2734
q11	594	519	485	485
q12	714	761	626	626
q13	3577	3881	3379	3379
q14	317	286	269	269
q15	525	479	471	471
q16	655	675	647	647
q17	1163	1592	1358	1358
q18	7739	7542	7357	7357
q19	811	808	977	808
q20	1944	2000	1894	1894
q21	5264	4868	4745	4745
q22	1099	1044	1045	1044
Total cold run time: 52964 ms
Total hot run time: 50625 ms

doris-robot · 2025-04-02T03:17:28Z

TPC-DS: Total hot run time: 193586 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a9d74d959828debbf9f947f2a0b5fc05afac86ec, data reload: false
query1	1395	1046	1037	1037
query2	6389	1957	1920	1920
query3	10983	4418	4568	4418
query4	54982	25644	23394	23394
query5	5073	531	475	475
query6	326	209	197	197
query7	4869	515	286	286
query8	333	254	235	235
query9	5625	2591	2607	2591
query10	411	327	268	268
query11	15098	15088	14731	14731
query12	164	115	110	110
query13	1050	519	400	400
query14	10223	6396	6498	6396
query15	201	190	185	185
query16	7070	675	487	487
query17	1077	771	614	614
query18	1604	420	314	314
query19	221	194	164	164
query20	128	127	121	121
query21	217	128	114	114
query22	4315	4463	4332	4332
query23	33993	33593	33299	33299
query24	6733	2447	2449	2447
query25	472	461	396	396
query26	730	276	149	149
query27	2504	512	350	350
query28	3143	2473	2449	2449
query29	594	573	454	454
query30	284	247	194	194
query31	871	877	822	822
query32	74	67	60	60
query33	459	393	316	316
query34	788	865	531	531
query35	823	857	777	777
query36	937	1019	935	935
query37	118	100	79	79
query38	4149	4217	4143	4143
query39	1467	1434	1445	1434
query40	223	124	111	111
query41	56	56	55	55
query42	127	110	111	110
query43	503	530	494	494
query44	1353	831	832	831
query45	181	176	166	166
query46	843	1042	651	651
query47	1890	1897	1763	1763
query48	387	429	317	317
query49	674	581	464	464
query50	665	697	443	443
query51	4289	4284	4268	4268
query52	116	110	108	108
query53	237	264	190	190
query54	597	584	538	538
query55	85	85	83	83
query56	307	304	291	291
query57	1173	1210	1103	1103
query58	272	260	272	260
query59	2644	2813	2785	2785
query60	337	333	313	313
query61	137	133	132	132
query62	745	760	664	664
query63	223	190	185	185
query64	2007	1102	728	728
query65	4386	4462	4373	4373
query66	756	411	306	306
query67	15889	15431	15331	15331
query68	7802	835	529	529
query69	541	304	265	265
query70	1202	1150	1115	1115
query71	527	313	287	287
query72	6053	4823	4886	4823
query73	1405	672	349	349
query74	9017	8884	8818	8818
query75	3857	3225	2765	2765
query76	4246	1211	763	763
query77	610	370	286	286
query78	10091	10171	9240	9240
query79	2292	819	579	579
query80	611	531	442	442
query81	483	259	234	234
query82	453	125	97	97
query83	262	257	229	229
query84	371	110	86	86
query85	784	361	319	319
query86	385	291	274	274
query87	4447	4437	4543	4437
query88	3557	2225	2334	2225
query89	408	316	279	279
query90	1767	212	208	208
query91	144	147	115	115
query92	75	61	55	55
query93	1784	962	579	579
query94	660	414	299	299
query95	377	294	292	292
query96	486	559	279	279
query97	3179	3193	3155	3155
query98	242	205	204	204
query99	1404	1385	1272	1272
Total cold run time: 300517 ms
Total hot run time: 193586 ms

doris-robot · 2025-04-02T03:22:57Z

ClickBench: Total hot run time: 31.65 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a9d74d959828debbf9f947f2a0b5fc05afac86ec, data reload: false
query1	0.04	0.04	0.03
query2	0.12	0.10	0.11
query3	0.26	0.19	0.20
query4	1.59	0.20	0.11
query5	0.55	0.54	0.54
query6	1.21	0.72	0.71
query7	0.02	0.02	0.02
query8	0.05	0.03	0.04
query9	0.57	0.52	0.53
query10	0.59	0.60	0.57
query11	0.16	0.11	0.11
query12	0.15	0.11	0.11
query13	0.61	0.59	0.60
query14	2.80	2.80	2.67
query15	0.91	0.84	0.86
query16	0.38	0.38	0.38
query17	1.02	1.03	1.05
query18	0.20	0.20	0.20
query19	2.07	1.76	1.86
query20	0.01	0.01	0.01
query21	15.36	0.88	0.54
query22	0.78	1.33	0.86
query23	14.71	1.37	0.61
query24	6.83	1.90	1.36
query25	0.48	0.16	0.13
query26	0.59	0.16	0.13
query27	0.06	0.05	0.05
query28	10.34	0.86	0.43
query29	12.53	3.90	3.28
query30	0.25	0.09	0.06
query31	2.83	0.58	0.39
query32	3.23	0.55	0.47
query33	3.04	3.06	3.06
query34	15.71	5.06	4.49
query35	4.45	4.46	4.48
query36	0.65	0.49	0.49
query37	0.08	0.07	0.06
query38	0.06	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 105.64 s
Total hot run time: 31.65 s

### What problem does this PR solve? Problem Summary: When insert data to a hive table on oss-hdfs, it may fail with following error: ``` Failed to delete directories for files: [oss://xxx] ``` This is because for oss-hdfs, we should use hadoop filesystem to do the operation. This PR fix it. When calling `getFSIdentity()`, we should pass the properties so that `LocationPath` can identify the right fs type.

Problem Summary: When insert data to a hive table on oss-hdfs, it may fail with following error: ``` Failed to delete directories for files: [oss://xxx] ``` This is because for oss-hdfs, we should use hadoop filesystem to do the operation. This PR fix it. When calling `getFSIdentity()`, we should pass the properties so that `LocationPath` can identify the right fs type.

…49754) (#49911) bp #49754

Problem Summary: When insert data to a hive table on oss-hdfs, it may fail with following error: ``` Failed to delete directories for files: [oss://xxx] ``` This is because for oss-hdfs, we should use hadoop filesystem to do the operation. This PR fix it. When calling `getFSIdentity()`, we should pass the properties so that `LocationPath` can identify the right fs type.

…49754) (#50284) bp #49754

### What problem does this PR solve? Problem Summary: When insert data to a hive table on oss-hdfs, it may fail with following error: ``` Failed to delete directories for files: [oss://xxx] ``` This is because for oss-hdfs, we should use hadoop filesystem to do the operation. This PR fix it. When calling `getFSIdentity()`, we should pass the properties so that `LocationPath` can identify the right fs type.

2
a9d74d9

morningman added usercase Important user case type label dev/2.1.x dev/3.0.x labels Apr 2, 2025

CalvinKirs approved these changes Apr 2, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Apr 2, 2025

github-actionsBot added the reviewed label Apr 2, 2025

wuwenchi approved these changes Apr 2, 2025
View reviewed changes

morningman merged commit c185bd7 into apache:masterApr 3, 2025

github-actionsBot added dev/3.0.x-conflict dev/2.1.x-conflict labels Apr 3, 2025

morningman mentioned this pull request Apr 9, 2025
branch-2.1: [fix](oss) the write to hive table on oss-hdfs may fail (#49754) #49911
Merged

yiguolei pushed a commit that referenced this pull request Apr 10, 2025
branch-2.1: [fix](oss) the write to hive table on oss-hdfs may fail (#…
bfe7469
…49754) (#49911) bp #49754

yiguolei added dev/2.1.10-merged and removed dev/2.1.x dev/2.1.x-conflict labels Apr 10, 2025

morningman mentioned this pull request Apr 22, 2025
branch-3.0: [fix](oss) the write to hive table on oss-hdfs may fail (#49754) #50284
Merged

dataroaring pushed a commit that referenced this pull request Apr 23, 2025
branch-3.0: [fix](oss) the write to hive table on oss-hdfs may fail (#…
ebbe891
…49754) (#50284) bp #49754

dataroaring added the dev/3.0.5-merged label Apr 23, 2025

dataroaring added dev/3.0.6-merged and removed dev/3.0.x dev/3.0.x-conflict dev/3.0.5-merged labels Apr 23, 2025

yiguolei mentioned this pull request May 13, 2025
2.1.10 Release Note #50853
Closed

Navigation Menu

Uh oh!

[fix](oss) the write to hive table on oss-hdfs may fail - #49754

[fix](oss) the write to hive table on oss-hdfs may fail #49754
morningman merged 1 commit into
apache:masterfrom
morningman:location_del

morningman commented Apr 2, 2025•
edited
Loading

Uh oh!

Thearas commented Apr 2, 2025

Uh oh!

morningman commented Apr 2, 2025

Uh oh!

github-actionsBot commented Apr 2, 2025

Uh oh!

github-actionsBot commented Apr 2, 2025

Uh oh!

doris-robot commented Apr 2, 2025

Uh oh!

doris-robot commented Apr 2, 2025

Uh oh!

doris-robot commented Apr 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Conversation

morningman commented Apr 2, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Apr 2, 2025

Uh oh!

morningman commented Apr 2, 2025

Uh oh!

github-actionsBot commented Apr 2, 2025

Uh oh!

github-actionsBot commented Apr 2, 2025

Uh oh!

doris-robot commented Apr 2, 2025

Uh oh!

doris-robot commented Apr 2, 2025

Uh oh!

doris-robot commented Apr 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

morningman commented Apr 2, 2025•
edited
Loading