branch-3.0: [fix](hive/iceberg) rectify the fs name if path already contains fs #49998 by github-actions[bot] · Pull Request #50197 · apache/doris

github-actions · 2025-04-19T23:44:01Z

Cherry-picked from #49998

…49998) ### What problem does this PR solve? In this PR #34520, we only handle HivePartitionWriter. But this should be applied to all hdfs writer. This PR fix it, unify the logic to make it work with both hive and iceberg writer. If the path is an absolute full path like `hdfs://host/path/to/file`, use `hdfs://host/` as fs name, otherwise, use default fs name.

Thearas · 2025-04-19T23:44:06Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Thearas · 2025-04-19T23:44:09Z

run buildall

doris-robot · 2025-04-20T00:23:21Z

TPC-H: Total hot run time: 39734 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cf91302de5d82edef69714d56921d6590f68b7a1, data reload: false
------ Round 1 ----------------------------------
q1	17579	6710	6577	6577
q2	2058	178	179	178
q3	10607	1092	1215	1092
q4	10568	815	733	733
q5	7762	2893	2688	2688
q6	219	135	130	130
q7	965	611	605	605
q8	9353	1911	2017	1911
q9	6590	6384	6391	6384
q10	6999	2268	2258	2258
q11	477	266	261	261
q12	392	216	211	211
q13	17808	2971	3051	2971
q14	233	208	215	208
q15	508	466	465	465
q16	696	579	592	579
q17	961	541	595	541
q18	7313	6724	6696	6696
q19	1414	1024	1051	1024
q20	478	198	204	198
q21	3954	3261	3045	3045
q22	1111	979	992	979
Total cold run time: 108045 ms
Total hot run time: 39734 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6628	6579	6562	6562
q2	326	238	233	233
q3	2887	2762	2908	2762
q4	2042	1844	1748	1748
q5	5779	5683	5686	5683
q6	208	127	131	127
q7	2212	1786	1832	1786
q8	3364	3562	3518	3518
q9	8926	8614	8845	8614
q10	3571	3517	3522	3517
q11	590	499	481	481
q12	789	587	631	587
q13	8719	3135	3110	3110
q14	313	272	276	272
q15	520	456	475	456
q16	690	660	666	660
q17	1833	1647	1584	1584
q18	8192	7764	7627	7627
q19	1687	1562	1505	1505
q20	2109	1860	1826	1826
q21	5366	5199	5328	5199
q22	1152	1088	1019	1019
Total cold run time: 67903 ms
Total hot run time: 58876 ms

doris-robot · 2025-04-20T00:35:57Z

TPC-DS: Total hot run time: 197238 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cf91302de5d82edef69714d56921d6590f68b7a1, data reload: false
query1	1285	927	938	927
query2	6273	2087	2110	2087
query3	10940	4421	4350	4350
query4	61287	28769	23481	23481
query5	5221	464	460	460
query6	406	171	173	171
query7	5478	315	328	315
query8	309	224	233	224
query9	8403	2630	2611	2611
query10	474	274	263	263
query11	17915	15170	16102	15170
query12	164	108	111	108
query13	1468	474	436	436
query14	10509	6928	6901	6901
query15	205	178	187	178
query16	7068	518	469	469
query17	1157	581	609	581
query18	1838	321	320	320
query19	207	168	166	166
query20	115	111	109	109
query21	208	104	109	104
query22	4632	4510	4579	4510
query23	34816	33771	33908	33771
query24	6234	3004	2985	2985
query25	531	389	395	389
query26	667	169	172	169
query27	2065	357	373	357
query28	4074	2458	2428	2428
query29	689	445	444	444
query30	240	162	158	158
query31	995	852	818	818
query32	69	53	59	53
query33	400	291	283	283
query34	913	536	532	532
query35	858	748	727	727
query36	1065	976	942	942
query37	111	64	71	64
query38	4092	3949	3950	3949
query39	1508	1471	1478	1471
query40	204	102	98	98
query41	50	47	48	47
query42	113	98	99	98
query43	532	510	516	510
query44	1223	820	808	808
query45	183	172	166	166
query46	1171	742	785	742
query47	2044	1929	1916	1916
query48	497	381	391	381
query49	737	452	409	409
query50	865	439	448	439
query51	7443	7215	7262	7215
query52	110	94	94	94
query53	267	201	193	193
query54	602	465	496	465
query55	86	84	89	84
query56	285	280	266	266
query57	1261	1191	1156	1156
query58	227	231	224	224
query59	3359	3165	3208	3165
query60	309	262	291	262
query61	153	142	131	131
query62	798	715	720	715
query63	236	198	200	198
query64	1512	693	638	638
query65	3314	3170	3207	3170
query66	723	298	301	298
query67	15878	15468	15589	15468
query68	4324	578	593	578
query69	438	270	271	270
query70	1135	1100	1071	1071
query71	353	252	260	252
query72	6346	4026	3956	3956
query73	757	353	361	353
query74	10225	8988	9172	8988
query75	3368	2679	2637	2637
query76	2127	1021	1014	1014
query77	520	272	276	272
query78	10587	9696	9552	9552
query79	1962	606	610	606
query80	1347	432	430	430
query81	533	245	237	237
query82	1279	93	87	87
query83	263	151	151	151
query84	285	89	77	77
query85	1029	313	325	313
query86	389	290	267	267
query87	4516	4211	4282	4211
query88	3861	2474	2406	2406
query89	417	294	288	288
query90	1995	188	187	187
query91	181	153	149	149
query92	63	49	50	49
query93	2699	569	559	559
query94	782	302	296	296
query95	352	259	262	259
query96	636	289	287	287
query97	3304	3120	3116	3116
query98	220	211	192	192
query99	1579	1302	1291	1291
Total cold run time: 316943 ms
Total hot run time: 197238 ms

hello-stephen · 2025-04-20T00:36:35Z

BE UT Coverage Report

Increment line coverage 0.00% (0/16) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	40.12% (10512/26199)
Line Coverage	30.76% (88292/287062)
Region Coverage	29.86% (45489/152354)
Branch Coverage	26.21% (23049/87928)

doris-robot · 2025-04-20T00:41:28Z

ClickBench: Total hot run time: 32.9 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cf91302de5d82edef69714d56921d6590f68b7a1, data reload: false
query1	0.03	0.03	0.03
query2	0.07	0.04	0.03
query3	0.23	0.06	0.07
query4	1.61	0.10	0.10
query5	0.54	0.50	0.53
query6	1.13	0.72	0.72
query7	0.02	0.02	0.02
query8	0.05	0.03	0.03
query9	0.54	0.50	0.49
query10	0.55	0.55	0.55
query11	0.14	0.10	0.12
query12	0.14	0.11	0.11
query13	0.60	0.59	0.59
query14	2.73	2.84	2.74
query15	0.89	0.83	0.83
query16	0.39	0.38	0.39
query17	1.06	1.07	1.07
query18	0.24	0.23	0.22
query19	1.92	2.03	1.75
query20	0.01	0.01	0.02
query21	15.38	0.59	0.58
query22	2.73	2.47	1.67
query23	17.08	0.91	0.75
query24	2.78	1.87	1.93
query25	0.22	0.10	0.11
query26	0.48	0.14	0.13
query27	0.05	0.04	0.04
query28	9.17	0.49	0.48
query29	12.59	3.21	3.20
query30	0.25	0.06	0.06
query31	2.88	0.39	0.38
query32	3.24	0.46	0.45
query33	2.97	3.01	3.00
query34	17.04	4.52	4.60
query35	4.60	4.56	4.58
query36	0.66	0.48	0.47
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.03	0.03
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 105.48 s
Total hot run time: 32.9 s

dataroaring

LGTM

github-actionsBot requested a review from dataroaring as a code owner April 19, 2025 23:44

dataroaring closed this Apr 19, 2025

dataroaring reopened this Apr 19, 2025

dataroaring approved these changes Apr 22, 2025
View reviewed changes

dataroaring merged commit 4d575fd into branch-3.0Apr 22, 2025

github-actionsBot deleted the auto-pick-49998-branch-3.0 branch April 22, 2025 03:32

Navigation Menu

Uh oh!

branch-3.0: [fix](hive/iceberg) rectify the fs name if path already contains fs #49998 - #50197

branch-3.0: [fix](hive/iceberg) rectify the fs name if path already contains fs #49998#50197
dataroaring merged 1 commit into
branch-3.0from
auto-pick-49998-branch-3.0

github-actionsBot commented Apr 19, 2025

Uh oh!

Thearas commented Apr 19, 2025

Uh oh!

Thearas commented Apr 19, 2025

Uh oh!

doris-robot commented Apr 20, 2025

Uh oh!

doris-robot commented Apr 20, 2025

Uh oh!

hello-stephen commented Apr 20, 2025

Uh oh!

doris-robot commented Apr 20, 2025

Uh oh!

dataroaring left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

github-actionsBot commented Apr 19, 2025

Uh oh!

Thearas commented Apr 19, 2025

Uh oh!

Thearas commented Apr 19, 2025

Uh oh!

doris-robot commented Apr 20, 2025

Uh oh!

doris-robot commented Apr 20, 2025

Uh oh!

hello-stephen commented Apr 20, 2025

BE UT Coverage Report

Uh oh!

doris-robot commented Apr 20, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants