Skip to content

[feature](Resource) Support to specify the root path for hdfs resource - #32632

Merged
dataroaring merged 3 commits into
apache:masterfrom
ByteYue:specify_hdfs_root_path
Mar 25, 2024
Merged

[feature](Resource) Support to specify the root path for hdfs resource#32632
dataroaring merged 3 commits into
apache:masterfrom
ByteYue:specify_hdfs_root_path

Conversation

@ByteYue

@ByteYueByteYue commented Mar 21, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

Proposed changes

Previously the root_path of hdfs resource doesn't take effect, since the hdfs file system created had ignored the passed root_path property. This pr tries to correct the behavior by using the user specified root path.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot

doris-robot commented Mar 21, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions

github-actionsBot commented Mar 21, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions

github-actionsBot commented Mar 21, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

dataroaring
dataroaring previously approved these changes Mar 21, 2024
View reviewed changes

@dataroaringdataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ByteYue
ByteYueforce-pushed the specify_hdfs_root_path branch from e71082a to b9fd180CompareMarch 22, 2024 02:22
@github-actions

github-actionsBot commented Mar 22, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions

github-actionsBot commented Mar 22, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@dataroaring

dataroaring commented Mar 22, 2024

Copy link
Copy Markdown
Contributor

run buildall

@doris-robot

doris-robot commented Mar 22, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 37901 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c00225bef8cb45d35daa16e817ae589b4d099fc6, data reload: false
------ Round 1 ----------------------------------
q1	17882	4410	4190	4190
q2	2537	162	161	161
q3	11484	1171	1226	1171
q4	10651	781	803	781
q5	8562	3032	2976	2976
q6	202	125	128	125
q7	1062	576	560	560
q8	9359	2055	2010	2010
q9	7027	6520	6514	6514
q10	8417	3406	3501	3406
q11	442	230	217	217
q12	370	212	198	198
q13	17798	2867	2884	2867
q14	231	201	213	201
q15	513	465	447	447
q16	481	373	376	373
q17	978	543	572	543
q18	7406	6503	6415	6415
q19	4835	1411	1445	1411
q20	546	261	246	246
q21	3536	2874	2809	2809
q22	351	280	296	280
Total cold run time: 114670 ms
Total hot run time: 37901 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4146	4091	4092	4091
q2	339	232	227	227
q3	2984	2886	2835	2835
q4	1881	1576	1555	1555
q5	5230	5274	5244	5244
q6	194	115	117	115
q7	2240	1873	1874	1873
q8	3174	3310	3310	3310
q9	8635	8586	8597	8586
q10	3670	3717	3700	3700
q11	543	455	438	438
q12	733	535	540	535
q13	16920	2882	2845	2845
q14	277	251	259	251
q15	492	457	473	457
q16	478	421	421	421
q17	1747	1506	1478	1478
q18	7443	7283	7070	7070
q19	1624	1526	1525	1525
q20	1940	1739	1716	1716
q21	4811	4767	4654	4654
q22	525	444	434	434
Total cold run time: 70026 ms
Total hot run time: 53360 ms

@doris-robot

doris-robot commented Mar 22, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 181503 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c00225bef8cb45d35daa16e817ae589b4d099fc6, data reload: false
query1	930	364	348	348
query2	7461	2059	1949	1949
query3	6720	221	216	216
query4	31284	20768	20772	20768
query5	4419	426	406	406
query6	276	184	169	169
query7	4629	307	295	295
query8	245	180	176	176
query9	9256	2366	2402	2366
query10	571	252	260	252
query11	15497	14245	14230	14230
query12	143	88	91	88
query13	1623	416	434	416
query14	12389	10366	11180	10366
query15	357	199	197	197
query16	8145	270	265	265
query17	2093	601	552	552
query18	2096	307	281	281
query19	368	154	155	154
query20	98	83	90	83
query21	201	126	130	126
query22	4602	4423	4389	4389
query23	32032	31086	31203	31086
query24	10697	2835	2852	2835
query25	639	381	383	381
query26	1381	161	161	161
query27	2986	354	359	354
query28	7543	1947	1924	1924
query29	898	662	620	620
query30	298	152	150	150
query31	978	754	743	743
query32	94	57	54	54
query33	770	343	255	255
query34	963	478	487	478
query35	836	615	619	615
query36	993	861	904	861
query37	129	79	76	76
query38	3531	3488	3458	3458
query39	1421	1399	1359	1359
query40	208	114	113	113
query41	52	50	50	50
query42	110	93	101	93
query43	503	461	453	453
query44	1234	717	730	717
query45	283	264	264	264
query46	1099	722	695	695
query47	1639	1582	1594	1582
query48	475	359	370	359
query49	1117	357	346	346
query50	785	387	378	378
query51	6680	6668	6629	6629
query52	105	97	91	91
query53	348	275	286	275
query54	310	254	253	253
query55	84	80	79	79
query56	249	240	238	238
query57	1090	1006	1008	1006
query58	241	214	218	214
query59	2805	2644	2551	2551
query60	281	259	267	259
query61	118	116	110	110
query62	596	400	394	394
query63	301	279	280	279
query64	5761	3902	3756	3756
query65	3058	3016	3018	3016
query66	1001	341	336	336
query67	14782	14372	14334	14334
query68	9199	539	547	539
query69	676	377	382	377
query70	1433	1147	1161	1147
query71	550	290	266	266
query72	6814	2639	2521	2521
query73	1606	321	318	318
query74	6991	6574	6511	6511
query75	5227	2828	2811	2811
query76	5832	934	804	804
query77	869	276	263	263
query78	10368	9739	9636	9636
query79	10799	530	535	530
query80	1119	400	395	395
query81	522	211	213	211
query82	350	201	202	201
query83	206	148	160	148
query84	290	78	79	78
query85	1126	343	313	313
query86	369	285	263	263
query87	3781	3581	3622	3581
query88	5206	2364	2375	2364
query89	490	383	378	378
query90	2139	180	177	177
query91	172	136	141	136
query92	59	51	48	48
query93	6358	520	496	496
query94	1399	184	179	179
query95	431	341	344	341
query96	594	275	270	270
query97	3021	2899	2871	2871
query98	226	220	213	213
query99	990	745	749	745
Total cold run time: 314119 ms
Total hot run time: 181503 ms

@doris-robot

doris-robot commented Mar 22, 2024

Copy link
Copy Markdown

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c00225bef8cb45d35daa16e817ae589b4d099fc6 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 19.4 seconds inserted 10000000 Rows, about 515K ops/s

@doris-robot

doris-robot commented Mar 22, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 35.26% (8733/24764)
Line Coverage: 27.06% (71513/264273)
Region Coverage: 26.30% (37103/141090)
Branch Coverage: 23.19% (18975/81814)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c00225bef8cb45d35daa16e817ae589b4d099fc6_c00225bef8cb45d35daa16e817ae589b4d099fc6/report/index.html

@github-actions

github-actionsBot commented Mar 25, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Mar 25, 2024
View reviewed changes

@dataroaringdataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring
dataroaring merged commit fa345dd into apache:masterMar 25, 2024
@ByteYueByteYue mentioned this pull request Jun 4, 2024
Merged
dataroaring pushed a commit that referenced this pull request Jun 5, 2024
@ByteYue
Pick "[feature](Resource) Support to specify the root path for hdfs r…
630fd06
…esource #32632" (#35848)
same as #32632
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@ByteYue@HappenLee
[feature](Resource) Support to specify the root path for hdfs resource (
3bebb8a
apache#32632)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchougavinchougavinchou approved these changes
@dataroaringdataroaringdataroaring approved these changes

Assignees

No one assigned

Labels

dev/2.1.4-mergedkind/behavior-changedmeta-changereviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@ByteYue@doris-robot@dataroaring@gavinchou@morningman