[feat](thirdparty) Refactor HDFS dependency management: move HDFS client jars to FE and slim down BE thirdparty by CalvinKirs · Pull Request #57202 · apache/doris

CalvinKirs · 2025-10-21T13:08:45Z

Background

Previously, BE depended on the HDFS official distribution through the thirdparty module. This approach directly compiled and packaged Hadoop’s full dependencies, including large and unnecessary modules such as YARN, MapReduce, etc. As a result:

The build artifact contained many duplicate jars.

The total size was unnecessarily large (hundreds of MB).

BE only acted as an HDFS client, not a Hadoop runtime, so most of these dependencies were not needed.

Changes in this PR

Move all HDFS-related jars to FE for unified dependency management, since FE already uses HDFS extensively.

Retain necessary native/C++ dependencies under thirdparty for BE.

Do not build an uber-jar for HDFS (unlike other BE thirdparty libs).

Exclude unused Hadoop modules (e.g., YARN, MR, tools).

The resulting target structure now includes:

target/
├── classes
├── lib/
│ └── (~57M total)
├── hadoop-deps.jar
└── ...

Compared with the previous build, the total artifact size has been reduced by ~150 MB.

Rationale

Since BE only functions as a client to access HDFS, managing Java-side HDFS dependencies in FE is cleaner and more maintainable. If BE needs custom logic (e.g., optimized HDFS client), we can extend or override the relevant classes and control class loading order.

Impact

Build size reduced by ~150 MB

Simpler dependency graph

Notes

Native/C++ HDFS dependencies remain managed under thirdparty.

FE now provides necessary jars for BE at runtime.

No need to modify existing C++ build logic.

…ent jars to FE and slim down BE thirdparty ### Background Previously, BE depended on the HDFS official distribution through the thirdparty module. This approach directly compiled and packaged Hadoop’s full dependencies, including large and unnecessary modules such as YARN, MapReduce, etc. As a result: The build artifact contained many duplicate jars. The total size was unnecessarily large (hundreds of MB). BE only acted as an HDFS client, not a Hadoop runtime, so most of these dependencies were not needed. ### Changes in this PR Move all HDFS-related jars to FE for unified dependency management, since FE already uses HDFS extensively. Retain necessary native/C++ dependencies under thirdparty for BE. Do not build an uber-jar for HDFS (unlike other BE thirdparty libs). Exclude unused Hadoop modules (e.g., YARN, MR, tools). The resulting target structure now includes: ``` target/ ├── classes ├── lib/ │ └── (~57M total) ├── hadoop-deps.jar └── ... ``` Compared with the previous build, the total artifact size has been reduced by ~150 MB. ### Rationale Since BE only functions as a client to access HDFS, managing Java-side HDFS dependencies in FE is cleaner and more maintainable. If BE needs custom logic (e.g., optimized HDFS client), we can extend or override the relevant classes and control class loading order. Impact Build size reduced by ~150 MB Simpler dependency graph ### Notes Native/C++ HDFS dependencies remain managed under thirdparty. FE now provides necessary jars for BE at runtime. No need to modify existing C++ build logic.

hello-stephen · 2025-10-21T13:08:50Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

CalvinKirs · 2025-10-21T13:13:04Z

run buildall

doris-robot · 2025-10-21T14:04:23Z

ClickBench: Total hot run time: 30.7 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit de1e6c293ece514062b07a598feb058c658e21f9, data reload: false
query1	0.07	0.06	0.06
query2	0.10	0.06	0.06
query3	0.26	0.09	0.10
query4	1.61	0.12	0.13
query5	0.29	0.28	0.29
query6	1.20	0.69	0.69
query7	0.04	0.04	0.03
query8	0.06	0.05	0.05
query9	0.67	0.57	0.56
query10	0.63	0.64	0.63
query11	0.19	0.13	0.13
query12	0.18	0.14	0.13
query13	0.66	0.63	0.62
query14	1.07	1.06	1.06
query15	0.92	0.90	0.95
query16	0.46	0.44	0.46
query17	1.20	1.28	1.15
query18	0.24	0.22	0.24
query19	2.07	1.92	1.95
query20	0.02	0.02	0.01
query21	15.36	0.20	0.15
query22	5.07	0.08	0.05
query23	15.60	0.29	0.11
query24	2.79	0.85	0.86
query25	0.09	0.07	0.06
query26	0.18	0.17	0.16
query27	0.07	0.07	0.06
query28	4.01	1.20	0.98
query29	12.62	4.73	3.88
query30	0.29	0.14	0.13
query31	2.85	0.67	0.43
query32	3.25	0.60	0.54
query33	3.23	3.20	3.25
query34	16.14	5.59	4.99
query35	5.01	5.02	5.06
query36	0.71	0.55	0.53
query37	0.12	0.08	0.08
query38	0.08	0.05	0.05
query39	0.05	0.04	0.03
query40	0.19	0.17	0.15
query41	0.10	0.04	0.03
query42	0.05	0.04	0.03
query43	0.05	0.04	0.04
Total cold run time: 99.85 s
Total hot run time: 30.7 s

CalvinKirs · 2025-10-24T06:35:53Z

run compile

CalvinKirs · 2025-10-24T08:36:17Z

run compile

CalvinKirs · 2025-10-24T09:42:23Z

run compile

CalvinKirs · 2025-10-24T10:38:22Z

run compile

CalvinKirs · 2025-10-24T12:26:58Z

run p0
run external

[feat](thirdparty) Refactor HDFS dependency management: move HDFS client jars to FE and slim down BE thirdparty ### Background Previously, BE depended on the HDFS official distribution through the thirdparty module. This approach directly compiled and packaged Hadoop’s full dependencies, including large and unnecessary modules such as YARN, MapReduce, etc. As a result: The build artifact contained many duplicate jars. The total size was unnecessarily large (hundreds of MB). BE only acted as an HDFS client, not a Hadoop runtime, so most of these dependencies were not needed. ### Changes in this PR Move all HDFS-related jars to FE for unified dependency management, since FE already uses HDFS extensively. Retain necessary native/C++ dependencies under thirdparty for BE. Do not build an uber-jar for HDFS (unlike other BE thirdparty libs). Exclude unused Hadoop modules (e.g., YARN, MR, tools). The resulting target structure now includes: ``` target/ ├── classes ├── lib/ │ └── (~57M total) ├── hadoop-deps.jar └── ... ``` Compared with the previous build, the total artifact size has been reduced by ~150 MB. ### Rationale Since BE only functions as a client to access HDFS, managing Java-side HDFS dependencies in FE is cleaner and more maintainable. If BE needs custom logic (e.g., optimized HDFS client), we can extend or override the relevant classes and control class loading order. Impact Build size reduced by ~150 MB Simpler dependency graph ### Notes Native/C++ HDFS dependencies remain managed under thirdparty. FE now provides necessary jars for BE at runtime. No need to modify existing C++ build logic.

CalvinKirs · 2025-10-24T12:33:19Z

run buildall

doris-robot · 2025-10-24T13:23:11Z

ClickBench: Total hot run time: 29.86 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0d9a629c7b2a3d437f77e2c5a3626d7e1a4bb2a3, data reload: false
query1	0.07	0.06	0.05
query2	0.11	0.06	0.06
query3	0.27	0.09	0.09
query4	1.61	0.13	0.13
query5	0.31	0.29	0.28
query6	1.19	0.70	0.68
query7	0.04	0.03	0.04
query8	0.07	0.06	0.05
query9	0.67	0.58	0.56
query10	0.62	0.62	0.63
query11	0.19	0.13	0.13
query12	0.18	0.14	0.14
query13	0.64	0.62	0.61
query14	1.07	1.03	1.04
query15	0.91	0.90	0.90
query16	0.42	0.42	0.44
query17	1.13	1.22	1.09
query18	0.24	0.22	0.22
query19	2.06	1.89	1.92
query20	0.02	0.01	0.01
query21	15.40	0.23	0.16
query22	5.04	0.08	0.06
query23	15.59	0.31	0.12
query24	3.36	0.98	1.43
query25	0.10	0.06	0.06
query26	0.17	0.18	0.17
query27	0.07	0.06	0.06
query28	5.63	1.23	0.97
query29	12.64	4.61	3.85
query30	0.31	0.15	0.13
query31	2.85	0.68	0.43
query32	3.24	0.59	0.49
query33	3.09	3.19	3.25
query34	15.76	5.26	4.66
query35	4.67	4.66	4.61
query36	0.73	0.53	0.54
query37	0.11	0.08	0.07
query38	0.08	0.05	0.04
query39	0.04	0.03	0.04
query40	0.19	0.16	0.15
query41	0.10	0.04	0.04
query42	0.05	0.04	0.03
query43	0.05	0.04	0.04
Total cold run time: 101.09 s
Total hot run time: 29.86 s

CalvinKirs · 2025-10-27T08:10:06Z

run buildall

doris-robot · 2025-10-27T08:51:16Z

TPC-DS: Total hot run time: 189151 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1f63c0c51665a1c0c71a3fce78fb9fc69747b0ef, data reload: false
query1	1081	412	398	398
query2	6556	1689	1687	1687
query3	6763	224	221	221
query4	26726	24148	23006	23006
query5	5912	656	476	476
query6	380	252	253	252
query7	4645	502	297	297
query8	307	257	252	252
query9	8719	2563	2572	2563
query10	549	342	282	282
query11	15797	15102	14892	14892
query12	180	122	115	115
query13	1688	549	439	439
query14	11635	9232	9143	9143
query15	216	190	185	185
query16	7674	698	506	506
query17	1612	792	641	641
query18	2115	486	355	355
query19	236	223	192	192
query20	139	154	131	131
query21	224	139	123	123
query22	4606	4649	4545	4545
query23	35079	33825	33954	33825
query24	9096	2504	2520	2504
query25	610	542	468	468
query26	1290	283	176	176
query27	3183	529	373	373
query28	4669	2248	2198	2198
query29	842	660	514	514
query30	311	243	203	203
query31	904	832	783	783
query32	86	92	73	73
query33	702	410	344	344
query34	800	846	516	516
query35	809	835	726	726
query36	942	984	905	905
query37	140	106	85	85
query38	3496	3521	3484	3484
query39	1489	1413	1447	1413
query40	224	124	117	117
query41	67	65	67	65
query42	122	112	119	112
query43	495	487	476	476
query44	1213	738	734	734
query45	184	178	174	174
query46	882	990	626	626
query47	1769	1802	1759	1759
query48	412	435	313	313
query49	778	507	425	425
query50	654	689	400	400
query51	3936	3957	3852	3852
query52	110	105	99	99
query53	237	266	201	201
query54	601	580	550	550
query55	85	88	86	86
query56	325	314	309	309
query57	1186	1206	1125	1125
query58	293	281	278	278
query59	2533	2572	2559	2559
query60	361	361	361	361
query61	169	156	152	152
query62	813	725	693	693
query63	234	195	188	188
query64	4428	1152	848	848
query65	4039	3925	3958	3925
query66	1022	429	335	335
query67	15447	15037	14975	14975
query68	9624	887	604	604
query69	517	319	289	289
query70	1338	1217	1302	1217
query71	527	350	320	320
query72	5506	4777	4769	4769
query73	723	562	356	356
query74	8899	9308	8715	8715
query75	4592	3340	2874	2874
query76	4021	1149	727	727
query77	1047	394	315	315
query78	9626	9906	8900	8900
query79	1103	821	595	595
query80	624	589	495	495
query81	486	266	232	232
query82	265	156	135	135
query83	273	272	248	248
query84	257	115	93	93
query85	855	466	423	423
query86	333	302	294	294
query87	3688	3797	3622	3622
query88	2859	2274	2245	2245
query89	378	325	304	304
query90	2021	218	216	216
query91	162	167	139	139
query92	83	75	68	68
query93	1105	971	642	642
query94	695	447	338	338
query95	414	330	317	317
query96	500	574	283	283
query97	2978	2988	2864	2864
query98	235	212	211	211
query99	1448	1418	1273	1273
Total cold run time: 282009 ms
Total hot run time: 189151 ms

doris-robot · 2025-10-27T08:56:25Z

ClickBench: Total hot run time: 28.25 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1f63c0c51665a1c0c71a3fce78fb9fc69747b0ef, data reload: false
query1	0.06	0.05	0.05
query2	0.09	0.05	0.05
query3	0.26	0.08	0.08
query4	1.61	0.11	0.12
query5	0.28	0.27	0.26
query6	1.18	0.66	0.64
query7	0.03	0.03	0.04
query8	0.06	0.05	0.06
query9	0.61	0.51	0.52
query10	0.58	0.58	0.57
query11	0.17	0.12	0.11
query12	0.17	0.12	0.13
query13	0.63	0.60	0.60
query14	1.01	1.01	1.01
query15	0.86	0.85	0.85
query16	0.41	0.39	0.40
query17	1.05	1.03	1.01
query18	0.22	0.20	0.20
query19	1.92	1.84	1.85
query20	0.01	0.02	0.01
query21	15.43	0.20	0.13
query22	5.10	0.07	0.04
query23	15.63	0.27	0.10
query24	2.86	1.25	0.85
query25	0.08	0.07	0.06
query26	0.14	0.13	0.14
query27	0.07	0.06	0.05
query28	4.72	1.16	0.94
query29	12.55	4.02	3.37
query30	0.29	0.14	0.12
query31	2.82	0.60	0.39
query32	3.22	0.55	0.49
query33	3.07	3.09	3.08
query34	15.96	5.20	4.54
query35	4.59	4.62	4.59
query36	0.67	0.50	0.50
query37	0.10	0.07	0.07
query38	0.06	0.05	0.04
query39	0.04	0.03	0.03
query40	0.19	0.14	0.16
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 98.97 s
Total hot run time: 28.25 s

hello-stephen · 2025-10-27T09:15:29Z

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2025-10-27T09:38:17Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.64% (17966/34129)
Line Coverage	37.91% (163024/429992)
Region Coverage	32.31% (124307/384679)
Branch Coverage	33.69% (54360/161346)

hello-stephen · 2025-10-27T10:57:25Z

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.39% (23921/33507)
Line Coverage	57.77% (248611/430334)
Region Coverage	52.89% (206227/389943)
Branch Coverage	54.57% (88595/162339)

CalvinKirs · 2025-10-28T02:17:20Z

run buildall

hello-stephen · 2025-10-28T02:48:05Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	80.64% (1649/2045)
Line Coverage	67.00% (29103/43437)
Region Coverage	67.36% (14428/21418)
Branch Coverage	57.73% (7673/13292)

hello-stephen · 2025-10-28T03:29:21Z

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

doris-robot · 2025-10-28T03:39:17Z

TPC-DS: Total hot run time: 190878 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 841b6cf07c117dd03471f939cec8ed24265ffa38, data reload: false
query1	1083	443	409	409
query2	6577	1684	1702	1684
query3	6767	222	223	222
query4	26445	23525	23072	23072
query5	4414	640	498	498
query6	347	274	253	253
query7	4651	498	307	307
query8	315	291	278	278
query9	8696	2574	2596	2574
query10	495	349	282	282
query11	15720	15686	15580	15580
query12	199	126	119	119
query13	2215	588	473	473
query14	12431	9472	9696	9472
query15	266	205	189	189
query16	8571	702	572	572
query17	2109	1043	634	634
query18	2429	446	379	379
query19	301	209	180	180
query20	149	134	145	134
query21	233	131	122	122
query22	4380	4689	4704	4689
query23	34531	33893	33403	33403
query24	8195	2431	2453	2431
query25	608	533	486	486
query26	1232	273	160	160
query27	2690	509	350	350
query28	4299	2235	2177	2177
query29	810	617	477	477
query30	300	220	196	196
query31	914	822	738	738
query32	85	74	71	71
query33	584	397	315	315
query34	792	859	506	506
query35	782	841	744	744
query36	952	975	915	915
query37	142	115	83	83
query38	3523	3584	3480	3480
query39	1483	1439	1460	1439
query40	220	126	114	114
query41	61	57	58	57
query42	125	119	111	111
query43	496	521	470	470
query44	1233	749	741	741
query45	195	186	172	172
query46	879	991	630	630
query47	1774	1793	1714	1714
query48	405	437	333	333
query49	781	508	438	438
query50	651	700	405	405
query51	3936	4108	3942	3942
query52	114	107	103	103
query53	239	257	204	204
query54	609	616	533	533
query55	90	91	83	83
query56	349	335	317	317
query57	1201	1202	1123	1123
query58	280	274	287	274
query59	2608	2587	2655	2587
query60	356	338	335	335
query61	189	165	161	161
query62	796	759	664	664
query63	230	196	198	196
query64	4493	1164	878	878
query65	4014	3969	3933	3933
query66	1085	442	333	333
query67	15280	15256	15050	15050
query68	8307	901	609	609
query69	499	332	297	297
query70	1336	1308	1271	1271
query71	515	341	316	316
query72	5946	4936	5084	4936
query73	702	603	358	358
query74	9030	9246	8969	8969
query75	4098	3346	2872	2872
query76	3728	1150	730	730
query77	820	414	325	325
query78	9593	9652	8974	8974
query79	3073	796	588	588
query80	718	565	524	524
query81	494	269	234	234
query82	479	162	137	137
query83	306	272	257	257
query84	302	113	102	102
query85	921	501	442	442
query86	343	304	286	286
query87	3747	3730	3691	3691
query88	3415	2252	2231	2231
query89	424	331	295	295
query90	2045	216	234	216
query91	187	183	136	136
query92	90	68	62	62
query93	2586	973	636	636
query94	705	444	340	340
query95	416	319	325	319
query96	488	595	280	280
query97	2937	2968	2888	2888
query98	243	222	213	213
query99	1452	1421	1350	1350
Total cold run time: 283349 ms
Total hot run time: 190878 ms

CalvinKirs · 2025-10-29T06:57:35Z

run buildall

doris-robot · 2025-10-29T07:28:10Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	80.66% (1656/2053)
Line Coverage	67.00% (29226/43619)
Region Coverage	67.37% (14498/21519)
Branch Coverage	57.71% (7711/13362)

github-actions · 2025-10-29T07:37:49Z

PR approved by at least one committer and no changes requested.

hello-stephen · 2025-10-29T09:50:28Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.75% (18064/34243)
Line Coverage	37.98% (163797/431224)
Region Coverage	32.36% (124874/385846)
Branch Coverage	33.72% (54606/161927)

hello-stephen · 2025-10-29T11:04:46Z

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.35% (23938/33548)
Line Coverage	57.79% (249080/431036)
Region Coverage	52.91% (206681/390610)
Branch Coverage	54.62% (88872/162712)

morningman · 2025-10-30T01:43:05Z

run performance

CalvinKirs · 2025-10-30T02:35:05Z

run performance

doris-robot · 2025-10-30T03:23:26Z

ClickBench: Total hot run time: 28.32 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f23d78d5ebbe57e753bd2bf9c0a92c86dd4a9605, data reload: false
query1	0.06	0.06	0.06
query2	0.10	0.06	0.05
query3	0.26	0.09	0.08
query4	1.60	0.12	0.12
query5	0.29	0.28	0.25
query6	1.18	0.66	0.66
query7	0.03	0.02	0.03
query8	0.05	0.05	0.05
query9	0.64	0.55	0.53
query10	0.61	0.61	0.58
query11	0.18	0.12	0.12
query12	0.16	0.12	0.12
query13	0.62	0.61	0.60
query14	1.04	1.03	1.02
query15	0.86	0.86	0.87
query16	0.42	0.39	0.41
query17	1.06	1.09	1.04
query18	0.23	0.21	0.21
query19	1.96	1.86	1.88
query20	0.02	0.01	0.02
query21	15.42	0.21	0.14
query22	4.98	0.07	0.05
query23	15.63	0.28	0.11
query24	2.71	0.52	1.01
query25	0.09	0.08	0.07
query26	0.15	0.15	0.14
query27	0.07	0.06	0.05
query28	5.69	1.23	0.94
query29	12.57	4.18	3.45
query30	0.28	0.15	0.12
query31	2.84	0.59	0.40
query32	3.25	0.56	0.49
query33	3.16	3.18	3.11
query34	15.86	5.23	4.57
query35	4.58	4.66	4.65
query36	0.68	0.51	0.50
query37	0.11	0.07	0.06
query38	0.07	0.05	0.04
query39	0.04	0.03	0.04
query40	0.18	0.14	0.14
query41	0.09	0.04	0.04
query42	0.05	0.04	0.03
query43	0.06	0.04	0.04
Total cold run time: 99.93 s
Total hot run time: 28.32 s

…ent jars to FE and slim down BE thirdparty (#57202) ### Background Previously, BE depended on the HDFS official distribution through the thirdparty module. This approach directly compiled and packaged Hadoop’s full dependencies, including large and unnecessary modules such as YARN, MapReduce, etc. As a result: The build artifact contained many duplicate jars. The total size was unnecessarily large (hundreds of MB). BE only acted as an HDFS client, not a Hadoop runtime, so most of these dependencies were not needed. ### Changes in this PR Move all HDFS-related jars to FE for unified dependency management, since FE already uses HDFS extensively. Retain necessary native/C++ dependencies under thirdparty for BE. Do not build an uber-jar for HDFS (unlike other BE thirdparty libs). Exclude unused Hadoop modules (e.g., YARN, MR, tools). The resulting target structure now includes: ``` target/ ├── classes ├── lib/ │ └── (~57M total) ├── hadoop-deps.jar └── ... ``` Compared with the previous build, the total artifact size has been reduced by ~150 MB. ### Rationale Since BE only functions as a client to access HDFS, managing Java-side HDFS dependencies in FE is cleaner and more maintainable. If BE needs custom logic (e.g., optimized HDFS client), we can extend or override the relevant classes and control class loading order. Impact Build size reduced by ~150 MB Simpler dependency graph ### Notes Native/C++ HDFS dependencies remain managed under thirdparty. FE now provides necessary jars for BE at runtime. No need to modify existing C++ build logic.

…ve HDFS client jars to FE and slim down BE thirdparty (apache#57202) apache#57202 (cherry picked from commit 4ea473c)

…ent jars to FE and slim down BE thirdparty (apache#57202) ### Background Previously, BE depended on the HDFS official distribution through the thirdparty module. This approach directly compiled and packaged Hadoop’s full dependencies, including large and unnecessary modules such as YARN, MapReduce, etc. As a result: The build artifact contained many duplicate jars. The total size was unnecessarily large (hundreds of MB). BE only acted as an HDFS client, not a Hadoop runtime, so most of these dependencies were not needed. ### Changes in this PR Move all HDFS-related jars to FE for unified dependency management, since FE already uses HDFS extensively. Retain necessary native/C++ dependencies under thirdparty for BE. Do not build an uber-jar for HDFS (unlike other BE thirdparty libs). Exclude unused Hadoop modules (e.g., YARN, MR, tools). The resulting target structure now includes: ``` target/ ├── classes ├── lib/ │ └── (~57M total) ├── hadoop-deps.jar └── ... ``` Compared with the previous build, the total artifact size has been reduced by ~150 MB. ### Rationale Since BE only functions as a client to access HDFS, managing Java-side HDFS dependencies in FE is cleaner and more maintainable. If BE needs custom logic (e.g., optimized HDFS client), we can extend or override the relevant classes and control class loading order. Impact Build size reduced by ~150 MB Simpler dependency graph ### Notes Native/C++ HDFS dependencies remain managed under thirdparty. FE now provides necessary jars for BE at runtime. No need to modify existing C++ build logic.

…ve HDFS client jars to FE and slim down BE thirdparty (#57202) (#57527) #57202

…ent jars to FE and slim down BE thirdparty (#57202) ### Background Previously, BE depended on the HDFS official distribution through the thirdparty module. This approach directly compiled and packaged Hadoop’s full dependencies, including large and unnecessary modules such as YARN, MapReduce, etc. As a result: The build artifact contained many duplicate jars. The total size was unnecessarily large (hundreds of MB). BE only acted as an HDFS client, not a Hadoop runtime, so most of these dependencies were not needed. ### Changes in this PR Move all HDFS-related jars to FE for unified dependency management, since FE already uses HDFS extensively. Retain necessary native/C++ dependencies under thirdparty for BE. Do not build an uber-jar for HDFS (unlike other BE thirdparty libs). Exclude unused Hadoop modules (e.g., YARN, MR, tools). The resulting target structure now includes: ``` target/ ├── classes ├── lib/ │ └── (~57M total) ├── hadoop-deps.jar └── ... ``` Compared with the previous build, the total artifact size has been reduced by ~150 MB. ### Rationale Since BE only functions as a client to access HDFS, managing Java-side HDFS dependencies in FE is cleaner and more maintainable. If BE needs custom logic (e.g., optimized HDFS client), we can extend or override the relevant classes and control class loading order. Impact Build size reduced by ~150 MB Simpler dependency graph ### Notes Native/C++ HDFS dependencies remain managed under thirdparty. FE now provides necessary jars for BE at runtime. No need to modify existing C++ build logic. (cherry picked from commit 4ea473c)

…57782) Cherry-picked from #57710 #57979 #57810 #57784 #57202 #57629 --------- Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>

CalvinKirs requested a review from morningman as a code owner October 21, 2025 13:08

CalvinKirs force-pushed the master-hadoop-dependencies branch from 8244fb9 to 65c151fCompare October 24, 2025 12:31

Merge branch 'master' into master-hadoop-dependencies
0d9a629

test
1f63c0c

test
841b6cf

CalvinKirs requested review from dataroaring, gavinchou and w41ter as code owners October 28, 2025 02:16

fix
f23d78d

CalvinKirs force-pushed the master-hadoop-dependencies branch from 4cd1e81 to f23d78dCompare October 29, 2025 06:51

gavinchou approved these changes Oct 29, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Oct 29, 2025

morningman approved these changes Oct 30, 2025
View reviewed changes

w41ter approved these changes Oct 30, 2025
View reviewed changes

CalvinKirs merged commit 4ea473c into apache:masterOct 30, 2025
29 of 32 checks passed

github-actionsBot added the dev/3.1.x-conflict label Oct 30, 2025

github-actionsBot mentioned this pull request Oct 30, 2025
branch-4.0: [feat](thirdparty) Refactor HDFS dependency management: move HDFS client jars to FE and slim down BE thirdparty #57202 #57483
Closed

CalvinKirs mentioned this pull request Oct 30, 2025
branch-3.1: [feat](thirdparty) Refactor HDFS dependency management: move HDFS client jars to FE and slim down BE thirdparty #57202 #57527
Merged

CalvinKirs removed the dev/3.1.x-conflict label Oct 30, 2025

morrySnow pushed a commit that referenced this pull request Nov 3, 2025
branch-3.1:[feat](thirdparty) Refactor HDFS dependency management: mo…
38ce7ce
…ve HDFS client jars to FE and slim down BE thirdparty (#57202) (#57527) #57202

morrySnow added dev/3.1.3-merged and removed dev/3.1.x labels Nov 3, 2025

CalvinKirs mentioned this pull request Nov 19, 2025
branch-4.0: [feat](web)Adapted for Spring Boot 3 and Jetty 12 #57710 #57782
Merged

yiguolei pushed a commit that referenced this pull request Nov 20, 2025
branch-4.0: [feat](web)Adapted for Spring Boot 3 and Jetty 12#57710(#…
ff742c4
…57782) Cherry-picked from #57710 #57979 #57810 #57784 #57202 #57629 --------- Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>

yiguolei added dev/4.0.2-merged and removed dev/4.0.x labels Nov 20, 2025

Uh oh!

Conversation

CalvinKirs commented Oct 21, 2025

Background

Changes in this PR

Rationale

Notes

Uh oh!

hello-stephen commented Oct 21, 2025

Uh oh!

CalvinKirs commented Oct 21, 2025

Uh oh!

doris-robot commented Oct 21, 2025

Uh oh!

CalvinKirs commented Oct 24, 2025

Uh oh!

CalvinKirs commented Oct 24, 2025

Uh oh!

CalvinKirs commented Oct 24, 2025

Uh oh!

CalvinKirs commented Oct 24, 2025

Uh oh!

CalvinKirs commented Oct 24, 2025

Uh oh!

CalvinKirs commented Oct 24, 2025

Uh oh!

doris-robot commented Oct 24, 2025

Uh oh!

CalvinKirs commented Oct 27, 2025

Uh oh!

doris-robot commented Oct 27, 2025

Uh oh!

doris-robot commented Oct 27, 2025

Uh oh!

hello-stephen commented Oct 27, 2025

FE UT Coverage Report

Uh oh!

hello-stephen commented Oct 27, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Oct 27, 2025

BE Regression && UT Coverage Report

Uh oh!

CalvinKirs commented Oct 28, 2025

Uh oh!

hello-stephen commented Oct 28, 2025

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Oct 28, 2025

FE UT Coverage Report

Uh oh!

doris-robot commented Oct 28, 2025

Uh oh!

CalvinKirs commented Oct 29, 2025

Uh oh!

doris-robot commented Oct 29, 2025

Cloud UT Coverage Report

Uh oh!

github-actionsBot commented Oct 29, 2025

Uh oh!

hello-stephen commented Oct 29, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Oct 29, 2025

BE Regression && UT Coverage Report

Uh oh!

morningman commented Oct 30, 2025

Uh oh!

CalvinKirs commented Oct 30, 2025

Uh oh!

doris-robot commented Oct 30, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!