[opt](packed-file) Avoid unnecessary HEAD Object requests when opening an S3 object by liaoxin01 · Pull Request #59414 · apache/doris

liaoxin01 · 2025-12-26T13:41:00Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-12-26T13:41:05Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

liaoxin01 · 2025-12-26T13:41:14Z

run buildall

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

doris-robot · 2025-12-26T14:07:29Z

TPC-H: Total hot run time: 35298 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3b3d9b885b2d957456d8172f3c8ad8aa3719267c, data reload: false
------ Round 1 ----------------------------------
q1	16908	4415	4146	4146
q2	1985	336	234	234
q3	9879	1312	736	736
q4	10136	914	325	325
q5	7451	2115	1956	1956
q6	188	166	132	132
q7	997	831	717	717
q8	9357	1436	1182	1182
q9	6981	5370	5321	5321
q10	6791	2403	2006	2006
q11	531	347	317	317
q12	637	734	557	557
q13	17780	3659	3110	3110
q14	305	291	279	279
q15	581	513	525	513
q16	708	681	630	630
q17	697	815	512	512
q18	7582	7216	7033	7033
q19	1226	983	620	620
q20	409	373	258	258
q21	4255	3984	3752	3752
q22	1015	1035	962	962
Total cold run time: 106399 ms
Total hot run time: 35298 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4117	4064	4099	4064
q2	320	392	311	311
q3	2158	2727	2280	2280
q4	1398	1770	1316	1316
q5	4273	4530	4726	4530
q6	231	180	138	138
q7	2037	1967	1820	1820
q8	2649	2539	2414	2414
q9	7683	7529	7512	7512
q10	3161	3208	2832	2832
q11	629	528	530	528
q12	685	935	629	629
q13	3515	4012	3409	3409
q14	301	285	268	268
q15	554	519	522	519
q16	666	710	639	639
q17	1230	1378	1406	1378
q18	8038	7633	7353	7353
q19	845	822	834	822
q20	1874	1932	1818	1818
q21	4615	4294	4141	4141
q22	1056	1012	995	995
Total cold run time: 52035 ms
Total hot run time: 49716 ms

doris-robot · 2025-12-26T14:18:40Z

TPC-DS: Total hot run time: 180633 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3b3d9b885b2d957456d8172f3c8ad8aa3719267c, data reload: false
query5	5038	627	453	453
query6	334	228	210	210
query7	4213	462	278	278
query8	308	253	247	247
query9	8793	2551	2518	2518
query10	535	367	328	328
query11	15472	14915	14976	14915
query12	180	115	121	115
query13	1264	508	393	393
query14	5995	3054	2802	2802
query14_1	2699	2615	2644	2615
query15	212	196	178	178
query16	924	438	468	438
query17	1100	704	578	578
query18	2462	440	336	336
query19	229	227	195	195
query20	127	122	113	113
query21	229	149	117	117
query22	3986	3957	3925	3925
query23	16797	16285	16260	16260
query23_1	16087	16452	16330	16330
query24	7159	1657	1247	1247
query24_1	1268	1255	1271	1255
query25	557	500	458	458
query26	1258	276	165	165
query27	2756	480	319	319
query28	4387	2150	2132	2132
query29	852	575	491	491
query30	316	237	211	211
query31	826	717	634	634
query32	86	68	74	68
query33	564	338	312	312
query34	892	907	550	550
query35	761	805	710	710
query36	875	906	824	824
query37	136	97	81	81
query38	2936	3025	3010	3010
query39	765	767	713	713
query39_1	697	704	702	702
query40	236	158	126	126
query41	72	72	68	68
query42	114	113	110	110
query43	432	434	404	404
query44	1328	752	744	744
query45	191	197	186	186
query46	888	982	626	626
query47	1662	1729	1622	1622
query48	333	342	262	262
query49	651	441	374	374
query50	680	314	221	221
query51	3771	3824	3762	3762
query52	111	112	102	102
query53	323	353	303	303
query54	298	282	267	267
query55	85	79	75	75
query56	302	325	324	324
query57	1192	1174	1110	1110
query58	290	259	262	259
query59	2354	2517	2544	2517
query60	327	312	307	307
query61	201	213	160	160
query62	743	695	674	674
query63	327	298	301	298
query64	4848	1308	1009	1009
query65	4040	3961	3954	3954
query66	1440	462	318	318
query67	15337	15029	14779	14779
query68	8350	1022	712	712
query69	494	349	319	319
query70	1087	997	916	916
query71	356	310	282	282
query72	6213	5038	5122	5038
query73	731	661	312	312
query74	8682	9089	8924	8924
query75	3200	3200	2796	2796
query76	4034	1137	762	762
query77	557	405	303	303
query78	9457	9524	8785	8785
query79	1389	864	620	620
query80	694	665	563	563
query81	507	272	238	238
query82	218	134	104	104
query83	267	258	244	244
query84	262	122	111	111
query85	905	542	470	470
query86	373	308	280	280
query87	3144	3209	3185	3185
query88	3725	2297	2284	2284
query89	460	426	409	409
query90	2183	170	164	164
query91	177	166	151	151
query92	89	67	67	67
query93	1835	935	549	549
query94	482	304	286	286
query95	582	325	309	309
query96	597	473	211	211
query97	2249	2294	2241	2241
query98	216	195	198	195
query99	1363	1344	1292	1292
Total cold run time: 261811 ms
Total hot run time: 180633 ms

doris-robot · 2025-12-26T14:23:45Z

ClickBench: Total hot run time: 27.43 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3b3d9b885b2d957456d8172f3c8ad8aa3719267c, data reload: false
query1	0.06	0.04	0.04
query2	0.10	0.06	0.05
query3	0.25	0.08	0.08
query4	1.61	0.12	0.12
query5	0.27	0.25	0.27
query6	1.18	0.65	0.63
query7	0.03	0.02	0.03
query8	0.05	0.04	0.04
query9	0.56	0.50	0.50
query10	0.55	0.54	0.54
query11	0.16	0.10	0.10
query12	0.16	0.12	0.13
query13	0.61	0.62	0.60
query14	0.98	0.98	1.00
query15	0.82	0.81	0.78
query16	0.39	0.39	0.38
query17	1.03	1.05	0.99
query18	0.23	0.22	0.22
query19	1.94	1.87	1.87
query20	0.02	0.01	0.01
query21	15.42	0.30	0.13
query22	4.71	0.05	0.05
query23	16.00	0.28	0.10
query24	2.13	0.45	0.80
query25	0.09	0.06	0.05
query26	0.13	0.14	0.13
query27	0.05	0.06	0.06
query28	4.89	1.21	1.04
query29	12.63	4.13	3.25
query30	0.28	0.13	0.12
query31	2.81	0.60	0.38
query32	3.23	0.54	0.45
query33	2.97	3.03	3.02
query34	16.91	5.20	4.49
query35	4.57	4.55	4.62
query36	0.66	0.50	0.52
query37	0.11	0.07	0.06
query38	0.08	0.04	0.04
query39	0.04	0.02	0.03
query40	0.18	0.15	0.13
query41	0.08	0.03	0.03
query42	0.05	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 99.06 s
Total hot run time: 27.43 s

hello-stephen · 2025-12-26T14:49:45Z

BE UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.40% (18952/35489)
Line Coverage	39.27% (175785/447674)
Region Coverage	33.86% (136117/402007)
Branch Coverage	34.76% (58724/168938)

hello-stephen · 2025-12-26T17:41:32Z

BE Regression && UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	58.15% (20176/34694)
Line Coverage	43.90% (196006/446482)
Region Coverage	38.53% (156492/406172)
Branch Coverage	39.20% (66453/169502)

hello-stephen · 2025-12-27T02:54:56Z

BE Regression && UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	58.15% (20176/34694)
Line Coverage	43.90% (196006/446482)
Region Coverage	38.53% (156492/406172)
Branch Coverage	39.20% (66453/169502)

…g an S3 object

liaoxin01 · 2025-12-27T05:50:18Z

run buildall

doris-robot · 2025-12-27T06:28:33Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.52% (1771/2227)
Line Coverage	64.83% (31311/48299)
Region Coverage	65.36% (15576/23831)
Branch Coverage	55.97% (8281/14796)

doris-robot · 2025-12-27T09:30:56Z

BE UT Coverage Report

Increment line coverage 76.92% (10/13) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.40% (18952/35489)
Line Coverage	39.27% (175795/447698)
Region Coverage	33.83% (135994/402017)
Branch Coverage	34.75% (58715/168942)

hello-stephen · 2025-12-27T10:40:33Z

BE Regression && UT Coverage Report

Increment line coverage 100.00% (15/15) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	72.16% (25035/34694)
Line Coverage	58.87% (262858/446508)
Region Coverage	53.65% (217905/406185)
Branch Coverage	55.22% (93599/169508)

doris-robot · 2025-12-27T13:13:12Z

TPC-H: Total hot run time: 34891 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7031fc6ff2d1bcad349400f73064e5e63c94ebbe, data reload: false
------ Round 1 ----------------------------------
q1	16457	4151	4062	4062
q2	2007	342	239	239
q3	9703	1305	749	749
q4	9899	921	321	321
q5	7496	2101	1953	1953
q6	182	170	134	134
q7	1002	858	716	716
q8	9260	1409	1160	1160
q9	6992	5312	5490	5312
q10	6794	2382	1986	1986
q11	529	340	306	306
q12	651	727	592	592
q13	17594	3686	3009	3009
q14	293	299	282	282
q15	584	516	515	515
q16	693	686	631	631
q17	675	803	500	500
q18	7404	7215	7002	7002
q19	922	963	575	575
q20	400	362	242	242
q21	4160	3986	3641	3641
q22	1052	1021	964	964
Total cold run time: 104749 ms
Total hot run time: 34891 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4061	4030	4029	4029
q2	355	399	331	331
q3	2089	2678	2280	2280
q4	1317	1777	1312	1312
q5	4207	4106	4736	4106
q6	247	195	144	144
q7	2125	1978	1827	1827
q8	2688	2512	2585	2512
q9	7553	7552	7531	7531
q10	3078	3207	2804	2804
q11	613	549	514	514
q12	692	770	639	639
q13	3718	3915	3396	3396
q14	282	317	292	292
q15	576	549	500	500
q16	639	700	643	643
q17	1178	1481	1413	1413
q18	8169	7646	7489	7489
q19	837	840	921	840
q20	1935	1956	1811	1811
q21	4635	4185	4037	4037
q22	1074	1031	971	971
Total cold run time: 52068 ms
Total hot run time: 49421 ms

doris-robot · 2025-12-27T13:24:21Z

TPC-DS: Total hot run time: 178887 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7031fc6ff2d1bcad349400f73064e5e63c94ebbe, data reload: false
query5	4759	587	463	463
query6	332	236	245	236
query7	4223	463	278	278
query8	318	264	239	239
query9	8800	2574	2570	2570
query10	512	373	318	318
query11	15612	15143	14776	14776
query12	178	118	117	117
query13	1258	488	411	411
query14	5866	3098	2802	2802
query14_1	2661	2674	2750	2674
query15	211	195	180	180
query16	870	462	461	461
query17	1147	710	602	602
query18	2446	445	359	359
query19	234	238	213	213
query20	123	118	114	114
query21	224	138	119	119
query22	3922	4046	3897	3897
query23	16526	16326	15950	15950
query23_1	16006	16011	16086	16011
query24	7041	1653	1237	1237
query24_1	1273	1241	1246	1241
query25	587	516	452	452
query26	1248	278	173	173
query27	2743	475	302	302
query28	4323	2153	2112	2112
query29	820	572	470	470
query30	316	239	218	218
query31	821	721	621	621
query32	82	71	71	71
query33	554	348	305	305
query34	892	889	550	550
query35	774	807	721	721
query36	870	968	813	813
query37	140	95	77	77
query38	3007	3022	3003	3003
query39	760	747	734	734
query39_1	705	732	699	699
query40	220	138	118	118
query41	68	62	60	60
query42	106	105	112	105
query43	438	437	414	414
query44	1320	750	741	741
query45	195	187	182	182
query46	880	991	603	603
query47	1675	1694	1616	1616
query48	319	325	244	244
query49	628	440	358	358
query50	667	294	214	214
query51	3811	3834	3820	3820
query52	112	107	102	102
query53	319	348	298	298
query54	283	255	264	255
query55	86	78	75	75
query56	301	309	293	293
query57	1191	1183	1093	1093
query58	266	257	248	248
query59	2429	2501	2368	2368
query60	306	311	289	289
query61	203	155	157	155
query62	781	696	672	672
query63	342	298	307	298
query64	4866	1317	1015	1015
query65	3993	3910	3959	3910
query66	1454	427	312	312
query67	15158	14862	14622	14622
query68	8405	1014	707	707
query69	506	343	315	315
query70	1070	953	998	953
query71	371	305	284	284
query72	6098	4879	4948	4879
query73	682	591	311	311
query74	8872	8927	8664	8664
query75	3205	3134	2778	2778
query76	3986	1126	749	749
query77	568	392	297	297
query78	9593	9598	8812	8812
query79	1631	898	612	612
query80	738	650	561	561
query81	529	268	234	234
query82	207	132	103	103
query83	264	260	248	248
query84	265	123	101	101
query85	918	513	463	463
query86	378	314	268	268
query87	3314	3293	3110	3110
query88	3282	2307	2298	2298
query89	476	426	391	391
query90	2121	156	155	155
query91	187	170	147	147
query92	82	69	63	63
query93	1596	935	577	577
query94	482	316	282	282
query95	570	323	353	323
query96	606	477	210	210
query97	2272	2308	2261	2261
query98	213	193	197	193
query99	1335	1404	1254	1254
Total cold run time: 260510 ms
Total hot run time: 178887 ms

doris-robot · 2025-12-27T13:29:32Z

ClickBench: Total hot run time: 28.38 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7031fc6ff2d1bcad349400f73064e5e63c94ebbe, data reload: false
query1	0.05	0.04	0.05
query2	0.14	0.07	0.07
query3	0.32	0.08	0.08
query4	1.61	0.10	0.11
query5	0.27	0.25	0.24
query6	1.18	0.63	0.64
query7	0.03	0.03	0.02
query8	0.08	0.06	0.06
query9	0.57	0.51	0.50
query10	0.55	0.54	0.55
query11	0.25	0.13	0.14
query12	0.26	0.15	0.14
query13	0.64	0.63	0.61
query14	1.01	1.01	1.00
query15	0.89	0.82	0.81
query16	0.37	0.37	0.38
query17	1.08	1.01	1.07
query18	0.24	0.22	0.22
query19	1.84	1.76	1.80
query20	0.02	0.02	0.01
query21	15.39	0.27	0.24
query22	4.96	0.10	0.09
query23	15.41	0.39	0.23
query24	2.38	0.47	0.32
query25	0.10	0.10	0.09
query26	0.18	0.17	0.18
query27	0.09	0.10	0.10
query28	3.89	1.34	1.16
query29	12.65	4.17	3.31
query30	0.32	0.13	0.10
query31	2.81	0.63	0.43
query32	3.24	0.60	0.50
query33	2.98	3.12	3.10
query34	16.81	5.24	4.57
query35	4.66	4.65	4.65
query36	0.62	0.50	0.49
query37	0.25	0.09	0.08
query38	0.21	0.05	0.05
query39	0.06	0.04	0.04
query40	0.20	0.17	0.16
query41	0.13	0.06	0.06
query42	0.08	0.05	0.05
query43	0.06	0.05	0.05
Total cold run time: 98.88 s
Total hot run time: 28.38 s

dataroaring

LGTM

github-actions · 2025-12-30T18:07:50Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-12-30T18:07:53Z

PR approved by anyone and no changes requested.

…g an S3 object (apache#59414)

CopilotAI review requested due to automatic review settings December 26, 2025 13:41

liaoxin01 requested review from dataroaring and gavinchou as code owners December 26, 2025 13:41

liaoxin01 marked this pull request as draft December 26, 2025 13:41

Copilotstarted reviewing on behalf of liaoxin01 December 26, 2025 13:41 View session

CopilotAI reviewed Dec 26, 2025

[opt](packed-file) Avoid unnecessary HEAD Object requests when openin…
7031fc6
…g an S3 object

liaoxin01 force-pushed the opt_packed_file_s3_head branch from 3b3d9b8 to 7031fc6Compare December 27, 2025 05:49

liaoxin01 marked this pull request as ready for review December 27, 2025 05:52

liaoxin01 requested a review from yiguolei as a code owner December 27, 2025 05:52

liaoxin01 added the packed-file label Dec 27, 2025

dataroaring approved these changes Dec 30, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Dec 30, 2025

github-actionsBot added the reviewed label Dec 30, 2025

luwei16 approved these changes Jan 4, 2026
View reviewed changes

liaoxin01 merged commit 9d34c04 into apache:masterJan 4, 2026
26 of 29 checks passed

liaoxin01 deleted the opt_packed_file_s3_head branch January 4, 2026 11:05

liaoxin01 added a commit to liaoxin01/doris that referenced this pull request Jan 8, 2026
[opt](packed-file) Avoid unnecessary HEAD Object requests when openin…
3cfde73
…g an S3 object (apache#59414)

liaoxin01 added a commit to liaoxin01/doris that referenced this pull request Jan 9, 2026
[opt](packed-file) Avoid unnecessary HEAD Object requests when openin…
6a13aee
…g an S3 object (apache#59414)

yiguolei added the dev/4.0.3-merged label Jan 12, 2026

Uh oh!

Conversation

liaoxin01 commented Dec 26, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Dec 26, 2025

Uh oh!

liaoxin01 commented Dec 26, 2025

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Dec 26, 2025

Uh oh!

doris-robot commented Dec 26, 2025

Uh oh!

doris-robot commented Dec 26, 2025

Uh oh!

hello-stephen commented Dec 26, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Dec 26, 2025

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Dec 27, 2025

BE Regression && UT Coverage Report

Uh oh!

liaoxin01 commented Dec 27, 2025

Uh oh!

doris-robot commented Dec 27, 2025

Cloud UT Coverage Report

Uh oh!

doris-robot commented Dec 27, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Dec 27, 2025

BE Regression && UT Coverage Report

Uh oh!

doris-robot commented Dec 27, 2025

Uh oh!

doris-robot commented Dec 27, 2025

Uh oh!

doris-robot commented Dec 27, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Dec 30, 2025

Uh oh!

github-actionsBot commented Dec 30, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants