Skip to content

[fix](packed-file) enable file cache for packed file - #59575

Merged
liaoxin01 merged 1 commit into
apache:masterfrom
liaoxin01:fix_packed_file_cache
Jan 6, 2026
Merged

[fix](packed-file) enable file cache for packed file#59575
liaoxin01 merged 1 commit into
apache:masterfrom
liaoxin01:fix_packed_file_cache

Conversation

@liaoxin01

@liaoxin01liaoxin01 commented Jan 5, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Enables file cache for packed file writers by setting write_file_cache = true in FileWriterOptions.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

CopilotAI review requested due to automatic review settings January 5, 2026 12:28
@hello-stephen

hello-stephen commented Jan 5, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@liaoxin01

liaoxin01 commented Jan 5, 2026

Copy link
Copy Markdown
ContributorAuthor

run buildall

CopilotAI reviewed Jan 5, 2026
View reviewed changes

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR enables write file cache for packed files to improve performance when writing to remote storage in cloud mode.

Key Changes:

  • Enables file cache for packed file writers by setting write_file_cache = true in FileWriterOptions

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@doris-robot

doris-robot commented Jan 5, 2026

Copy link
Copy Markdown
TPC-H: Total hot run time: 31739 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 789db4b08475005fc93de35a7eff44c12027514b, data reload: false
------ Round 1 ----------------------------------
q1	17615	4208	4037	4037
q2	2037	346	254	254
q3	10128	1327	717	717
q4	10211	899	320	320
q5	7527	2109	1842	1842
q6	188	169	139	139
q7	943	802	650	650
q8	9277	1411	1173	1173
q9	4908	4694	4536	4536
q10	6755	1810	1413	1413
q11	532	300	280	280
q12	710	736	583	583
q13	17752	3902	3086	3086
q14	283	296	299	296
q15	570	518	507	507
q16	674	688	652	652
q17	685	813	493	493
q18	6756	6426	6759	6426
q19	1322	1051	681	681
q20	441	412	268	268
q21	3227	2659	2361	2361
q22	1118	1105	1025	1025
Total cold run time: 103659 ms
Total hot run time: 31739 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4361	4440	4217	4217
q2	328	422	338	338
q3	2343	2761	2450	2450
q4	1432	1849	1455	1455
q5	4441	4243	4394	4243
q6	235	178	131	131
q7	1958	1904	1794	1794
q8	2530	2425	2415	2415
q9	7254	7156	7259	7156
q10	2488	2717	2368	2368
q11	554	473	453	453
q12	677	787	646	646
q13	3740	4131	3339	3339
q14	282	302	275	275
q15	522	496	478	478
q16	601	649	596	596
q17	1083	1322	1303	1303
q18	7632	7319	7332	7319
q19	830	801	832	801
q20	1899	1967	1820	1820
q21	4482	4242	4058	4058
q22	1066	1025	984	984
Total cold run time: 50738 ms
Total hot run time: 48639 ms

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Jan 5, 2026
@github-actions

github-actionsBot commented Jan 5, 2026

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Jan 5, 2026

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@doris-robot

doris-robot commented Jan 5, 2026

Copy link
Copy Markdown
TPC-DS: Total hot run time: 172714 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 789db4b08475005fc93de35a7eff44c12027514b, data reload: false
query5	5086	590	429	429
query6	326	236	220	220
query7	4214	461	261	261
query8	335	274	237	237
query9	8766	2638	2659	2638
query10	551	364	323	323
query11	15355	15064	14923	14923
query12	189	112	111	111
query13	1243	492	363	363
query14	6758	2991	2717	2717
query14_1	2602	2596	2601	2596
query15	206	192	169	169
query16	1000	489	391	391
query17	1078	674	586	586
query18	2704	468	322	322
query19	216	213	193	193
query20	123	118	113	113
query21	210	137	116	116
query22	4041	4042	4045	4042
query23	16154	15588	15447	15447
query23_1	15490	15554	15643	15554
query24	7269	1551	1170	1170
query24_1	1163	1152	1175	1152
query25	532	434	393	393
query26	1248	283	164	164
query27	2726	449	286	286
query28	4510	2159	2144	2144
query29	799	522	438	438
query30	311	235	212	212
query31	772	620	541	541
query32	72	66	64	64
query33	517	342	296	296
query34	922	876	526	526
query35	748	804	706	706
query36	877	883	853	853
query37	132	96	81	81
query38	2758	2812	2659	2659
query39	783	755	747	747
query39_1	712	723	723	723
query40	224	139	121	121
query41	75	68	66	66
query42	103	101	100	100
query43	468	450	438	438
query44	1298	725	727	725
query45	192	182	178	178
query46	850	965	589	589
query47	1385	1398	1402	1398
query48	312	322	245	245
query49	623	450	338	338
query50	633	283	211	211
query51	3796	3822	3755	3755
query52	113	116	102	102
query53	294	325	274	274
query54	302	270	259	259
query55	77	77	70	70
query56	292	299	310	299
query57	1023	988	938	938
query58	273	261	257	257
query59	2002	2145	2136	2136
query60	332	317	303	303
query61	190	183	188	183
query62	391	364	321	321
query63	302	269	270	269
query64	5142	1414	1076	1076
query65	3745	3711	3618	3618
query66	1370	426	295	295
query67	15244	14646	15458	14646
query68	8092	978	698	698
query69	501	376	311	311
query70	1011	986	933	933
query71	374	302	272	272
query72	6023	3427	3449	3427
query73	773	720	305	305
query74	8944	8808	8642	8642
query75	2839	2815	2455	2455
query76	4022	1061	652	652
query77	581	375	287	287
query78	9740	9813	9173	9173
query79	1576	841	588	588
query80	603	554	466	466
query81	534	266	226	226
query82	494	140	108	108
query83	261	250	237	237
query84	257	127	105	105
query85	924	507	447	447
query86	395	326	286	286
query87	2851	2894	2782	2782
query88	4322	2223	2232	2223
query89	384	356	333	333
query90	2205	168	159	159
query91	171	170	141	141
query92	86	66	62	62
query93	1664	893	532	532
query94	566	313	309	309
query95	571	329	303	303
query96	583	478	204	204
query97	2353	2385	2281	2281
query98	227	202	198	198
query99	634	582	496	496
Total cold run time: 258176 ms
Total hot run time: 172714 ms

@doris-robot

doris-robot commented Jan 5, 2026

Copy link
Copy Markdown
ClickBench: Total hot run time: 27.02 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 789db4b08475005fc93de35a7eff44c12027514b, data reload: false
query1	0.05	0.05	0.04
query2	0.10	0.04	0.04
query3	0.26	0.08	0.08
query4	1.60	0.11	0.12
query5	0.28	0.26	0.24
query6	1.15	0.65	0.66
query7	0.03	0.03	0.02
query8	0.05	0.04	0.04
query9	0.56	0.50	0.50
query10	0.56	0.55	0.55
query11	0.14	0.10	0.09
query12	0.14	0.11	0.11
query13	0.61	0.58	0.59
query14	0.95	0.97	0.94
query15	0.79	0.78	0.78
query16	0.43	0.39	0.40
query17	1.07	1.05	1.06
query18	0.24	0.21	0.21
query19	1.98	1.84	1.80
query20	0.02	0.02	0.01
query21	15.46	0.23	0.14
query22	5.23	0.05	0.05
query23	16.09	0.28	0.10
query24	1.10	0.43	0.48
query25	0.09	0.06	0.06
query26	0.16	0.13	0.14
query27	0.06	0.06	0.05
query28	4.39	1.06	0.88
query29	12.66	3.92	3.19
query30	0.27	0.13	0.13
query31	2.82	0.66	0.40
query32	3.23	0.56	0.45
query33	2.99	3.04	3.10
query34	16.89	5.10	4.50
query35	4.49	4.43	4.48
query36	0.65	0.50	0.49
query37	0.11	0.06	0.07
query38	0.06	0.04	0.03
query39	0.05	0.03	0.03
query40	0.17	0.14	0.12
query41	0.08	0.04	0.02
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 98.14 s
Total hot run time: 27.02 s

@doris-robot

doris-robot commented Jan 5, 2026

Copy link
Copy Markdown

BE UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage53.21% (18964/35637)
Line Coverage39.21% (175981/448801)
Region Coverage33.70% (136016/403632)
Branch Coverage34.68% (58723/169312)

dataroaring
dataroaring approved these changes Jan 5, 2026
View reviewed changes

@dataroaringdataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liaoxin01
liaoxin01 merged commit 33436ac into apache:masterJan 6, 2026
39 of 41 checks passed
@liaoxin01
liaoxin01 deleted the fix_packed_file_cache branch January 6, 2026 02:20
liaoxin01 added a commit to liaoxin01/doris that referenced this pull request Jan 8, 2026
@liaoxin01
[fix](packed-file) enable file cache for packed file (apache#59575)
ebeb626
liaoxin01 added a commit to liaoxin01/doris that referenced this pull request Jan 9, 2026
@liaoxin01
[fix](packed-file) enable file cache for packed file (apache#59575)
f0ef90d
zzzxl1993 pushed a commit to zzzxl1993/doris that referenced this pull request Jan 13, 2026
@liaoxin01@zzzxl1993
[fix](packed-file) enable file cache for packed file (apache#59575)
48c7e65
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@gavinchougavinchougavinchou approved these changes
@luwei16luwei16luwei16 approved these changes
@dataroaringdataroaringdataroaring approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/4.0.3-mergedpacked-filereviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@liaoxin01@hello-stephen@doris-robot@gavinchou@luwei16@dataroaring@yiguolei