Skip to content

[fix](packed-file) use timestamp-only cleanup for packed slice index - #59369

Merged
gavinchou merged 1 commit into
apache:masterfrom
luwei16:luwei/fix-packed-file-cleanup
Jan 5, 2026
Merged

[fix](packed-file) use timestamp-only cleanup for packed slice index#59369
gavinchou merged 1 commit into
apache:masterfrom
luwei16:luwei/fix-packed-file-cleanup

Conversation

@luwei16

@luwei16luwei16 commented Dec 25, 2025

Copy link
Copy Markdown
Contributor

No description provided.

@Thearas

Thearas commented Dec 25, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@luwei16

luwei16 commented Dec 25, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Dec 25, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 36353 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7c547e0543a06b1018a46a40fbff53856fea6aec, data reload: false
------ Round 1 ----------------------------------
q1	17633	4291	4070	4070
q2	2001	345	238	238
q3	10210	1293	729	729
q4	10243	927	329	329
q5	7523	2119	1959	1959
q6	188	166	133	133
q7	1020	839	709	709
q8	9375	1451	1148	1148
q9	7013	5362	5400	5362
q10	6833	2411	1980	1980
q11	539	323	315	315
q12	662	721	599	599
q13	17775	3650	3065	3065
q14	297	286	275	275
q15	591	510	513	510
q16	692	697	656	656
q17	704	824	506	506
q18	7499	7624	7740	7624
q19	1162	998	610	610
q20	438	379	264	264
q21	4468	4233	4233	4233
q22	1127	1088	1039	1039
Total cold run time: 107993 ms
Total hot run time: 36353 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4299	4281	4325	4281
q2	330	414	309	309
q3	2448	2891	2540	2540
q4	1514	1798	1389	1389
q5	4432	4556	4421	4421
q6	231	197	151	151
q7	2043	1922	1783	1783
q8	2587	2520	2477	2477
q9	7531	7539	7523	7523
q10	3101	3133	2688	2688
q11	592	533	509	509
q12	626	702	570	570
q13	3300	3640	3064	3064
q14	267	317	263	263
q15	532	500	486	486
q16	635	661	614	614
q17	1135	1397	1374	1374
q18	7518	7331	7141	7141
q19	863	815	862	815
q20	1923	1988	1845	1845
q21	4691	4262	4173	4173
q22	1032	1050	981	981
Total cold run time: 51630 ms
Total hot run time: 49397 ms

@doris-robot

doris-robot commented Dec 25, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 180431 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7c547e0543a06b1018a46a40fbff53856fea6aec, data reload: false
query5	5404	603	431	431
query6	334	228	217	217
query7	4214	464	275	275
query8	319	256	232	232
query9	8774	2518	2536	2518
query10	535	372	319	319
query11	15376	15384	15484	15384
query12	183	115	113	113
query13	1265	494	383	383
query14	6660	2976	2783	2783
query14_1	2662	2633	2643	2633
query15	203	199	175	175
query16	965	468	451	451
query17	1150	714	615	615
query18	2712	454	353	353
query19	247	234	202	202
query20	122	117	115	115
query21	221	144	118	118
query22	3993	3996	3898	3898
query23	16640	16228	16238	16228
query23_1	16198	16138	16123	16123
query24	7286	1660	1219	1219
query24_1	1258	1253	1260	1253
query25	591	498	467	467
query26	1267	273	166	166
query27	2753	473	319	319
query28	4489	2104	2118	2104
query29	827	568	469	469
query30	321	247	218	218
query31	836	714	626	626
query32	78	70	68	68
query33	566	362	302	302
query34	895	892	535	535
query35	775	813	718	718
query36	848	911	817	817
query37	133	100	85	85
query38	3009	2939	2917	2917
query39	761	742	740	740
query39_1	728	717	689	689
query40	223	145	119	119
query41	89	76	63	63
query42	111	100	100	100
query43	431	415	390	390
query44	1304	760	739	739
query45	192	191	180	180
query46	859	973	601	601
query47	1704	1711	1642	1642
query48	314	315	240	240
query49	629	428	354	354
query50	661	306	223	223
query51	3790	3793	3869	3793
query52	110	111	100	100
query53	328	356	306	306
query54	293	250	251	250
query55	86	74	76	74
query56	298	311	295	295
query57	1182	1151	1115	1115
query58	264	255	266	255
query59	2437	2466	2440	2440
query60	319	316	294	294
query61	163	199	154	154
query62	745	681	669	669
query63	330	295	300	295
query64	4950	1295	999	999
query65	4052	3961	3969	3961
query66	1373	442	310	310
query67	15265	15042	14984	14984
query68	8236	1004	713	713
query69	502	352	318	318
query70	1084	1008	982	982
query71	371	311	293	293
query72	6052	4955	5090	4955
query73	685	607	308	308
query74	9166	8999	8758	8758
query75	3191	3177	2809	2809
query76	3855	1146	761	761
query77	523	411	289	289
query78	9487	9715	8903	8903
query79	1638	874	614	614
query80	739	676	554	554
query81	517	270	231	231
query82	209	139	104	104
query83	259	255	233	233
query84	259	121	106	106
query85	901	515	461	461
query86	366	308	285	285
query87	3089	3176	3036	3036
query88	3247	2291	2289	2289
query89	468	427	402	402
query90	2156	163	155	155
query91	176	173	145	145
query92	78	70	65	65
query93	1295	907	556	556
query94	470	306	292	292
query95	563	392	304	304
query96	589	464	216	216
query97	2285	2285	2238	2238
query98	218	194	199	194
query99	1314	1340	1353	1340
Total cold run time: 262412 ms
Total hot run time: 180431 ms

@doris-robot

doris-robot commented Dec 25, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 27.71 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7c547e0543a06b1018a46a40fbff53856fea6aec, data reload: false
query1	0.06	0.05	0.06
query2	0.10	0.05	0.04
query3	0.25	0.09	0.09
query4	1.60	0.11	0.11
query5	0.26	0.28	0.27
query6	1.17	0.64	0.63
query7	0.03	0.03	0.02
query8	0.05	0.04	0.04
query9	0.55	0.49	0.50
query10	0.56	0.54	0.55
query11	0.15	0.11	0.11
query12	0.14	0.12	0.11
query13	0.62	0.60	0.61
query14	0.98	0.99	0.98
query15	0.81	0.79	0.82
query16	0.40	0.41	0.40
query17	0.96	1.00	1.05
query18	0.24	0.22	0.21
query19	1.91	1.79	1.80
query20	0.02	0.02	0.02
query21	15.45	0.26	0.13
query22	5.24	0.05	0.04
query23	16.17	0.29	0.11
query24	1.15	0.60	0.70
query25	0.10	0.06	0.06
query26	0.15	0.14	0.14
query27	0.10	0.08	0.05
query28	5.54	1.22	1.03
query29	12.59	3.93	3.22
query30	0.27	0.14	0.11
query31	2.81	0.62	0.39
query32	3.23	0.54	0.47
query33	2.99	3.06	3.06
query34	16.68	5.19	4.57
query35	4.55	4.55	4.59
query36	0.66	0.52	0.51
query37	0.11	0.07	0.07
query38	0.08	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.14	0.14
query41	0.09	0.04	0.03
query42	0.04	0.04	0.04
query43	0.04	0.04	0.03
Total cold run time: 99.11 s
Total hot run time: 27.71 s

@doris-robot

doris-robot commented Dec 25, 2025

Copy link
Copy Markdown

BE UT Coverage Report

Increment line coverage 33.33% (1/3) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage53.41% (18953/35489)
Line Coverage39.26% (175727/447621)
Region Coverage33.82% (135933/401945)
Branch Coverage34.76% (58713/168924)

@hello-stephen

hello-stephen commented Dec 25, 2025

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (3/3) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage73.85% (25621/34694)
Line Coverage61.26% (273470/446429)
Region Coverage56.26% (228464/406116)
Branch Coverage58.06% (98407/169488)

liaoxin01
liaoxin01 approved these changes Jan 4, 2026
View reviewed changes

@liaoxin01liaoxin01 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

github-actionsBot commented Jan 4, 2026

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Jan 4, 2026
@github-actions

github-actionsBot commented Jan 4, 2026

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

gavinchou
gavinchou approved these changes Jan 5, 2026
View reviewed changes

@gavinchougavinchou left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment threadbe/src/io/fs/packed_file_manager.h
std::string packed_file_path;
int64_t offset;
int64_t size;
int64_t create_time = 0;

@gavinchougavinchouJan 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

naming creation_time

@gavinchou
gavinchou merged commit 6ec35ec into apache:masterJan 5, 2026
29 of 31 checks passed
liaoxin01 pushed a commit to liaoxin01/doris that referenced this pull request Jan 8, 2026
@luwei16@liaoxin01
[fix](packed-file) use timestamp-only cleanup for packed slice index (a…
d51d22c
…pache#59369)
liaoxin01 pushed a commit to liaoxin01/doris that referenced this pull request Jan 9, 2026
@luwei16@liaoxin01
[fix](packed-file) use timestamp-only cleanup for packed slice index (a…
0a651f7
…pache#59369)
zzzxl1993 pushed a commit to zzzxl1993/doris that referenced this pull request Jan 13, 2026
@luwei16@zzzxl1993
[fix](packed-file) use timestamp-only cleanup for packed slice index (a…
98b7d3b
…pache#59369)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchougavinchougavinchou approved these changes
@liaoxin01liaoxin01liaoxin01 approved these changes
@dataroaringdataroaringAwaiting requested review from dataroaring

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/4.0.3-mergedpacked-filereviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@luwei16@Thearas@doris-robot@hello-stephen@gavinchou@liaoxin01@yiguolei