[fix](auto-partition) Register auto-partition tables with retention_count in DynamicPartitionScheduler on restart by zclllyybb · Pull Request #61954 · apache/doris

zclllyybb · 2026-03-31T11:12:45Z

Related PR: #57013

Auto-partition tables with partition.retention_count set were not being registered in DynamicPartitionScheduler during FE restart, causing retention policies to stop working until manual re-registration. Modified initDynamicPartitionTable() to check for both dynamic partition tables and OlapTables with retention_count > 0, ensuring these tables are properly tracked after restart

hello-stephen · 2026-03-31T11:12:51Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zclllyybb · 2026-03-31T11:13:04Z

run buildall

Copilot

Pull request overview

Fixes a restart-time gap where AUTO PARTITION tables configured with partition.retention_count were not being re-registered into DynamicPartitionScheduler, causing partition retention cleanup to stop working after FE restart until manual re-registration.

Changes:

Extend DynamicPartitionScheduler.initDynamicPartitionTable() to also register OlapTables with partition.retention_count > 0.
Add an FE unit test to verify retention-count auto-partition tables are re-registered when the scheduler is (re-)initialized.
Add a docker regression test intended to validate retention behavior across FE restart.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
`fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java`	Registers AUTO RANGE auto-partition tables with `partition.retention_count` during scheduler initialization on restart.
`fe/fe-core/src/test/java/org/apache/doris/catalog/DynamicPartitionTableTest.java`	Adds a unit test verifying `initDynamicPartitionTable()` re-registers retention-count auto-partition tables.
`regression-test/suites/partition_p0/auto_partition/test_auto_partition_retention_restart.groovy`	Adds a docker regression test for retention-count behavior after FE restart.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…ount in DynamicPartitionScheduler on restart Auto-partition tables with `partition.retention_count` set were not being registered in DynamicPartitionScheduler during FE restart, causing retention policies to stop working until manual re-registration. Modified `initDynamicPartitionTable()` to check for both dynamic partition tables and OlapTables with retention_count > 0, ensuring these tables are properly tracked after restart

zclllyybb · 2026-03-31T11:39:09Z

run buildall

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

github-actions · 2026-03-31T11:47:57Z

PR approved by anyone and no changes requested.

github-actions · 2026-03-31T11:53:09Z

PR approved by at least one committer and no changes requested.

doris-robot · 2026-03-31T11:57:19Z

TPC-H: Total hot run time: 26804 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fb4962a8e76f7e046d3bc2c4c905c6194306f4dc, data reload: false
------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	17595	4480	4316	4316
q2	q3	10649	766	528	528
q4	4685	360	257	257
q5	7550	1195	1035	1035
q6	176	175	148	148
q7	776	845	686	686
q8	9289	1487	1325	1325
q9	4797	4750	4756	4750
q10	6266	1945	1652	1652
q11	476	257	268	257
q12	720	590	463	463
q13	18024	2736	1922	1922
q14	229	237	212	212
q15	q16	712	744	665	665
q17	714	851	447	447
q18	5821	5485	5301	5301
q19	1250	968	638	638
q20	567	490	381	381
q21	4654	1854	1430	1430
q22	348	398	391	391
Total cold run time: 95298 ms
Total hot run time: 26804 ms
----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	4891	4642	4644	4642
q2	q3	3896	4359	3842	3842
q4	903	1208	785	785
q5	4054	4402	4353	4353
q6	182	172	143	143
q7	1774	1643	1518	1518
q8	2473	2730	2607	2607
q9	7833	7723	7354	7354
q10	3777	4019	3631	3631
q11	533	423	415	415
q12	501	608	447	447
q13	2552	2867	2342	2342
q14	299	309	279	279
q15	q16	713	774	725	725
q17	1187	1309	1353	1309
q18	7184	6969	6719	6719
q19	952	933	931	931
q20	2121	2195	2035	2035
q21	3956	3526	3370	3370
q22	516	446	376	376
Total cold run time: 50297 ms
Total hot run time: 47823 ms

doris-robot · 2026-03-31T12:08:15Z

TPC-DS: Total hot run time: 169701 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fb4962a8e76f7e046d3bc2c4c905c6194306f4dc, data reload: false
query5	4358	630	519	519
query6	340	227	203	203
query7	4218	465	261	261
query8	333	242	231	231
query9	8734	2764	2768	2764
query10	529	391	352	352
query11	6961	5079	4853	4853
query12	189	129	125	125
query13	1280	465	353	353
query14	5781	3675	3520	3520
query14_1	2865	2859	2824	2824
query15	217	195	178	178
query16	992	491	451	451
query17	883	755	632	632
query18	2435	457	358	358
query19	219	232	185	185
query20	132	129	125	125
query21	211	142	113	113
query22	13167	14999	14487	14487
query23	16605	16432	16021	16021
query23_1	15663	15845	15769	15769
query24	7122	1638	1273	1273
query24_1	1227	1238	1252	1238
query25	568	462	396	396
query26	1248	264	148	148
query27	2770	476	292	292
query28	4490	1852	1847	1847
query29	872	564	495	495
query30	299	241	195	195
query31	1019	944	870	870
query32	93	77	75	75
query33	521	335	294	294
query34	886	887	530	530
query35	635	689	599	599
query36	1121	1148	964	964
query37	139	96	84	84
query38	2898	2939	2884	2884
query39	863	826	804	804
query39_1	793	802	785	785
query40	237	187	136	136
query41	63	60	60	60
query42	263	251	252	251
query43	239	250	219	219
query44 query45	205	186	183	183
query46	877	987	603	603
query47	2108	2164	2062	2062
query48	313	316	236	236
query49	643	459	381	381
query50	685	281	216	216
query51	4060	4024	4017	4017
query52	263	268	255	255
query53	300	337	288	288
query54	309	280	274	274
query55	92	89	86	86
query56	315	317	320	317
query57	1919	1813	1723	1723
query58	285	270	273	270
query59	2811	2978	2744	2744
query60	340	338	322	322
query61	160	159	165	159
query62	630	597	539	539
query63	305	278	273	273
query64	5124	1281	996	996
query65 query66	1460	457	353	353
query67	24313	24170	24111	24111
query68 query69	432	316	286	286
query70	937	945	983	945
query71	332	310	303	303
query72	2842	2734	2497	2497
query73	544	532	312	312
query74	9597	9586	9404	9404
query75	2836	2753	2464	2464
query76	2289	1039	668	668
query77	368	401	320	320
query78	10918	11153	10472	10472
query79	1132	777	596	596
query80	1367	633	531	531
query81	540	258	226	226
query82	1283	168	124	124
query83	337	267	255	255
query84	302	120	110	110
query85	933	504	457	457
query86	425	339	328	328
query87	3158	3107	3023	3023
query88	3508	2669	2657	2657
query89	436	366	345	345
query90	2005	187	174	174
query91	169	159	142	142
query92	79	75	73	73
query93	926	840	515	515
query94	638	322	311	311
query95	595	397	326	326
query96	646	515	228	228
query97	2474	2512	2432	2432
query98	237	227	222	222
query99	1030	998	934	934
Total cold run time: 248763 ms
Total hot run time: 169701 ms

…ount in DynamicPartitionScheduler on restart (#61954) Auto-partition tables with `partition.retention_count` set were not being registered in DynamicPartitionScheduler during FE restart, causing retention policies to stop working until manual re-registration. Modified `initDynamicPartitionTable()` to check for both dynamic partition tables and OlapTables with retention_count > 0, ensuring these tables are properly tracked after restart

… retention_count in DynamicPartitionScheduler on restart #61954 (#61972) Cherry-picked from #61954 Co-authored-by: zclllyybb <zhaochangle@selectdb.com>

…ount in DynamicPartitionScheduler on restart (#61954) Auto-partition tables with `partition.retention_count` set were not being registered in DynamicPartitionScheduler during FE restart, causing retention policies to stop working until manual re-registration. Modified `initDynamicPartitionTable()` to check for both dynamic partition tables and OlapTables with retention_count > 0, ensuring these tables are properly tracked after restart

…uplication (#62477) Related PR: #61954 Problem Summary: Extract the dynamic partition table check logic into a private helper method `isDynamicScheduleTable()` and add documentation explaining that DynamicPartitionScheduler tracks both dynamic partition tables with scheduling enabled and auto partition tables using `partition.retention_count` for periodic cleanup.

… retention_count in DynamicPartitionScheduler on restart #61954 (#61971) Cherry-picked from #61954 Co-authored-by: zclllyybb <zhaochangle@selectdb.com>

zclllyybb added p0_b dev/4.0.x dev/4.1.x labels Mar 31, 2026

zclllyybb requested a review from Copilot March 31, 2026 11:12

Copilotstarted reviewing on behalf of zclllyybb March 31, 2026 11:18 View session

CopilotAI reviewed Mar 31, 2026
View reviewed changes

Comment threadregression-test/suites/partition_p0/auto_partition/test_auto_partition_retention_restart.groovy Outdated

zclllyybb force-pushed the rentention_count_fix branch from 5dde7d0 to fb4962aCompare March 31, 2026 11:35

zclllyybb requested a review from Copilot March 31, 2026 11:39

CopilotAI reviewed Mar 31, 2026
View reviewed changes

Mryange approved these changes Mar 31, 2026
View reviewed changes

github-actionsBot added the reviewed label Mar 31, 2026

BiteTheDDDDt approved these changes Mar 31, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Mar 31, 2026

BiteTheDDDDt approved these changes Mar 31, 2026
View reviewed changes

zclllyybb merged commit 09eb587 into apache:masterMar 31, 2026
36 of 37 checks passed

zclllyybb deleted the rentention_count_fix branch March 31, 2026 14:15

github-actionsBot mentioned this pull request Mar 31, 2026
branch-4.0: [fix](auto-partition) Register auto-partition tables with retention_count in DynamicPartitionScheduler on restart #61954 #61971
Merged

github-actionsBot mentioned this pull request Mar 31, 2026
branch-4.1: [fix](auto-partition) Register auto-partition tables with retention_count in DynamicPartitionScheduler on restart #61954 #61972
Merged

yiguolei removed the dev/4.1.x label Apr 1, 2026

yiguolei added the dev/4.1.0-merged label Apr 1, 2026

zclllyybb mentioned this pull request Apr 14, 2026
[refactor](fe) Extract isDynamicScheduleTable method to reduce code duplication #62477
Merged

yiguolei added dev/4.0.6-merged and removed dev/4.0.x labels Apr 15, 2026

mymeiyi mentioned this pull request Apr 21, 2026
[fix](wal) Fix WAL replay loading 0 rows due to columns being marked as missing #62658
Merged

morningman mentioned this pull request Jun 3, 2026
4.0.6 Release Notes #64065
Closed

Navigation Menu

Uh oh!

[fix](auto-partition) Register auto-partition tables with retention_count in DynamicPartitionScheduler on restart - #61954

[fix](auto-partition) Register auto-partition tables with retention_count in DynamicPartitionScheduler on restart#61954
zclllyybb merged 1 commit into
apache:masterfrom
zclllyybb:rentention_count_fix

zclllyybb commented Mar 31, 2026•
edited
Loading

Uh oh!

hello-stephen commented Mar 31, 2026

Uh oh!

zclllyybb commented Mar 31, 2026

Uh oh!

CopilotAI left a comment

Uh oh!

Uh oh!

zclllyybb commented Mar 31, 2026

Uh oh!

CopilotAI left a comment

Uh oh!

github-actionsBot commented Mar 31, 2026

Uh oh!

github-actionsBot commented Mar 31, 2026

Uh oh!

doris-robot commented Mar 31, 2026

Uh oh!

doris-robot commented Mar 31, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Conversation

zclllyybb commented Mar 31, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hello-stephen commented Mar 31, 2026

Uh oh!

zclllyybb commented Mar 31, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

zclllyybb commented Mar 31, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

github-actionsBot commented Mar 31, 2026

Uh oh!

github-actionsBot commented Mar 31, 2026

Uh oh!

doris-robot commented Mar 31, 2026

Uh oh!

doris-robot commented Mar 31, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

zclllyybb commented Mar 31, 2026•
edited
Loading