[fix](recycler) Add recycle state for rs meta to avoid data loss by wyxxxcat · Pull Request #58459 · apache/doris

wyxxxcat · 2025-11-27T09:04:58Z

What problem does this PR solve?

Summary

Add a RECYCLE state for rowset/meta (rs meta) and update the recycler logic to mark metadata as RECYCLE before final deletion. This reduces the risk of accidental data loss.

Problem

The recycler sometimes deletes rs meta too early (race conditions, restarts, or recovery cases), which can cause metadata and file inconsistencies or data loss.

Solution

Introduce a RECYCLE intermediate state for rs meta.
When an item is chosen for cleanup, mark it RECYCLE and record a timestamp.
Only perform the final delete after a confirmation window or additional checks.
Make recovery/restart logic treat RECYCLE items as recoverable until final deletion.

Main changes

Add RECYCLE to the rs meta state enum.
Update metadata APIs to set/query RECYCLE.
Update recycler to use two-step deletion: mark -> confirm -> abort txn/job and delete.
Add logs and tests for the new flow.

Test case

1. begin_txn -> prepare_rowset -> force_recycle -> commit_rowset -> commit_txn
2. start_job -> prepare_rowset -> force_recycle -> commit_rowset -> finish_job
Rowset will be marked as recycled to prevent commit_rowset and finish job/txn
3. begin_txn -> prepare_rowset -> commit_rowset -> force_recycle -> commit_txn
4. start_job -> prepare_rowset -> commit_rowset -> force_recycle -> finish_job
Rowset will be marked as recycled to prevent finish job/txn
5. begin_txn -> prepare_rowset -> force_recycle * 2 -> commit_rowset -> commit_txn
6. start_job -> prepare_rowset -> force_recycle * 2 -> commit_rowset -> finish_job
7. begin_txn -> prepare_rowset -> commit_rowset -> force_recycle * 2 -> commit_txn
9. start_job -> prepare_rowset -> commit_rowset -> force_recycle * 2 -> finish_job
10. delete_job -> commit_rowset -> force_recycle * 2 -> finish_job
11. delete_job -> prepare_rowset -> commit_rowset -> force_recycle * 2 -> finish_job
12. delete_job -> prepare_rowset -> force_recycle * 2 -> commit_rowset -> finish_job
Double recycle job will mark rowset as recycled and abort job/txn, then delete data and kv

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-11-27T09:05:04Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

gavinchou · 2025-12-03T05:56:59Z

+ optional RecycleStatePB recycle_state = 111;
+}
+
+enum RecycleStatePB {


add scope to this state, easily to be conflict

wyxxxcat · 2025-12-16T03:04:05Z

run buildall

hello-stephen · 2025-12-16T04:09:44Z

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

doris-robot · 2025-12-16T06:08:38Z

TPC-H: Total hot run time: 36404 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 97b92434f2c898d0f48eea9333e8b4f81bdde4ce, data reload: false
------ Round 1 ----------------------------------
q1	17625	4283	4065	4065
q2	2026	363	244	244
q3	10434	1360	756	756
q4	10364	833	306	306
q5	9454	2118	1954	1954
q6	216	168	133	133
q7	1022	862	716	716
q8	9379	1437	1237	1237
q9	7298	5376	5412	5376
q10	6869	2404	1947	1947
q11	517	321	288	288
q12	697	744	589	589
q13	17801	3655	2984	2984
q14	288	292	276	276
q15	590	523	525	523
q16	682	668	624	624
q17	674	825	522	522
q18	7475	8195	7731	7731
q19	1310	1012	606	606
q20	422	381	248	248
q21	4562	4404	4205	4205
q22	1128	1086	1074	1074
Total cold run time: 110833 ms
Total hot run time: 36404 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4291	4228	4247	4228
q2	430	416	312	312
q3	2502	2816	2397	2397
q4	1452	1887	1466	1466
q5	4669	4270	4620	4270
q6	204	174	126	126
q7	2065	1953	1815	1815
q8	2683	2579	2545	2545
q9	7659	7503	7375	7375
q10	2899	3116	2673	2673
q11	574	499	480	480
q12	624	706	562	562
q13	3272	3595	3023	3023
q14	265	300	261	261
q15	532	507	493	493
q16	614	638	593	593
q17	1131	1487	1368	1368
q18	7216	7042	7015	7015
q19	815	798	808	798
q20	1905	2054	1800	1800
q21	4686	4324	4192	4192
q22	1107	1091	977	977
Total cold run time: 51595 ms
Total hot run time: 48769 ms

doris-robot · 2025-12-16T06:19:45Z

TPC-DS: Total hot run time: 177868 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 97b92434f2c898d0f48eea9333e8b4f81bdde4ce, data reload: false
query5	4834	626	477	477
query6	329	228	207	207
query7	4223	478	282	282
query8	331	266	256	256
query9	8805	2559	2594	2559
query10	520	398	337	337
query11	15111	15180	14535	14535
query12	183	118	115	115
query13	1268	497	389	389
query14	5894	3283	2986	2986
query14_1	2868	2894	2915	2894
query15	206	192	182	182
query16	859	478	464	464
query17	1129	726	609	609
query18	2543	454	354	354
query19	230	242	216	216
query20	120	115	112	112
query21	231	136	114	114
query22	3942	4084	3869	3869
query23	16761	16247	15821	15821
query23_1	15995	16128	15960	15960
query24	7381	1662	1244	1244
query24_1	1245	1254	1240	1240
query25	594	498	464	464
query26	1262	278	167	167
query27	2750	470	326	326
query28	4459	2156	2151	2151
query29	836	577	479	479
query30	315	246	221	221
query31	851	700	624	624
query32	88	69	72	69
query33	553	353	340	340
query34	890	911	535	535
query35	756	814	731	731
query36	880	889	824	824
query37	131	94	82	82
query38	2839	2812	2784	2784
query39	766	733	871	733
query39_1	692	695	699	695
query40	222	134	118	118
query41	66	62	63	62
query42	106	102	102	102
query43	415	435	422	422
query44	1308	756	747	747
query45	198	194	181	181
query46	867	967	617	617
query47	1611	1689	1607	1607
query48	313	323	255	255
query49	628	422	353	353
query50	664	285	227	227
query51	3781	4073	3749	3749
query52	106	113	98	98
query53	315	345	294	294
query54	295	261	243	243
query55	77	75	69	69
query56	281	291	305	291
query57	1132	1142	1078	1078
query58	302	253	266	253
query59	2455	2533	2385	2385
query60	328	304	290	290
query61	167	159	155	155
query62	685	672	611	611
query63	330	290	299	290
query64	4983	1319	997	997
query65	3982	3942	3962	3942
query66	1423	450	315	315
query67	15390	14899	14632	14632
query68	8382	1018	724	724
query69	513	342	301	301
query70	1074	981	941	941
query71	372	313	291	291
query72	6051	4882	4908	4882
query73	666	575	318	318
query74	8856	8706	8543	8543
query75	3149	3119	2748	2748
query76	3967	1154	745	745
query77	541	408	282	282
query78	9546	9707	8942	8942
query79	1669	885	641	641
query80	741	663	561	561
query81	514	273	233	233
query82	204	135	111	111
query83	284	258	237	237
query84	259	127	106	106
query85	905	509	464	464
query86	386	291	276	276
query87	3060	3072	3010	3010
query88	3216	2322	2322	2322
query89	472	413	386	386
query90	2132	158	157	157
query91	177	165	143	143
query92	80	69	63	63
query93	1240	902	568	568
query94	475	305	315	305
query95	599	380	308	308
query96	594	467	212	212
query97	2291	2313	2262	2262
query98	216	200	203	200
query99	1301	1290	1217	1217
Total cold run time: 260081 ms
Total hot run time: 177868 ms

doris-robot · 2025-12-16T06:24:56Z

ClickBench: Total hot run time: 28.43 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 97b92434f2c898d0f48eea9333e8b4f81bdde4ce, data reload: false
query1	0.05	0.05	0.05
query2	0.15	0.07	0.07
query3	0.38	0.08	0.08
query4	1.61	0.10	0.10
query5	0.26	0.25	0.25
query6	1.17	0.63	0.65
query7	0.04	0.03	0.02
query8	0.07	0.05	0.06
query9	0.60	0.51	0.52
query10	0.56	0.58	0.55
query11	0.26	0.14	0.13
query12	0.26	0.14	0.14
query13	0.63	0.63	0.61
query14	1.02	0.99	1.01
query15	0.90	0.80	0.81
query16	0.39	0.39	0.39
query17	0.98	1.06	1.00
query18	0.24	0.22	0.23
query19	1.85	1.88	1.86
query20	0.02	0.01	0.02
query21	15.39	0.28	0.23
query22	4.96	0.10	0.10
query23	15.43	0.38	0.22
query24	2.43	0.47	0.29
query25	0.10	0.09	0.09
query26	0.19	0.17	0.19
query27	0.10	0.09	0.09
query28	3.89	1.34	1.16
query29	12.59	4.06	3.30
query30	0.34	0.14	0.12
query31	2.81	0.66	0.42
query32	3.24	0.60	0.48
query33	2.95	3.03	3.01
query34	16.89	5.19	4.68
query35	4.65	4.62	4.65
query36	0.62	0.49	0.48
query37	0.24	0.09	0.09
query38	0.20	0.06	0.06
query39	0.07	0.05	0.05
query40	0.20	0.17	0.18
query41	0.13	0.07	0.05
query42	0.08	0.06	0.04
query43	0.06	0.05	0.05
Total cold run time: 99 s
Total hot run time: 28.43 s

hello-stephen · 2025-12-16T06:40:20Z

BE UT Coverage Report

Increment line coverage 89.66% (26/29) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.43% (18846/35273)
Line Coverage	39.21% (174429/444848)
Region Coverage	33.84% (135000/398983)
Branch Coverage	34.78% (58098/167033)

hello-stephen · 2026-01-07T06:36:43Z

BE UT Coverage Report

Increment line coverage 89.66% (26/29) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.27% (18988/35646)
Line Coverage	39.22% (175875/448454)
Region Coverage	33.76% (136212/403468)
Branch Coverage	34.76% (58819/169232)

hello-stephen · 2026-01-07T08:09:09Z

BE Regression && UT Coverage Report

Increment line coverage 96.55% (28/29) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.87% (25742/34846)
Line Coverage	61.20% (273696/447235)
Region Coverage	56.10% (228646/407587)
Branch Coverage	57.96% (98403/169781)

github-actions · 2026-01-07T08:31:31Z

PR approved by at least one committer and no changes requested.

…che#58459) Add a `RECYCLE` state for rowset/meta (rs meta) and update the recycler logic to mark metadata as `RECYCLE` before final deletion. This reduces the risk of accidental data loss. ## Problem The recycler sometimes deletes rs meta too early (race conditions, restarts, or recovery cases), which can cause metadata and file inconsistencies or data loss. ## Solution - Introduce a `RECYCLE` intermediate state for rs meta. - When an item is chosen for cleanup, mark it `RECYCLE` and record a timestamp. - Only perform the final delete after a confirmation window or additional checks. - Make recovery/restart logic treat `RECYCLE` items as recoverable until final deletion. ## Main changes - Add `RECYCLE` to the rs meta state enum. - Update metadata APIs to set/query `RECYCLE`. - Update recycler to use two-step deletion: ***mark -> confirm -> abort txn/job and delete***. - Add logs and tests for the new flow. ## Test case ``` 1. begin_txn -> prepare_rowset -> force_recycle -> commit_rowset -> commit_txn 2. start_job -> prepare_rowset -> force_recycle -> commit_rowset -> finish_job Rowset will be marked as recycled to prevent commit_rowset and finish job/txn 3. begin_txn -> prepare_rowset -> commit_rowset -> force_recycle -> commit_txn 4. start_job -> prepare_rowset -> commit_rowset -> force_recycle -> finish_job Rowset will be marked as recycled to prevent finish job/txn 5. begin_txn -> prepare_rowset -> force_recycle * 2 -> commit_rowset -> commit_txn 6. start_job -> prepare_rowset -> force_recycle * 2 -> commit_rowset -> finish_job 7. begin_txn -> prepare_rowset -> commit_rowset -> force_recycle * 2 -> commit_txn 9. start_job -> prepare_rowset -> commit_rowset -> force_recycle * 2 -> finish_job 10. delete_job -> commit_rowset -> force_recycle * 2 -> finish_job 11. delete_job -> prepare_rowset -> commit_rowset -> force_recycle * 2 -> finish_job 12. delete_job -> prepare_rowset -> force_recycle * 2 -> commit_rowset -> finish_job Double recycle job will mark rowset as recycled and abort job/txn, then delete data and kv ```

…ta loss #58459 (#59765) pick: #58459

…che#58459) Add a `RECYCLE` state for rowset/meta (rs meta) and update the recycler logic to mark metadata as `RECYCLE` before final deletion. This reduces the risk of accidental data loss. ## Problem The recycler sometimes deletes rs meta too early (race conditions, restarts, or recovery cases), which can cause metadata and file inconsistencies or data loss. ## Solution - Introduce a `RECYCLE` intermediate state for rs meta. - When an item is chosen for cleanup, mark it `RECYCLE` and record a timestamp. - Only perform the final delete after a confirmation window or additional checks. - Make recovery/restart logic treat `RECYCLE` items as recoverable until final deletion. ## Main changes - Add `RECYCLE` to the rs meta state enum. - Update metadata APIs to set/query `RECYCLE`. - Update recycler to use two-step deletion: ***mark -> confirm -> abort txn/job and delete***. - Add logs and tests for the new flow. ## Test case ``` 1. begin_txn -> prepare_rowset -> force_recycle -> commit_rowset -> commit_txn 2. start_job -> prepare_rowset -> force_recycle -> commit_rowset -> finish_job Rowset will be marked as recycled to prevent commit_rowset and finish job/txn 3. begin_txn -> prepare_rowset -> commit_rowset -> force_recycle -> commit_txn 4. start_job -> prepare_rowset -> commit_rowset -> force_recycle -> finish_job Rowset will be marked as recycled to prevent finish job/txn 5. begin_txn -> prepare_rowset -> force_recycle * 2 -> commit_rowset -> commit_txn 6. start_job -> prepare_rowset -> force_recycle * 2 -> commit_rowset -> finish_job 7. begin_txn -> prepare_rowset -> commit_rowset -> force_recycle * 2 -> commit_txn 9. start_job -> prepare_rowset -> commit_rowset -> force_recycle * 2 -> finish_job 10. delete_job -> commit_rowset -> force_recycle * 2 -> finish_job 11. delete_job -> prepare_rowset -> commit_rowset -> force_recycle * 2 -> finish_job 12. delete_job -> prepare_rowset -> force_recycle * 2 -> commit_rowset -> finish_job Double recycle job will mark rowset as recycled and abort job/txn, then delete data and kv ```

…ta loss apache#58459 (apache#59765) pick: apache#58459

when enable_recycle_delete_rowset_key_check is false, idempotent check is invalid bug pr related: #58459

when enable_recycle_delete_rowset_key_check is false, idempotent check is invalid bug pr related: apache#58459

…ally reading key (#62476) fix: #58459 This PR reduces point-read overhead in Cloud recycler rowset cleanup. Previously, each scanned rowset could immediately trigger metadata reads/writes to mark it as recycled or abort its related transaction/job. The new flow records rowset keys during scanning, then batch-processes recycled marks and deferred abort tasks in worker batches. Prepare rowset deletion is also deferred so the recycler re-reads the latest metadata before deleting data. This keeps the existing recycle safety semantics while reducing per-rowset KV operations during large recycle scans. **Release mode test** Recycling 10,000 rowsets, after enabling `enable_mark_delete_rowset_before_recycle` and `enable_abort_txn_and_job_for_delete_rowset_before_recycle`, the processing time increased by approximately 10%. **3514 ms -> 175 ms(mark) + 3811(abort and recycle)**

gavinchou reviewed Dec 3, 2025
View reviewed changes

Comment threadcloud/src/recycler/recycler.cpp

gavinchou reviewed Dec 3, 2025
View reviewed changes

Comment threadcloud/src/meta-service/meta_service.cpp Outdated

gavinchou reviewed Dec 3, 2025
View reviewed changes

Comment threadcloud/src/meta-service/meta_service.cpp Outdated

wyxxxcat force-pushed the recycle_rowsets_loss branch 8 times, most recently from a97bb6b to 331af0cCompare December 8, 2025 08:10

wyxxxcat marked this pull request as ready for review December 8, 2025 08:10

wyxxxcat requested review from dataroaring, w41ter and yiguolei as code owners December 8, 2025 08:10

wyxxxcat force-pushed the recycle_rowsets_loss branch 4 times, most recently from 1914e9e to ee7e572Compare December 9, 2025 07:13

wyxxxcat marked this pull request as draft December 9, 2025 07:13

wyxxxcat force-pushed the recycle_rowsets_loss branch from ee7e572 to 97b9243Compare December 16, 2025 02:57

wyxxxcat marked this pull request as ready for review December 16, 2025 02:58

gavinchou approved these changes Jan 7, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jan 7, 2026

0AyanamiRei approved these changes Jan 7, 2026
View reviewed changes

gavinchou merged commit e3fc71b into apache:masterJan 8, 2026
26 of 29 checks passed

github-actionsBot added dev/3.1.x-conflict dev/4.0.x-conflict labels Jan 8, 2026

wyxxxcat mentioned this pull request Jan 12, 2026
branch-4.0: [fix](recycler) Add recycle state for rs meta to avoid data loss #58459 #59765
Merged

yiguolei pushed a commit that referenced this pull request Jan 13, 2026
branch-4.0: [fix](recycler) Add recycle state for rs meta to avoid da…
a1ceaf0
…ta loss #58459 (#59765) pick: #58459

yiguolei added dev/4.0.3-merged and removed dev/4.0.x dev/4.0.x-conflict labels Jan 13, 2026

ybtsdst pushed a commit to ybtsdst/doris that referenced this pull request Feb 27, 2026
branch-4.0: [fix](recycler) Add recycle state for rs meta to avoid da…
2f47698
…ta loss apache#58459 (apache#59765) pick: apache#58459

gavinchou added int/3.1.5-merged and removed dev/3.0.x dev/3.1.x dev/3.1.x-conflict labels Mar 16, 2026

wyxxxcat mentioned this pull request Mar 17, 2026
[fix](ms) Add job check for stale commit rowset #61427
Merged
16 tasks

gavinchou pushed a commit that referenced this pull request Mar 24, 2026
[fix](ms) Add job check for stale commit rowset (#61427)
528c7a2
when enable_recycle_delete_rowset_key_check is false, idempotent check is invalid bug pr related: #58459

github-actionsBot pushed a commit that referenced this pull request Mar 24, 2026
[fix](ms) Add job check for stale commit rowset (#61427)
3f96b19
when enable_recycle_delete_rowset_key_check is false, idempotent check is invalid bug pr related: #58459

github-actionsBot pushed a commit that referenced this pull request Mar 24, 2026
[fix](ms) Add job check for stale commit rowset (#61427)
89ffa0c
when enable_recycle_delete_rowset_key_check is false, idempotent check is invalid bug pr related: #58459

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Mar 31, 2026
[fix](ms) Add job check for stale commit rowset (apache#61427)
89fb402
when enable_recycle_delete_rowset_key_check is false, idempotent check is invalid bug pr related: apache#58459

wyxxxcat mentioned this pull request Apr 14, 2026
[improvement](recycler) Avoid single-point read/write during sequentially reading key #62476
Merged
16 tasks

Navigation Menu

Uh oh!

[fix](recycler) Add recycle state for rs meta to avoid data loss - #58459

[fix](recycler) Add recycle state for rs meta to avoid data loss#58459
gavinchou merged 4 commits into
apache:masterfrom
wyxxxcat:recycle_rowsets_loss

wyxxxcat commented Nov 27, 2025•
edited
Loading

Uh oh!

hello-stephen commented Nov 27, 2025

Uh oh!

gavinchouDec 3, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wyxxxcat commented Dec 16, 2025

Uh oh!

hello-stephen commented Dec 16, 2025

Uh oh!

doris-robot commented Dec 16, 2025

Uh oh!

doris-robot commented Dec 16, 2025

Uh oh!

doris-robot commented Dec 16, 2025

Uh oh!

hello-stephen commented Dec 16, 2025

Uh oh!

hello-stephen commented Jan 7, 2026

Uh oh!

hello-stephen commented Jan 7, 2026

Uh oh!

github-actionsBot commented Jan 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Uh oh!

Conversation

wyxxxcat commented Nov 27, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Summary

Problem

Solution

Main changes

Test case

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Nov 27, 2025

Uh oh!

gavinchouDec 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wyxxxcat commented Dec 16, 2025

Uh oh!

hello-stephen commented Dec 16, 2025

FE UT Coverage Report

Uh oh!

doris-robot commented Dec 16, 2025

Uh oh!

doris-robot commented Dec 16, 2025

Uh oh!

doris-robot commented Dec 16, 2025

Uh oh!

hello-stephen commented Dec 16, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Jan 7, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jan 7, 2026

BE Regression && UT Coverage Report

Uh oh!

github-actionsBot commented Jan 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

wyxxxcat commented Nov 27, 2025•
edited
Loading