[opt](recycler) Optimize rowset recycling for recycler

[wyxxxcat]

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

• Test

  • Regression test
  • Unit Test
  • Manual test (add detailed scripts or steps below)
  • No need to test or manual test. Explain why:
    • This is a refactor/code format and no logic has been changed.
    • Previous test can cover this change.
    • No code files have been changed.
    • Other reason

• Behavior changed:

  • No.
  • Yes.

• Does this need documentation?

  • No.
  • Yes.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[wyxxxcat]

run buildall

[hello-stephen]

Cloud UT Coverage Report

Increment line coverage 84.46% (299/354) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	78.14% (1866/2388)
Line Coverage	64.63% (33612/52007)
Region Coverage	65.26% (16640/25497)
Branch Coverage	55.80% (8904/15958)

[github-actions]

PR approved by at least one committer and no changes requested.

[github-actions]

PR approved by anyone and no changes requested.

[wyxxxcat]

run buildall

[wyxxxcat]

/review

[github-actions]

I found blocking issues in the rowset recycler changes.

Critical checkpoints:

• Goal/test coverage: The PR aims to optimize rowset recycling and packed-file cleanup. It adds recycler unit coverage, but the PREPARE rowset abort-before-delete ordering and invalid mutable config values are not covered.
• Scope/focus: The core changes are focused on recycler batching, but the new split mark/abort worker changes behavior in a correctness-sensitive path.
• Concurrency: The new mark/abort pool runs independently from delete jobs, creating an ordering race for prepared rowsets.
• Lifecycle/static initialization: No new static initialization lifecycle issue found.
• Configuration: A mutable batch-size config was added without lower-bound handling at use sites.
• Compatibility/persistence: No storage format or RPC compatibility issue found in the reviewed diff.
• Parallel paths: PREPARE rowsets and normal rowsets now diverge; PREPARE loses the old abort-before-delete sequencing.
• Special checks: Per-tablet skipping is present, but the delete batch size needs the same positive clamp pattern used for per-tablet batch size.
• Tests/results: Added tests cover batching behavior, but not the two failure modes above.
• Observability: Existing logs are mostly sufficient; no separate observability blocker found.
• Transaction/persistence/data writes: The PREPARE path can delete physical data before aborting the owning txn/job, which can cause committed metadata to reference missing files.
• Performance: Non-positive delete batch size can force retaining all scanned rowsets in memory until the scan finishes.

User focus: No additional user-provided review focus was specified.

[wyxxxcat]

run buildall

[wyxxxcat]

/review

[github-actions]

I found one additional blocking issue beyond the already-open inline threads.

Critical checkpoint conclusions:

• Goal and tests: the PR aims to throttle/batch recycle rowset work per tablet and avoid deleting packed slice paths directly. The packed-file path looked consistent, but the rowset batching path still misses an important metadata-only case; tests do not cover a large backlog of empty/metadata-only recycle entries.
• Small/focused change: mostly focused, but the recycle_rowsets control flow now splits mark/abort/delete and batching decisions across multiple queues, which introduced ordering issues already noted in existing threads and the additional flush-threshold gap below.
• Concurrency: new worker pools introduce concurrent mark/abort and delete chains. Existing review threads already cover the abort-before-delete race; no additional distinct thread-safety issue was found.
• Lifecycle/static initialization: no new static initialization or non-obvious object lifetime issue found. Worker pools are stopped before final metric reporting.
• Configuration: new mutable batching configs are added. Existing review thread covers lower-bound clamping for recycle_rowsets_delete_batch_size.
• Compatibility: no wire/storage-format incompatibility found.
• Parallel paths: packed file deletion updates both single and batch delete_rowset_data paths. Rowset batching missed the rowset_keys_without_data path, commented inline.
• Conditional checks: the new loop_done early return only checks rowsets.size(), which is insufficient for metadata-only rows.
• Test coverage: tests cover normal rowset batches and small metadata-only counts, but not the large metadata-only backlog that breaks the memory cap.
• Test results: I did not run tests in this review runner.
• Observability: logs were added for scanning and mark/abort jobs; no separate observability blocker found.
• Transaction/persistence/data correctness: existing threads cover the abort-before-delete ordering risk. The new inline comment covers a resource-control/cleanup batching regression.
• User focus: no additional user-provided review focus was specified.

[wyxxxcat]

run buildall

[hello-stephen]

Cloud UT Coverage Report

Increment line coverage 90.28% (223/247) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	78.39% (1915/2443)
Line Coverage	64.87% (34188/52701)
Region Coverage	65.28% (17575/26922)
Branch Coverage	53.98% (9335/17294)

[wyxxxcat]

/review

[github-actions]

I found one additional blocking issue in the added tests. The existing inline threads already cover the recycle_rowsets abort-before-delete ordering and memory/batch-threshold concerns, so I did not duplicate those comments.

Critical checkpoint conclusions:

• Goal/test coverage: The PR aims to limit/batch cloud recycle_rowsets processing and adds targeted tests, but the new test file currently does not compile because it directly accesses a private helper.
• Scope/focus: The production change is focused on recycler batching, though the added unused private declaration should be cleaned up with the test helper fix.
• Concurrency/lifecycle: The modified recycler path introduces additional worker-pool concurrency; existing review threads already cover the main ordering hazards around mark/abort versus deletion.
• Config: Two mutable configs are added; existing review context already covers the missing lower clamp for recycle_rowsets_delete_batch_size.
• Compatibility/storage/transactions: No new persisted format was identified in this pass; rowset deletion ordering concerns remain covered by existing comments.
• Tests/results: Added tests cover batching scenarios, but one added test currently violates C++ access control and will fail compilation.
• Observability/performance: The batching intent is observable through logs/metrics already in the path; no additional distinct issue found beyond the existing batching-memory comments.

User focus: No additional user-provided review focus was specified.

[liaoxin01]

LGTM

fixed

[wyxxxcat]

4.1/4.0 only merge filter packed slice deletion