Skip to content

branch-4.1: [fix](cloud) Correct cumulative point advancement when enable parallel cumulative compaction (#65915) - #67425

Open
mymeiyi wants to merge 1 commit into
apache:branch-4.1from
mymeiyi:branch-4.1-pick-65915
Open

branch-4.1: [fix](cloud) Correct cumulative point advancement when enable parallel cumulative compaction (#65915)#67425
mymeiyi wants to merge 1 commit into
apache:branch-4.1from
mymeiyi:branch-4.1-pick-65915

Conversation

@mymeiyi

Copy link
Copy Markdown
Contributor

pick #65915

CopilotAI lite review requested due to automatic review settings September 2, 2026 03:47
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@mymeiyi

Copy link
Copy Markdown
ContributorAuthor

run buildall

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The change touches core compaction correctness across BE/meta-service with parallel/concurrent behavior, so it warrants final human verification despite tests and only minor review nits found.

Pull request overview

Backports the cloud compaction fix from #65915 to ensure cumulative_point is advanced safely when parallel cumulative compactions can finish out of order, adding validation/guardrails in meta-service and tightening BE-side application rules with expanded test coverage.

Changes:

  • Add meta-service logic to validate/ignore stale cumulative-point proposals and update tablet stats more defensively during compaction FINISH.
  • Update BE compaction flows to attach compaction counter snapshots and only apply cumulative results when the response counter is the expected “next” value (otherwise force a tablet sync).
  • Add/extend unit tests covering legacy vs current behavior, parallel ordering, counter snapshot handling, and sync-forcing scenarios.
File summaries
FileDescription
cloud/test/meta_service_job_test.cppAdds helpers and multiple regression tests for parallel/legacy cumulative-point advancement and counter validation.
cloud/src/meta-service/meta_service_job.cppIntroduces cumulative-point acceptance logic and adds validation around cumulative counter presence in FINISH.
be/test/storage/compaction/cloud_index_change_compaction_test.cppAdds test ensuring base compaction response with newer cumulative point forces a sync (without mutating local point).
be/test/cloud/cloud_compaction_test.cppAdds comprehensive tests for applying cumulative results, parallel pick behavior, delete handling, and sync triggering.
be/src/storage/compaction/compaction.hDeclares CloudCompactionMixin::should_apply_cumulative_compaction_result (header-lock required).
be/src/storage/compaction/compaction.cppImplements the “apply only next cumulative result, else force sync” gating logic.
be/src/cloud/cloud_index_change_compaction.cppIncludes counter snapshots in jobs and uses the new gating logic for cumulative result application; forces sync on newer point from base response.
be/src/cloud/cloud_cumulative_compaction.hAdds parallel-mode state/snapshots and updates cumulative-point update API.
be/src/cloud/cloud_cumulative_compaction.cppImplements conflict-window-aware picking/point advancement, stable execution mode snapshotting, and counter snapshot propagation.
be/src/cloud/cloud_base_compaction.cppForces sync when meta-service reports a newer cumulative point after base compaction.
Review details
  • Files reviewed: 10/10 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

compaction.type() != TabletCompactionJobPB::EMPTY_CUMULATIVE) {
return true;
}
// Safe because tablet stats keep max(current, proposal).
@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 91.76% (78/85) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage78.35% (1943/2480)
Line Coverage64.92% (34611/53310)
Region Coverage52.48% (33229/63320)
Branch Coverage55.12% (10221/18544)

@mymeiyi

Copy link
Copy Markdown
ContributorAuthor

run buildall

…l cumulative compaction (apache#65915)
When enable parallel cumulative compaction, the cu compaction with higher version range may finish before the cu compaction with lower range.
For example, cu1: v10-v20, cu2: v21-v30. The cu2 may finish before cu1. When cu2 finish, it may set cu point to v31, does not consider if cu1 finish or fail.
This pr correct the cu point advancement.
related to apache#64619
@yiguolei
yiguoleiforce-pushed the branch-4.1-pick-65915 branch from 7e9d4a4 to 62cc246CompareSeptember 5, 2026 22:08
@yiguolei

Copy link
Copy Markdown
Contributor

run buildall

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@mymeiyi@hello-stephen@yiguolei