[fix](compaction) Make creating tablet idempotently to keep compaction Job idempotent by wyxxxcat · Pull Request #56061 · apache/doris

wyxxxcat · 2025-09-15T07:37:33Z

What problem does this PR solve?

If rpc create_tablets is not idempotent, it will recreate tablet stats, which is tragic for compaction task idempotence

if (compaction.base_compaction_cnt() < stats.base_compaction_cnt() ||
compaction.cumulative_compaction_cnt() < stats.cumulative_compaction_cnt()) {
code = MetaServiceCode::STALE_TABLET_CACHE;
SS << "could not perform compaction on expired tablet cache."
<< " req_base_compaction_cnt=" << compaction.base_compaction_cnt()
<< ", base_compaction_cnt=" << stats.base_compaction_cnt()
<< ", req_cumulative_compaction_cnt=" << compaction.cumulative_compaction_cnt()
<< ", cumulative_compaction_cnt=" << stats.cumulative_compaction_cnt();
msg = ss.str();
return;
}

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-09-15T07:37:42Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

wyxxxcat · 2025-09-15T07:44:55Z

run buildall

wyxxxcat · 2025-09-15T10:43:05Z

run buildall

hello-stephen · 2025-09-15T11:01:40Z

Cloud UT Coverage Report

Increment line coverage 87.50% (28/32) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	84.58% (1536/1816)
Line Coverage	68.30% (27657/40493)
Region Coverage	69.03% (13626/19740)
Branch Coverage	59.16% (7282/12310)

wyxxxcat · 2025-09-15T12:34:02Z

run buildall

wyxxxcat · 2025-09-15T13:34:52Z

run buildall

doris-robot · 2025-09-15T13:55:41Z

Cloud UT Coverage Report

Increment line coverage 87.88% (29/33) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	84.01% (1534/1826)
Line Coverage	68.08% (27542/40457)
Region Coverage	68.56% (13564/19783)
Branch Coverage	58.78% (7241/12318)

wyxxxcat · 2025-09-15T16:34:37Z

run buildall

doris-robot · 2025-09-15T16:47:57Z

Cloud UT Coverage Report

Increment line coverage 88.24% (30/34) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	84.01% (1534/1826)
Line Coverage	68.12% (27558/40458)
Region Coverage	68.56% (13564/19783)
Branch Coverage	58.87% (7252/12318)

github-actions · 2025-09-16T06:15:26Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-09-16T06:15:28Z

PR approved by anyone and no changes requested.

dataroaring

LGTM

…n Job idempotent (apache#56061) If rpc create_tablets is not idempotent, it will recreate tablet stats, which is tragic for compaction task idempotence ``` if (compaction.base_compaction_cnt() < stats.base_compaction_cnt() || compaction.cumulative_compaction_cnt() < stats.cumulative_compaction_cnt()) { code = MetaServiceCode::STALE_TABLET_CACHE; SS << "could not perform compaction on expired tablet cache." << " req_base_compaction_cnt=" << compaction.base_compaction_cnt() << ", base_compaction_cnt=" << stats.base_compaction_cnt() << ", req_cumulative_compaction_cnt=" << compaction.cumulative_compaction_cnt() << ", cumulative_compaction_cnt=" << stats.cumulative_compaction_cnt(); msg = ss.str(); return; } ```

…ep compaction Job idempotent #56061 (#56108) picked from #56061

…n Job idempotent (apache#56061) If rpc create_tablets is not idempotent, it will recreate tablet stats, which is tragic for compaction task idempotence ``` if (compaction.base_compaction_cnt() < stats.base_compaction_cnt() || compaction.cumulative_compaction_cnt() < stats.cumulative_compaction_cnt()) { code = MetaServiceCode::STALE_TABLET_CACHE; SS << "could not perform compaction on expired tablet cache." << " req_base_compaction_cnt=" << compaction.base_compaction_cnt() << ", base_compaction_cnt=" << stats.base_compaction_cnt() << ", req_cumulative_compaction_cnt=" << compaction.cumulative_compaction_cnt() << ", cumulative_compaction_cnt=" << stats.cumulative_compaction_cnt(); msg = ss.str(); return; } ```

…mpotently to keep compaction Job idempotent (apache#4688) pick: apache#56061

Related PR: #56061 Problem Summary: ShortGetTabletStatsDebugStringTest enables the global idempotent request injection configuration but does not restore it after completion. Because meta_service_http_test.cpp and meta_service_test.cpp are linked into the same test binary, later tests may create delayed background bthreads. Those tasks capture MetaServiceProxy through a raw this pointer and can access impl_ after the proxy is destroyed, causing intermittent heap-use-after-free, heap-buffer-overflow, or SEGV failures. Remove the test to prevent this test-binary-wide configuration pollution and the resulting flaky failures.

wyxxxcat requested review from dataroaring, gavinchou and w41ter as code owners September 15, 2025 07:37

gavinchou reviewed Sep 15, 2025
View reviewed changes

Comment threadcloud/src/meta-service/meta_service.cpp Outdated

gavinchou reviewed Sep 15, 2025
View reviewed changes

Comment threadcloud/src/meta-service/meta_service.cpp Outdated

gavinchou reviewed Sep 15, 2025
View reviewed changes

Comment threadcloud/src/meta-service/meta_service.cpp

wyxxxcat force-pushed the compaction_idempotent branch from fa263f4 to 7505926Compare September 15, 2025 08:50

dataroaring added dev/3.0.x dev/3.1.x p0_b labels Sep 15, 2025

wyxxxcat force-pushed the compaction_idempotent branch 2 times, most recently from 9fbdea8 to 0a028caCompare September 15, 2025 10:42

wyxxxcat force-pushed the compaction_idempotent branch 2 times, most recently from 7a8dd26 to c236e70Compare September 15, 2025 12:32

wyxxxcat force-pushed the compaction_idempotent branch from c236e70 to e660827Compare September 15, 2025 13:34

gavinchou reviewed Sep 15, 2025
View reviewed changes

Comment threadcloud/src/meta-service/meta_service.cpp

gavinchou reviewed Sep 15, 2025
View reviewed changes

Comment threadcloud/src/meta-service/meta_service.h

1
748c320

wyxxxcat force-pushed the compaction_idempotent branch from e660827 to 748c320Compare September 15, 2025 16:34

gavinchou approved these changes Sep 16, 2025
View reviewed changes

gavinchou added the dev/4.0.x label Sep 16, 2025

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 16, 2025

github-actionsBot added the reviewed label Sep 16, 2025

dataroaring approved these changes Sep 16, 2025
View reviewed changes

dataroaring merged commit bef8f36 into apache:masterSep 16, 2025
33 of 35 checks passed

github-actionsBot added dev/3.0.x-conflict dev/3.1.x-conflict labels Sep 16, 2025

This was referenced Sep 16, 2025
[branch-3.0] [fix](compaction) Make creating tablet idempotently to keep compaction Job idempotent #56061 #56106
Closed
branch-3.1: [fix](compaction) Make creating tablet idempotently to keep compaction Job idempotent #56061 #56108
Merged

morrySnow pushed a commit that referenced this pull request Sep 17, 2025
branch-3.1: [fix](compaction) Make creating tablet idempotently to ke…
b57b405
…ep compaction Job idempotent #56061 (#56108) picked from #56061

morrySnow added dev/3.1.1-merged and removed dev/3.1.x dev/3.1.x-conflict labels Sep 17, 2025

morrySnow mentioned this pull request Sep 22, 2025
Release Note 3.1.1 #56289
Closed

yiguolei pushed a commit to yiguolei/incubator-doris that referenced this pull request Sep 28, 2025
branch-selectdb-doris-3.0: [fix](compaction) Make creating tablet ide…
a8b0f93
…mpotently to keep compaction Job idempotent (apache#4688) pick: apache#56061

yiguolei removed the dev/4.0.x label Sep 29, 2025

wyxxxcat mentioned this pull request Aug 13, 2026
[fix](cloud) Remove flaky idempotent injection debug test #66726
Merged
16 tasks

bobhan1 mentioned this pull request Aug 13, 2026
[fix](cloud) Restore idempotent injection config after test #66727
Closed
7 tasks

Uh oh!

Conversation

wyxxxcat commented Sep 15, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Sep 15, 2025

Uh oh!

wyxxxcat commented Sep 15, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wyxxxcat commented Sep 15, 2025

Uh oh!

hello-stephen commented Sep 15, 2025

Cloud UT Coverage Report

Uh oh!

wyxxxcat commented Sep 15, 2025

Uh oh!

wyxxxcat commented Sep 15, 2025

Uh oh!

doris-robot commented Sep 15, 2025

Cloud UT Coverage Report

Uh oh!

Uh oh!

Uh oh!

wyxxxcat commented Sep 15, 2025

Uh oh!

doris-robot commented Sep 15, 2025

Cloud UT Coverage Report

Uh oh!

github-actionsBot commented Sep 16, 2025

Uh oh!

github-actionsBot commented Sep 16, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

wyxxxcat commented Sep 15, 2025•
edited
Loading