[Enhancement](compaction) Do not set failure time when cumulative compaction dealing with delete rowset by Yukang-Lian · Pull Request #43466 · apache/doris

Yukang-Lian · 2024-11-07T15:00:06Z

What problem does this PR solve?

Problem Summary:

Before this PR, in cases where there is an alternating distribution of data rowset -> delete rowset -> data rowset -> delete rowset, cumulative compaction would only move the cumulative point forward to allow base compaction to handle the delete rowset. Cumulative compaction itself would not process the data and would return be marked as failure. This would cause the compaction submission task process to pause for 5 seconds, impacting efficiency.

This PR modifies the return status to OK for such cases, which improves the efficiency of the compaction submission task.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-11-07T15:00:11Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Yukang-Lian · 2024-11-07T15:00:16Z

run buildall

github-actions · 2024-11-07T15:04:48Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-11-07T16:58:16Z

TeamCity be ut coverage result:
Function Coverage: 37.90% (9864/26026)
Line Coverage: 29.07% (82064/282331)
Region Coverage: 28.28% (42257/149422)
Branch Coverage: 24.85% (21432/86240)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b8742bb3c57c6fea9325807e086a313a66c73f75_b8742bb3c57c6fea9325807e086a313a66c73f75/report/index.html

github-actions · 2024-11-10T15:50:52Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-11-10T15:50:54Z

PR approved by anyone and no changes requested.

Yukang-Lian · 2024-11-11T16:18:01Z

run cloud_p0

github-actions · 2024-11-12T08:30:18Z

clang-tidy review says "All clean, LGTM! 👍"

Yukang-Lian · 2024-11-12T13:09:14Z

run buildall

Yukang-Lian · 2024-11-12T13:32:55Z

run beut

Yukang-Lian · 2024-11-27T03:00:32Z

run buildall

doris-robot · 2024-11-27T03:40:27Z

TeamCity be ut coverage result:
Function Coverage: 38.34% (9975/26016)
Line Coverage: 29.43% (83466/283610)
Region Coverage: 28.58% (42969/150337)
Branch Coverage: 25.19% (21828/86658)
Coverage Report: http://coverage.selectdb-in.cc/coverage/899ea902c8c49e92a8488e7b524fdcc0febd1afe_899ea902c8c49e92a8488e7b524fdcc0febd1afe/report/index.html

Yukang-Lian · 2024-12-05T07:14:20Z

run buildall

Yukang-Lian · 2024-12-05T13:07:16Z

run buildall

doris-robot · 2024-12-05T14:28:11Z

TeamCity be ut coverage result:
Function Coverage: 38.48% (10006/26003)
Line Coverage: 29.50% (83881/284316)
Region Coverage: 28.61% (43115/150700)
Branch Coverage: 25.20% (21915/86958)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d350da140c018b64820d262f6abda8f96642e7fc_d350da140c018b64820d262f6abda8f96642e7fc/report/index.html

Yukang-Lian · 2024-12-05T18:36:40Z

run buildall

doris-robot · 2024-12-05T19:04:59Z

TeamCity be ut coverage result:
Function Coverage: 38.48% (10007/26003)
Line Coverage: 29.50% (83885/284315)
Region Coverage: 28.61% (43118/150700)
Branch Coverage: 25.21% (21918/86958)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4407fc4f8facf3f771f017231a69aa110d183a8a_4407fc4f8facf3f771f017231a69aa110d183a8a/report/index.html

Yukang-Lian · 2024-12-06T02:18:24Z

run buildall

doris-robot · 2024-12-06T04:22:45Z

TeamCity be ut coverage result:
Function Coverage: 38.48% (10007/26003)
Line Coverage: 29.51% (83894/284318)
Region Coverage: 28.62% (43128/150700)
Branch Coverage: 25.21% (21918/86958)
Coverage Report: http://coverage.selectdb-in.cc/coverage/fcd8adb6e72de7e4d73b02f951238beba4e06592_fcd8adb6e72de7e4d73b02f951238beba4e06592/report/index.html

Yukang-Lian · 2024-12-10T13:24:34Z

run buildall

github-actions · 2024-12-10T13:28:55Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-12-10T15:14:33Z

TeamCity be ut coverage result:
Function Coverage: 38.80% (10106/26046)
Line Coverage: 29.70% (84750/285321)
Region Coverage: 28.77% (43502/151216)
Branch Coverage: 25.32% (22093/87272)
Coverage Report: http://coverage.selectdb-in.cc/coverage/abb7d02432ff6a3e030f6b89f7d08115e5765ccf_abb7d02432ff6a3e030f6b89f7d08115e5765ccf/report/index.html

Yukang-Lian · 2024-12-11T14:47:11Z

run buildall

github-actions · 2024-12-11T14:51:31Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-12-11T16:45:04Z

TeamCity be ut coverage result:
Function Coverage: 38.75% (10103/26073)
Line Coverage: 29.68% (84735/285540)
Region Coverage: 28.74% (43501/151335)
Branch Coverage: 25.30% (22100/87338)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5e9fc93c7d52f682ed437443f445662727caa8b1_5e9fc93c7d52f682ed437443f445662727caa8b1/report/index.html

github-actions · 2024-12-12T02:32:17Z

PR approved by at least one committer and no changes requested.

luwei16

Continuous deletes (without inserts) will cause a 5-second interval for tablet compaction?

…paction dealing with delete rowset (#43466) Before this PR, in cases where there is an alternating distribution of data rowset -> delete rowset -> data rowset -> delete rowset, cumulative compaction would only move the cumulative point forward to allow base compaction to handle the delete rowset. Cumulative compaction itself would not process the data and would return be marked as failure. This would cause the compaction submission task process to pause for 5 seconds, impacting efficiency. This PR modifies the return status to OK for such cases, which improves the efficiency of the compaction submission task.

…tive compaction dealing with delete rowset (apache#43466)" This reverts commit 55f450c.

…mulative compaction dealing with delete rowset #43466 (#45404) Cherry-picked from #43466 Co-authored-by: abmdocrt <lianyukang@selectdb.com>

…paction dealing with delete rowset (apache#43466) Before this PR, in cases where there is an alternating distribution of data rowset -> delete rowset -> data rowset -> delete rowset, cumulative compaction would only move the cumulative point forward to allow base compaction to handle the delete rowset. Cumulative compaction itself would not process the data and would return be marked as failure. This would cause the compaction submission task process to pause for 5 seconds, impacting efficiency. This PR modifies the return status to OK for such cases, which improves the efficiency of the compaction submission task.

…failure time when cumulative compaction dealing with delete rowset (#43466)" (#46117) Before this PR, in cases where there is an alternating distribution of data rowset -> delete rowset -> data rowset -> delete rowset, cumulative compaction would only move the cumulative point forward to allow base compaction to handle the delete rowset. Cumulative compaction itself would not process the data and would return be marked as failure. This would cause the compaction submission task process to pause for 5 seconds, impacting efficiency. This PR modifies the return status to OK for such cases, which improves the efficiency of the compaction submission task.

luwei16

Use error codes to distinguish failure, no suite version, and delete.

…paction dealing with delete rowset (apache#43466) Before this PR, in cases where there is an alternating distribution of data rowset -> delete rowset -> data rowset -> delete rowset, cumulative compaction would only move the cumulative point forward to allow base compaction to handle the delete rowset. Cumulative compaction itself would not process the data and would return be marked as failure. This would cause the compaction submission task process to pause for 5 seconds, impacting efficiency. This PR modifies the return status to OK for such cases, which improves the efficiency of the compaction submission task.

gavinchou previously approved these changes Nov 10, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Nov 10, 2024

github-actionsBot added the reviewed label Nov 10, 2024

gavinchou added the dev/3.0.x label Nov 12, 2024

Yukang-Lian dismissed gavinchou’s stale review via b17d93cNovember 12, 2024 08:25

Yukang-Lian force-pushed the Enhance-Delete-Compaction branch from b8742bb to b17d93cCompare November 12, 2024 08:25

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Nov 12, 2024

Yukang-Lian force-pushed the Enhance-Delete-Compaction branch from b17d93c to 899ea90Compare November 27, 2024 03:00

Yukang-Lian changed the title ~~[Enhancement](compaction) set status to ok for alternating rowsets compaction~~[Enhancement](compaction) Do not set failure time when cumulative compaction dealing with delete rowsetNov 27, 2024

Yukang-Lian force-pushed the Enhance-Delete-Compaction branch 2 times, most recently from 96ab537 to 75d9dbcCompare December 4, 2024 15:43

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Dec 10, 2024

11
5e9fc93

gavinchou approved these changes Dec 12, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Dec 12, 2024

luwei16 reviewed Dec 12, 2024
View reviewed changes

luwei16 approved these changes Dec 13, 2024
View reviewed changes

gavinchou merged commit 55f450c into apache:masterDec 13, 2024

github-actionsBot mentioned this pull request Dec 13, 2024
branch-3.0: [Enhancement](compaction) Do not set failure time when cumulative compaction dealing with delete rowset #43466 #45404
Merged

github-actionsBot added the dev/2.1.x-conflict label Dec 13, 2024

mymeiyi added a commit to mymeiyi/doris that referenced this pull request Dec 16, 2024
Revert "[Enhancement](compaction) Do not set failure time when cumula…
7547650
…tive compaction dealing with delete rowset (apache#43466)" This reverts commit 55f450c.

gavinchou pushed a commit that referenced this pull request Dec 17, 2024
branch-3.0: [Enhancement](compaction) Do not set failure time when cu…
0d7d700
…mulative compaction dealing with delete rowset #43466 (#45404) Cherry-picked from #43466 Co-authored-by: abmdocrt <lianyukang@selectdb.com>

wm1581066 added dev/3.0.4-merged and removed dev/3.0.x labels Dec 17, 2024

Yukang-Lian mentioned this pull request Dec 27, 2024
[Cherry-pick](branch-2.1) Pick "[Enhancement](compaction) Do not set failure time when cumulative compaction dealing with delete rowset (#43466)" #46117
Merged
16 tasks

wm1581066 added dev/2.1.8-merged and removed dev/2.1.x labels Dec 30, 2024

gavinchou mentioned this pull request Feb 18, 2025
Release Notes 3.0.4 #48013
Closed

shuke987 removed the dev/2.1.x-conflict label Mar 6, 2025

luwei16 reviewed May 16, 2025
View reviewed changes

Uh oh!

Conversation

Yukang-Lian commented Nov 7, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

doris-robot commented Nov 7, 2024

Uh oh!

Yukang-Lian commented Nov 7, 2024

Uh oh!

github-actionsBot commented Nov 7, 2024

Uh oh!

doris-robot commented Nov 7, 2024

Uh oh!

github-actionsBot commented Nov 10, 2024

Uh oh!

github-actionsBot commented Nov 10, 2024

Uh oh!

Yukang-Lian commented Nov 11, 2024

Uh oh!

github-actionsBot commented Nov 12, 2024

Uh oh!

Yukang-Lian commented Nov 12, 2024

Uh oh!

Yukang-Lian commented Nov 12, 2024

Uh oh!

Yukang-Lian commented Nov 27, 2024

Uh oh!

doris-robot commented Nov 27, 2024

Uh oh!

Yukang-Lian commented Dec 5, 2024

Uh oh!

Yukang-Lian commented Dec 5, 2024

Uh oh!

doris-robot commented Dec 5, 2024

Uh oh!

Yukang-Lian commented Dec 5, 2024

Uh oh!

doris-robot commented Dec 5, 2024

Uh oh!

Yukang-Lian commented Dec 6, 2024

Uh oh!

doris-robot commented Dec 6, 2024

Uh oh!

Yukang-Lian commented Dec 10, 2024

Uh oh!

github-actionsBot commented Dec 10, 2024

Uh oh!

doris-robot commented Dec 10, 2024

Uh oh!

Yukang-Lian commented Dec 11, 2024

Uh oh!

github-actionsBot commented Dec 11, 2024

Uh oh!

doris-robot commented Dec 11, 2024

Uh oh!

github-actionsBot commented Dec 12, 2024

Uh oh!

luwei16 left a comment

Choose a reason for hiding this comment

Uh oh!

luwei16 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Yukang-Lian commented Nov 7, 2024•
edited
Loading