[opt](cloud) Make get tablet stats and update delete bitmap update lock be able to be in different fdb txns by bobhan1 · Pull Request #45206 · apache/doris

bobhan1 · 2024-12-09T10:46:34Z

What problem does this PR solve?

#37670 let the meta service return the tablet compaction stats along with the getDeleteBitmapUpdateLockResponse to FE to let the BE know whether it should sync_rowsets() due to successful compaction on other BEs on the same tablet. That PR makes the process of reading tablets' stats and writing the delete bitmap update lock KV in one fdb txn to achieve the atomic sematic. However, when a load involves a large number of tablets, the process of reading tablets' stats may take longer than fdb txn's 5 seconds limitation and cause TXN_TOO_OLD error.
This PR re-arrange the process so that the read of tablet stats can be not necessarily in the same fdb txn with the txn which update the lock_info.lock_id. In detail, we do as the following:

gain the delete bitmap update lock in MS (write delete bitmap update lock KV)
read tablets' stats to get the compaction counts.
check if the delete bitmap update lock is still held by the current load.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-12-09T10:46:40Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

github-actions · 2024-12-09T10:50:53Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-12-10T04:00:55Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-12-10T04:08:03Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions

clang-tidy made some suggestions

github-actions · 2024-12-10T05:22:19Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions

clang-tidy made some suggestions

github-actions

clang-tidy made some suggestions

github-actions

clang-tidy made some suggestions

github-actions

clang-tidy made some suggestions

github-actions

clang-tidy made some suggestions

github-actions

clang-tidy made some suggestions

bobhan1 · 2024-12-10T07:12:30Z

run buildall

zhannngchen

LGTM

github-actions · 2024-12-11T11:14:34Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-12-11T11:14:36Z

PR approved by anyone and no changes requested.

github-actions

clang-tidy made some suggestions

github-actions

clang-tidy made some suggestions

… will not be cleared

github-actions

clang-tidy made some suggestions

bobhan1 · 2024-12-13T08:46:08Z

run buildall

zhannngchen

LGTM

github-actions · 2024-12-17T09:28:25Z

PR approved by at least one committer and no changes requested.

…ck be able to be in different fdb txns (#45206) #37670 let the meta service return the tablet compaction stats along with the `getDeleteBitmapUpdateLockResponse` to FE to let the BE know whether it should `sync_rowsets()` due to successful compaction on other BEs on the same tablet. That PR makes the process of reading tablets' stats and writing the delete bitmap update lock KV in one fdb txn to achieve the atomic sematic. However, when a load involves a large number of tablets, the process of reading tablets' stats may take longer than fdb txn's 5 seconds limitation and cause `TXN_TOO_OLD` error. This PR re-arrange the process so that the read of tablet stats can be not necessarily in the same fdb txn with the txn which update the lock_info.lock_id. In detail, we do as the following: 1. gain the delete bitmap update lock in MS (write delete bitmap update lock KV) 2. read tablets' stats to get the compaction counts. 3. check if the delete bitmap update lock is still held by the current load.

…ap update lock be able to be in different fdb txns #45206 (#45559) Cherry-picked from #45206 Co-authored-by: bobhan1 <baohan@selectdb.com>

…pdate delete bitmap update lock be able to be in different fdb txns apache#45206 (apache#45559) (apache#3910)

…ck be able to be in different fdb txns (apache#45206) apache#37670 let the meta service return the tablet compaction stats along with the `getDeleteBitmapUpdateLockResponse` to FE to let the BE know whether it should `sync_rowsets()` due to successful compaction on other BEs on the same tablet. That PR makes the process of reading tablets' stats and writing the delete bitmap update lock KV in one fdb txn to achieve the atomic sematic. However, when a load involves a large number of tablets, the process of reading tablets' stats may take longer than fdb txn's 5 seconds limitation and cause `TXN_TOO_OLD` error. This PR re-arrange the process so that the read of tablet stats can be not necessarily in the same fdb txn with the txn which update the lock_info.lock_id. In detail, we do as the following: 1. gain the delete bitmap update lock in MS (write delete bitmap update lock KV) 2. read tablets' stats to get the compaction counts. 3. check if the delete bitmap update lock is still held by the current load.

bobhan1 force-pushed the fix-ms-get-delete-bitmap-lock-get-stats branch from 86d4945 to 0df6084Compare December 10, 2024 04:03

github-actionsBot reviewed Dec 10, 2024
View reviewed changes

Comment threadcloud/test/meta_service_test.cpp Outdated

bobhan1 force-pushed the fix-ms-get-delete-bitmap-lock-get-stats branch from f74de71 to 54905d9Compare December 10, 2024 05:18

github-actionsBot reviewed Dec 10, 2024
View reviewed changes

Comment threadcloud/test/meta_service_test.cpp
Comment threadcloud/test/meta_service_test.cpp

github-actionsBot reviewed Dec 10, 2024
View reviewed changes

Comment threadcloud/test/meta_service_test.cpp

github-actionsBot reviewed Dec 10, 2024
View reviewed changes

Comment threadcloud/test/meta_service_test.cpp
Comment threadcloud/test/meta_service_test.cpp

bobhan1 force-pushed the fix-ms-get-delete-bitmap-lock-get-stats branch 2 times, most recently from ae7e0df to fc8d9feCompare December 10, 2024 06:54

github-actionsBot reviewed Dec 10, 2024
View reviewed changes

Comment threadcloud/test/meta_service_test.cpp
Comment threadcloud/test/meta_service_test.cpp

github-actionsBot reviewed Dec 10, 2024
View reviewed changes

Comment threadcloud/test/meta_service_test.cpp
Comment threadcloud/test/meta_service_test.cpp

github-actionsBot reviewed Dec 10, 2024
View reviewed changes

Comment threadcloud/test/meta_service_test.cpp

zhannngchen previously approved these changes Dec 11, 2024
View reviewed changes

zhannngchen added the dev/3.0.x label Dec 11, 2024

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Dec 11, 2024

github-actionsBot added the reviewed label Dec 11, 2024

zhannngchen requested a review from gavinchou December 11, 2024 11:15

gavinchou reviewed Dec 12, 2024
View reviewed changes

Comment threadcloud/src/meta-service/meta_service.cpp Outdated

gavinchou reviewed Dec 12, 2024
View reviewed changes

Comment threadcloud/src/meta-service/meta_service.cpp Outdated

gavinchou reviewed Dec 12, 2024
View reviewed changes

Comment threadcloud/src/meta-service/meta_service.cpp

gavinchou reviewed Dec 12, 2024
View reviewed changes

Comment threadcloud/src/meta-service/meta_service.cpp Outdated

gavinchou reviewed Dec 12, 2024
View reviewed changes

Comment threadcloud/src/meta-service/meta_service.cpp Outdated

update
7070d5b

bobhan1 added 4 commits December 13, 2024 11:34

update
01e1472

update
83a561d

add total retry limit
c24dfa5

update
2097edc

bobhan1 dismissed zhannngchen’s stale review via 2097edcDecember 13, 2024 06:54

bobhan1 force-pushed the fix-ms-get-delete-bitmap-lock-get-stats branch from 379f2f2 to 2097edcCompare December 13, 2024 06:54

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Dec 13, 2024

github-actionsBot reviewed Dec 13, 2024
View reviewed changes

Comment threadcloud/src/meta-service/meta_service.cpp

fix case
c834a14

github-actionsBot reviewed Dec 13, 2024
View reviewed changes

Comment threadcloud/test/meta_service_test.cpp

don't check compaction cnts in ut if has error becase compaction cnts…
85c841b
… will not be cleared

github-actionsBot reviewed Dec 13, 2024
View reviewed changes

Comment threadcloud/test/meta_service_test.cpp
Comment threadcloud/test/meta_service_test.cpp

add check in fe
f4d0a54

bobhan1 requested a review from gavinchou December 13, 2024 08:34

fix format
c032eeb

zhannngchen approved these changes Dec 17, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Dec 17, 2024

gavinchou approved these changes Dec 17, 2024
View reviewed changes

gavinchou merged commit bc6c2e7 into apache:masterDec 17, 2024

github-actionsBot mentioned this pull request Dec 17, 2024
branch-3.0: [opt](cloud) Make get tablet stats and update delete bitmap update lock be able to be in different fdb txns #45206 #45559
Merged

dataroaring added dev/3.0.4-merged and removed dev/3.0.x labels Dec 26, 2024

zhannngchen mentioned this pull request Jan 22, 2025
[improve](cloud-mow) batch get tablet stats when get_delete_bitmap_update_lock #47281
Merged
16 tasks

suxiaogang223 pushed a commit to suxiaogang223/doris that referenced this pull request Jul 15, 2025
(selectdb-cloud-4.0.4): pick [opt](cloud) Make get tablet stats and u…
8397e94
…pdate delete bitmap update lock be able to be in different fdb txns apache#45206 (apache#45559) (apache#3910)

Uh oh!

Conversation

bobhan1 commented Dec 9, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

doris-robot commented Dec 9, 2024

Uh oh!

github-actionsBot commented Dec 9, 2024

Uh oh!

github-actionsBot commented Dec 10, 2024

Uh oh!

github-actionsBot commented Dec 10, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actionsBot commented Dec 10, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bobhan1 commented Dec 10, 2024

Uh oh!

zhannngchen left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Dec 11, 2024

Uh oh!

github-actionsBot commented Dec 11, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bobhan1 commented Dec 13, 2024

Uh oh!

bobhan1 commented Dec 9, 2024•
edited
Loading