branch-4.0: [fix](cloud) Use bthread-aware shared mutex for tablet header lock by liaoxin01 · Pull Request #66019 · apache/doris

liaoxin01 · 2026-07-24T13:17:14Z

### What problem does this PR solve? Issue Number: None Related PR: apache#64574 Problem Summary: The tablet header lock used a thread-affine shared mutex. In cloud mode, a bthread could suspend while holding the write lock and resume on another worker pthread, leaving the lock permanently wedged. Replace the tablet header lock with BthreadSharedMutex, which uses bthread-aware synchronization and supports cross-worker resume safely. Keep unrelated shared mutexes unchanged. ### Release note Fix a potential permanent tablet header lock wedge after bthread migration in cloud mode. ### Check List (For Author) - Test: Unit Test - ./run-be-ut.sh --run --filter=BthreadSharedMutexTest.* -j 32 (7 tests passed) - Behavior changed: Yes. The tablet header lock is now bthread-aware. - Does this need documentation: No

hello-stephen · 2026-07-24T13:17:19Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Copilot

Pull request overview

Note

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

This PR backports #64574 to replace the tablet header/meta lock with a bthread-migration-safe shared mutex, and updates call sites to avoid self-deadlock from recursive shared acquisition under writer-preferring semantics.

Changes:

Introduces BthreadSharedMutex (bthread-based std::shared_mutex analogue) and adds dedicated unit tests.
Replaces std::shared_mutex tablet header locks with BthreadSharedMutex and updates locking at key metadata mutation/read sites.
Adds/uses _unlocked APIs in tablet/cloud-tablet paths to prevent recursive lock acquisition under a shared lock.

Reviewed changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
be/src/util/bthread_shared_mutex.h	Adds bthread-aware shared mutex implementation.
be/test/util/bthread_shared_mutex_test.cpp	Adds correctness/regression/stress coverage for the new mutex.
be/src/olap/base_tablet.h	Switches meta lock type to `BthreadSharedMutex`; adds “_unlocked” score API declaration.
be/src/olap/base_tablet.cpp	Refactors compaction score calculation into locked + unlocked variants.
be/src/olap/tablet.h	Switches header lock return type; adds unlocked rowset-pick APIs to avoid recursive locking.
be/src/olap/tablet.cpp	Updates lock usage and routes through unlocked helper paths under existing shared locks.
be/src/cloud/cloud_tablet.h	Updates APIs taking header locks to use `BthreadSharedMutex`; introduces unlocked pick APIs.
be/src/cloud/cloud_tablet.cpp	Removes inner re-locking in “caller already holds lock” paths; updates lock types and pick APIs.
be/src/cloud/cloud_meta_mgr.h	Updates `fill_version_holes` lock parameter type.
be/src/cloud/cloud_meta_mgr.cpp	Updates lock type usage related to tablet header lock.
be/src/olap/task/index_builder.cpp	Updates header lock guard type to match new mutex.
be/src/olap/task/engine_clone_task.cpp	Updates header lock guard type to match new mutex.
be/src/olap/tablet_manager.cpp	Updates header lock guard type to match new mutex.
be/src/olap/schema_change.cpp	Updates header lock guard type to match new mutex.
be/src/olap/rowset_builder.cpp	Updates header lock guard type to match new mutex.
be/src/olap/full_compaction.cpp	Updates header lock guard type to match new mutex.
be/src/olap/compaction.cpp	Updates header lock guard type to match new mutex.
be/src/olap/cumulative_compaction_time_series_policy.cpp	Uses unlocked tablet API when caller already holds `_meta_lock`.
be/src/cloud/cloud_full_compaction.cpp	Uses unlocked cloud-tablet API under an existing shared header lock.
be/src/cloud/cloud_base_compaction.cpp	Uses unlocked cloud-tablet API under an existing shared header lock.
be/src/cloud/cloud_cumulative_compaction.cpp	Switches traversal call to unlocked variant in locked context.
be/test/cloud/cloud_meta_mgr_test.cpp	Updates tests to lock with new header lock type (via CTAD).
be/test/cloud/cloud_empty_rowset_compaction_test.cpp	Updates tests to lock with new header lock type (via CTAD).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

liaoxin01 · 2026-07-27T07:13:03Z

run buildall

hello-stephen · 2026-07-27T09:16:25Z

BE UT Coverage Report

Increment line coverage 81.20% (95/117) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.56% (19592/36579)
Line Coverage	36.62% (183333/500618)
Region Coverage	33.17% (142487/429546)
Branch Coverage	34.04% (61625/181046)

liaoxin01 · 2026-07-27T09:35:05Z

run buildall

github-actions · 2026-07-27T09:42:58Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-07-27T09:43:00Z

PR approved by anyone and no changes requested.

hello-stephen · 2026-07-27T11:09:55Z

BE Regression && UT Coverage Report

Increment line coverage 92.93% (92/99) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.62% (25628/35784)
Line Coverage	54.51% (271966/498945)
Region Coverage	52.05% (225366/432964)
Branch Coverage	53.57% (97212/181462)

hello-stephen · 2026-07-27T12:41:06Z

BE UT Coverage Report

Increment line coverage 81.20% (95/117) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.57% (19597/36579)
Line Coverage	36.63% (183371/500647)
Region Coverage	33.20% (142635/429583)
Branch Coverage	34.04% (61637/181056)

hello-stephen · 2026-07-27T12:52:06Z

BE Regression && UT Coverage Report

Increment line coverage 92.93% (92/99) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.62% (25629/35784)
Line Coverage	54.47% (271801/498974)
Region Coverage	51.95% (224945/433001)
Branch Coverage	53.52% (97130/181472)

hello-stephen · 2026-07-27T13:48:14Z

BE Regression && UT Coverage Report

Increment line coverage 92.93% (92/99) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.62% (25629/35784)
Line Coverage	54.47% (271801/498974)
Region Coverage	51.95% (224945/433001)
Branch Coverage	53.52% (97130/181472)

hello-stephen · 2026-07-27T13:59:14Z

BE Regression && UT Coverage Report

Increment line coverage 92.93% (92/99) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.62% (25629/35784)
Line Coverage	54.47% (271812/498974)
Region Coverage	51.96% (225007/433001)
Branch Coverage	53.53% (97134/181472)

hello-stephen · 2026-07-27T14:37:12Z

BE Regression && UT Coverage Report

Increment line coverage 92.93% (92/99) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.62% (25630/35784)
Line Coverage	54.48% (271838/498974)
Region Coverage	52.00% (225166/433001)
Branch Coverage	53.54% (97153/181472)

liaoxin01 · 2026-07-27T14:43:45Z

run p0

hello-stephen · 2026-07-27T16:13:40Z

BE Regression && UT Coverage Report

Increment line coverage 92.93% (92/99) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.64% (25637/35784)
Line Coverage	54.51% (271983/498974)
Region Coverage	52.03% (225290/433001)
Branch Coverage	53.56% (97193/181472)

liaoxin01 requested a review from morningman as a code owner July 24, 2026 13:17

CopilotAI review requested due to automatic review settings July 24, 2026 13:17

CopilotAI reviewed Jul 24, 2026
View reviewed changes

Comment threadbe/src/util/bthread_shared_mutex.h
Comment threadbe/src/cloud/cloud_meta_mgr.cpp

Copilotstarted reviewing on behalf of liaoxin01 July 24, 2026 13:53 View session

morningman approved these changes Jul 27, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 27, 2026

github-actionsBot added the reviewed label Jul 27, 2026

liaoxin01 merged commit 88db7a4 into apache:branch-4.0Jul 28, 2026
28 of 31 checks passed

liaoxin01 deleted the codex/pick-64574-to-branch-4.0 branch July 28, 2026 09:11

Uh oh!

Conversation

liaoxin01 commented Jul 24, 2026

Uh oh!

hello-stephen commented Jul 24, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

liaoxin01 commented Jul 27, 2026

Uh oh!

hello-stephen commented Jul 27, 2026

BE UT Coverage Report

Uh oh!

liaoxin01 commented Jul 27, 2026

Uh oh!

github-actionsBot commented Jul 27, 2026

Uh oh!

github-actionsBot commented Jul 27, 2026

Uh oh!

hello-stephen commented Jul 27, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Jul 27, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jul 27, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Jul 27, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Jul 27, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Jul 27, 2026

BE Regression && UT Coverage Report

Uh oh!

liaoxin01 commented Jul 27, 2026

Uh oh!

hello-stephen commented Jul 27, 2026

BE Regression && UT Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants