Uh oh!
There was an error while loading. Please reload this page.
[fix](balance) do not pick tablets whose data size is zero for BE balance - #66499
Merged
Conversation
…ance A backend's load score is capacityProportion * capCoeff + replicaNumProportion * (1 - capCoeff) A tablet whose replica data size is zero relocates nothing, so migrating it leaves the capacity term untouched while still improving the replica count term. LoadStatisticForTag.isMoreBalanced() therefore accepts the move even though not a single byte is copied. The disk usage difference that triggers a balance is measured as (totalCapacity - availableCapacity), so it also covers non-Doris files and plain disk size differences, which no tablet migration can ever remove. With zero sized tablets as the only available lever, the scheduler keeps shifting replica counts until the composite score converges, and the per partition replica distribution pays for it: a freshly created table that createTablets() spread round robin over 16 backends as 16/17 ends up as 12-18, while every backend still reports a MID load score. Skip such tablets in BE load balance, both when selecting candidates and again when the context is finally scheduled, since the two can be far apart in time and the size is refreshed in between. Zero also means the size has not been reported yet, because replica data size is not persisted in the FE image and stays zero after a restart until the next tablet stat update; balancing on an unknown size is guesswork, so skipping is right there as well. This is the same rule DiskRebalancer.completeSchedCtx() already applies to disk balance, and it is unconditional here too, including urgent balance: moving an empty replica frees no space on a filling disk either. Add TStat counterBalanceRejectByZeroDataSize for the scheduling time rejection, and report the number of tablets skipped during selection in the existing per round log instead of a counter, because after a restart every tablet of every high load backend hits that path on every round.
hello-stephen
commented
Aug 5, 2026
Contributor
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
deardeng
commented
Aug 5, 2026
ContributorAuthor
run buildall |
deardeng
commented
Aug 5, 2026
ContributorAuthor
/review |
Contributor
Codex automated review failed and did not complete. Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-08T03:32:00Z. Please trigger /review again after that time. |
hello-stephen
commented
Aug 5, 2026
Contributor
FE UT Coverage ReportIncrement line coverage |
hello-stephen
commented
Aug 5, 2026
Contributor
TPC-H: Total hot run time: 28768 ms |
hello-stephen
commented
Aug 5, 2026
Contributor
TPC-DS: Total hot run time: 165926 ms |
hello-stephen
commented
Aug 5, 2026
Contributor
ClickBench: Total hot run time: 23.9 s |
hello-stephen
commented
Aug 5, 2026
Contributor
FE Regression Coverage ReportIncrement line coverage |
gavinchou
approved these changes
Aug 10, 2026
Contributor
PR approved by at least one committer and no changes requested. |
Contributor
PR approved by anyone and no changes requested. |
Uh oh!
There was an error while loading. Please reload this page.
github-actionsBot
pushed a commit
that referenced
this pull request
Aug 11, 2026
…ance (#66499) A backend's load score is capacityProportion * capCoeff + replicaNumProportion * (1 - capCoeff) A tablet whose replica data size is zero relocates nothing, so migrating it leaves the capacity term untouched while still improving the replica count term. LoadStatisticForTag.isMoreBalanced() therefore accepts the move even though not a single byte is copied. The disk usage difference that triggers a balance is measured as (totalCapacity - availableCapacity), so it also covers non-Doris files and plain disk size differences, which no tablet migration can ever remove. With zero sized tablets as the only available lever, the scheduler keeps shifting replica counts until the composite score converges, and the per partition replica distribution pays for it: a freshly created table that createTablets() spread round robin over 16 backends as 16/17 ends up as 12-18, while every backend still reports a MID load score. Skip such tablets in BE load balance, both when selecting candidates and again when the context is finally scheduled, since the two can be far apart in time and the size is refreshed in between. Zero also means the size has not been reported yet, because replica data size is not persisted in the FE image and stays zero after a restart until the next tablet stat update; balancing on an unknown size is guesswork, so skipping is right there as well. This is the same rule DiskRebalancer.completeSchedCtx() already applies to disk balance, and it is unconditional here too, including urgent balance: moving an empty replica frees no space on a filling disk either. Add TStat counterBalanceRejectByZeroDataSize for the scheduling time rejection, and report the number of tablets skipped during selection in the existing per round log instead of a counter, because after a restart every tablet of every high load backend hits that path on every round.
github-actionsBot
pushed a commit
that referenced
this pull request
Aug 11, 2026
…ance (#66499) A backend's load score is capacityProportion * capCoeff + replicaNumProportion * (1 - capCoeff) A tablet whose replica data size is zero relocates nothing, so migrating it leaves the capacity term untouched while still improving the replica count term. LoadStatisticForTag.isMoreBalanced() therefore accepts the move even though not a single byte is copied. The disk usage difference that triggers a balance is measured as (totalCapacity - availableCapacity), so it also covers non-Doris files and plain disk size differences, which no tablet migration can ever remove. With zero sized tablets as the only available lever, the scheduler keeps shifting replica counts until the composite score converges, and the per partition replica distribution pays for it: a freshly created table that createTablets() spread round robin over 16 backends as 16/17 ends up as 12-18, while every backend still reports a MID load score. Skip such tablets in BE load balance, both when selecting candidates and again when the context is finally scheduled, since the two can be far apart in time and the size is refreshed in between. Zero also means the size has not been reported yet, because replica data size is not persisted in the FE image and stays zero after a restart until the next tablet stat update; balancing on an unknown size is guesswork, so skipping is right there as well. This is the same rule DiskRebalancer.completeSchedCtx() already applies to disk balance, and it is unconditional here too, including urgent balance: moving an empty replica frees no space on a filling disk either. Add TStat counterBalanceRejectByZeroDataSize for the scheduling time rejection, and report the number of tablets skipped during selection in the existing per round log instead of a counter, because after a restart every tablet of every high load backend hits that path on every round.
wyxxxcat pushed a commit
to wyxxxcat/doris
that referenced
this pull request
Aug 17, 2026
…ance (apache#66499) A backend's load score is capacityProportion * capCoeff + replicaNumProportion * (1 - capCoeff) A tablet whose replica data size is zero relocates nothing, so migrating it leaves the capacity term untouched while still improving the replica count term. LoadStatisticForTag.isMoreBalanced() therefore accepts the move even though not a single byte is copied. The disk usage difference that triggers a balance is measured as (totalCapacity - availableCapacity), so it also covers non-Doris files and plain disk size differences, which no tablet migration can ever remove. With zero sized tablets as the only available lever, the scheduler keeps shifting replica counts until the composite score converges, and the per partition replica distribution pays for it: a freshly created table that createTablets() spread round robin over 16 backends as 16/17 ends up as 12-18, while every backend still reports a MID load score. Skip such tablets in BE load balance, both when selecting candidates and again when the context is finally scheduled, since the two can be far apart in time and the size is refreshed in between. Zero also means the size has not been reported yet, because replica data size is not persisted in the FE image and stays zero after a restart until the next tablet stat update; balancing on an unknown size is guesswork, so skipping is right there as well. This is the same rule DiskRebalancer.completeSchedCtx() already applies to disk balance, and it is unconditional here too, including urgent balance: moving an empty replica frees no space on a filling disk either. Add TStat counterBalanceRejectByZeroDataSize for the scheduling time rejection, and report the number of tablets skipped during selection in the existing per round log instead of a counter, because after a restart every tablet of every high load backend hits that path on every round.
16 tasks
morrySnow pushed a commit
that referenced
this pull request
Aug 31, 2026
…ance (#66499) A backend's load score is capacityProportion * capCoeff + replicaNumProportion * (1 - capCoeff) A tablet whose replica data size is zero relocates nothing, so migrating it leaves the capacity term untouched while still improving the replica count term. LoadStatisticForTag.isMoreBalanced() therefore accepts the move even though not a single byte is copied. The disk usage difference that triggers a balance is measured as (totalCapacity - availableCapacity), so it also covers non-Doris files and plain disk size differences, which no tablet migration can ever remove. With zero sized tablets as the only available lever, the scheduler keeps shifting replica counts until the composite score converges, and the per partition replica distribution pays for it: a freshly created table that createTablets() spread round robin over 16 backends as 16/17 ends up as 12-18, while every backend still reports a MID load score. Skip such tablets in BE load balance, both when selecting candidates and again when the context is finally scheduled, since the two can be far apart in time and the size is refreshed in between. Zero also means the size has not been reported yet, because replica data size is not persisted in the FE image and stays zero after a restart until the next tablet stat update; balancing on an unknown size is guesswork, so skipping is right there as well. This is the same rule DiskRebalancer.completeSchedCtx() already applies to disk balance, and it is unconditional here too, including urgent balance: moving an empty replica frees no space on a filling disk either. Add TStat counterBalanceRejectByZeroDataSize for the scheduling time rejection, and report the number of tablets skipped during selection in the existing per round log instead of a counter, because after a restart every tablet of every high load backend hits that path on every round.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A backend's load score is
A tablet whose replica data size is zero relocates nothing, so migrating it leaves the capacity term untouched while still improving the replica count term. LoadStatisticForTag.isMoreBalanced() therefore accepts the move even though not a single byte is copied.
The disk usage difference that triggers a balance is measured as (totalCapacity - availableCapacity), so it also covers non-Doris files and plain disk size differences, which no tablet migration can ever remove. With zero sized tablets as the only available lever, the scheduler keeps shifting replica counts until the composite score converges, and the per partition replica distribution pays for it: a freshly created table that createTablets() spread round robin over 16 backends as 16/17 ends up as 12-18, while every backend still reports a MID load score.
Skip such tablets in BE load balance, both when selecting candidates and again when the context is finally scheduled, since the two can be far apart in time and the size is refreshed in between. Zero also means the size has not been reported yet, because replica data size is not persisted in the FE image and stays zero after a restart until the next tablet stat update; balancing on an unknown size is guesswork, so skipping is right there as well.
This is the same rule DiskRebalancer.completeSchedCtx() already applies to disk balance, and it is unconditional here too, including urgent balance: moving an empty replica frees no space on a filling disk either.
Add TStat counterBalanceRejectByZeroDataSize for the scheduling time rejection, and report the number of tablets skipped during selection in the existing per round log instead of a counter, because after a restart every tablet of every high load backend hits that path on every round.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)