Uh oh!
There was an error while loading. Please reload this page.
branch-4.1: [improvement](cloud) Add dry-run mode for BE-to-MS RPC rate limiting #66977 - #67240
Merged
Merged
Conversation
hello-stephen
commented
Aug 28, 2026
Contributor
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
hello-stephen
commented
Aug 28, 2026
Contributor
run buildall |
hello-stephen
commented
Aug 28, 2026
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
hello-stephen
commented
Aug 28, 2026
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
bobhan1
commented
Aug 28, 2026
Contributor
run external |
hello-stephen
commented
Aug 28, 2026
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
morrySnowforce-pushed
the
auto-pick-66977-branch-4.1
branch
from
August 31, 2026 07:14
5775718 to
a10f932Comparebobhan1
commented
Sep 3, 2026
Contributor
run buildall |
hello-stephen
commented
Sep 3, 2026
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
…66977) ### What problem does this PR solve? Issue Number: N/A Related PR: #66969, #66940 Problem Summary: BE-to-MetaService host-level and table-level rate limiting needs production-safe observability before thresholds are enforced. This PR adds dry-run switches for both paths. Dry-run keeps limit evaluation, would-wait metrics, per-table QPS collection, and table-level MS_BUSY state transitions active, but does not delay requests. | Scope | New config | Default | Existing enforcement config | Purpose | | --- | --- | --- | --- | --- | | Host-level | `enable_ms_rpc_host_level_rate_limit_dry_run` | `true` | `enable_ms_rpc_host_level_rate_limit` (default `false`) | Evaluate each configured per-RPC token bucket, record would-wait observations, and log would-throttle decisions without sleeping. | | Table-level | `enable_ms_backpressure_handling_dry_run` | `true` | `enable_ms_backpressure_handling` (default `false`) | Keep per-table QPS collection and the MS_BUSY adaptive-throttling state machine active, but do not sleep on a throttle decision. | For each scope, the dry-run and enforcement configs interact as follows: - When both configs are `false`, rate-limit evaluation for that scope is disabled. - When only the enforcement config is `true`, the limiter is enforced and the request sleeps for the calculated wait time. - When the dry-run config is `true`, dry-run takes precedence regardless of the enforcement config: the limiter state, metrics, and logs are updated, but the request does not sleep. Both host-level and table-level dry-run paths reserve against the same limiter state used by enforcement, matching the FE dry-run model: dry-run suppresses sleeping but preserves reservation state. When a table-level downgrade restores a previous QPS limit, it resets queued reservations so stale wait debt does not carry into the relaxed limit; the final downgrade removes the limiter. The table-level coordinator uses 64-bit counters with overflow-free saturation at each timing threshold. Its MS_BUSY downgrade timer is reset to inactive when there is no pending upgrade history, while the upgrade cooldown counter is preserved and stops at the configured threshold so runtime cooldown changes retain the existing timing semantics. ### Signals after rate limiting is triggered Throttle-decision logs use `INFO` level and are suppressed independently per RPC type to at most one log per second. They are emitted only when the limiter calculates a positive wait. Host-level dry-run logs explicitly report the estimated wait: ```text [ms-throttle] host-level rate limiter dry run would throttle MS RPC request, rpc=get rowset, estimated_wait_ns=1250000, qps_limit=320 ``` With enforcement enabled and dry-run disabled, the corresponding log reports the actual sleep: ```text [ms-throttle] host-level rate limiter throttled MS RPC request, rpc=get rowset, sleep_ns=1250000, qps_limit=320 ``` Table-level dry-run and enforcement use the same trigger log so operators can search one stable pattern: ```text [ms-throttle] table-level rate limiter triggered for MS RPC request, rpc=commit_rowset, table_id=10001, wait_us=8200, current_qps=146.7, qps_limit=100 ``` For table-level dry-run, `wait_us` is the estimated wait and the request is not delayed. With enforcement enabled and dry-run disabled, the request sleeps for that wait. MS_BUSY-driven table-level state transitions also emit `INFO` logs, for example: ```text [ms-throttle] received MS_BUSY, triggering upgrade [ms-throttle] upgrade: rpc=commit_rowset, table_id=10001, current_qps=200, old_limit=0, new_limit=150 [ms-throttle] downgrade: rpc=commit_rowset, table_id=10001, removed limit ``` The same events are observable through bvars: host-level per-RPC latency recorders based on `host_level_ms_rpc_rate_limit_sleep`, table-level `ms_rpc_backpressure_throttle_wait_<rpc>` latency recorders, and `ms_rpc_backpressure_ms_busy_*`, `ms_rpc_backpressure_upgrade_*`, and `ms_rpc_backpressure_downgrade_*` counters/windows. ### Release note Add dry-run observation for BE-to-MetaService host-level and table-level RPC rate limiting without delaying requests. Both dry-run switches are enabled by default and take precedence over enforcement. Dry-run and enforcement share limiter reservation state, table-level downgrade resets queued reservations when restoring a previous QPS limit, and long-running table-level coordinator timers no longer overflow. ### Check List (For Author) - Test - [ ] Regression test - [x] Unit Test - `./run-be-ut.sh --run --filter='HostLevelMSRpcRateLimitersTest.*:StrictQpsLimiterTest.*:TableRpcThrottlerTest.*:MSBackpressureHandlerTest.*:RpcThrottleStateMachineTest.*:RpcThrottleIntegrationTest.*' -j100` (56 tests passed) - `./run-be-ut.sh --run --filter='TokenBucketRateLimiterTest.*:S3RateLimiterManagerTest.*:S3RateLimiterMetricsTest.*' -j100` (11 tests passed) - `./run-be-ut.sh --run --filter='MSBackpressureHandlerTest.*:RpcThrottleStateMachineTest.*:RpcThrottleCoordinatorTest.*:RpcThrottleIntegrationTest.*' -j100` (44 tests passed) - `./build.sh --be -j100` - `build-support/check-format.sh` - `build-support/check-build-hygiene.sh` - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [x] Yes. Dry-run observation for both limiter levels is enabled by default and does not delay MetaService RPCs; dry-run and enforcement share reservation state; table-level downgrade resets queued reservations; coordinator timing counters saturate without overflowing; actual and dry-run decisions emit per-RPC rate-limited `INFO` logs. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
yiguoleiforce-pushed
the
auto-pick-66977-branch-4.1
branch
from
September 5, 2026 22:17
a10f932 to
f86296bCompareyiguolei
commented
Sep 5, 2026
Contributor
run buildall |
hello-stephen
commented
Sep 5, 2026
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
hello-stephen
commented
Sep 6, 2026
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
approved these changes
Sep 6, 2026
ContributorAuthor
PR approved by anyone and no changes requested. |
ContributorAuthor
PR approved by at least one committer and no changes requested. |
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picked from #66977