Uh oh!
There was an error while loading. Please reload this page.
branch-4.1: [Opt](cloud) Add rate limit for BE to MS rpc - #64396
Merged
Conversation
This PR implements a two-layer MS (Meta Service) RPC rate limiting system for Doris cloud mode BE: 1. **Host-level rate limiting** — Token-bucket based QPS limiter for all 21 MS RPC types, preventing a single BE from overwhelming the MS with burst traffic. 2. **Table-level adaptive backpressure** — When MS returns `MS_BUSY` error code, BE dynamically identifies and throttles the top-k highest-QPS tables using a state machine, and automatically relaxes limits after the pressure subsides. --- In cloud mode, all BE nodes send RPCs (get_tablet, prepare_rowset, commit_rowset, etc.) to a shared Meta Service. A single BE experiencing load spikes (e.g., large batch imports, compaction storms) can send excessive RPC traffic that overwhelms MS, degrading service for all BEs. Introduce `HostLevelMSRpcRateLimiters` — a per-BE, per-RPC-type rate limiter using token bucket algorithm. **Architecture:** - All 21 MS RPC types are enumerated in `MetaServiceRPC` enum (defined via X-macro for maintainability) - Each RPC type has an independent `TokenBucketRateLimiterHolder` with its own QPS limit - QPS limits are configured per CPU core: `actual_qps = config_value × num_cores` - Thread-safe design using `atomic_shared_ptr<RpcRateLimiter>` array for lock-free concurrent access during `limit()` calls - Each rate limiter includes a `bvar::LatencyRecorder` to monitor sleep durations caused by rate limiting **Configuration:** | Config | Default | Description | |--------|---------|-------------| | `enable_ms_rpc_host_level_rate_limit` | `true` | Global enable/disable switch | | `ms_rpc_qps_default` | `100` | Default per-core QPS for all RPCs | | `ms_rpc_qps_<rpc_name>` | `-1` | Per-RPC override (`-1` = use default, `0` = disabled) | All QPS configs are mutable (`DEFINE_mInt32`), allowing runtime adjustment without restart. `reset_all()` re-reads configs and recreates rate limiters. **Integration:** Rate limiting is applied inside the `retry_rpc()` template function in `cloud_meta_mgr.cpp`, which wraps all MS RPC calls. The `RpcRateLimitCtx` struct carries the rate limiter reference. Rate limiting executes before each RPC attempt (including retries), with the call to `apply_rate_limit()` performing a `bthread_usleep` if the token bucket requires waiting. **New files:** - `be/src/cloud/cloud_ms_rpc_rate_limiters.h` / `.cpp` - `be/test/cloud/cloud_ms_rpc_rate_limiters_test.cpp` --- Host-level rate limiting applies uniformly across all tables. When MS reports overload (`MAX_QPS_LIMIT`), it's often caused by a small number of high-traffic tables (e.g., tables with many concurrent stream load jobs). A uniform rate limit would unnecessarily penalize all tables, while the hot tables continue to dominate the RPC traffic. Implement table-level adaptive throttling for load-related RPCs. When MS returns `MAX_QPS_LIMIT`, BE identifies the top-k highest-QPS tables and progressively reduces their QPS limits, while leaving other tables unaffected. **Scope:** Only 5 load-related RPC types participate in table-level throttling: - `PREPARE_ROWSET` - `COMMIT_ROWSET` - `UPDATE_TMP_ROWSET` - `UPDATE_PACKED_FILE_INFO` - `UPDATE_DELETE_BITMAP` **Architecture (4 components with clear separation of concerns):** ``` MS_BUSY signal (MAX_QPS_LIMIT) │ ▼ ┌─────────────────────────┐ ┌──────────────────────────┐ │ RpcThrottleCoordinator │──────▶│ RpcThrottleStateMachine │ │ (timing control) │ │ (pure state logic) │ │ - upgrade cooldown │ │ - upgrade history stack │ │ - downgrade trigger │ │ - limit calculation │ └─────────────────────────┘ └──────────┬───────────────┘ │ Actions ▼ ┌─────────────────────────┐ ┌──────────────────────────┐ │ TableRpcQpsRegistry │ │ TableRpcThrottler │ │ (QPS statistics) │ │ (limit enforcement) │ │ - per-table bvar │ │ - StrictQpsLimiter │ │ - top-k query │ │ - per (rpc, table) │ └─────────────────────────┘ └──────────────────────────┘ ``` **Component details:** 1. **`TableRpcQpsRegistry`** — Tracks per-(rpc_type, table_id) QPS using `bvar::PerSecond<bvar::Adder>`. Supports efficient top-k query via min-heap. Configurable time window via `ms_rpc_table_qps_window_sec` (immutable, default 10s). 2. **`RpcThrottleStateMachine`** — Pure state machine with no time awareness or side effects. Maintains upgrade history as a stack for clean rollback. - `on_upgrade(snapshot)`: For each top-k table in the QPS snapshot, calculates `new_limit = current_qps × ratio` (first time) or `current_limit × ratio` (already limited), with a floor of `ms_rpc_table_qps_limit_floor`. Returns `SET_LIMIT` actions. - `on_downgrade()`: Pops the most recent upgrade from history. If the table had a prior limit, restores it (`SET_LIMIT`). If no prior limit, removes it (`REMOVE_LIMIT`). 3. **`RpcThrottleCoordinator`** — Timing control layer using tick counts (1 tick = 1 ms). - `report_ms_busy()`: Returns true if enough ticks have passed since last upgrade (cooldown). - `tick(n)`: Advances time by n ticks. Returns true if downgrade should trigger (no MS_BUSY for `downgrade_after_ticks`). 4. **`TableRpcThrottler`** — Enforces QPS limits using `StrictQpsLimiter` (strict fixed-interval, no burst allowed). Each (rpc_type, table_id) pair has its own limiter. Returns the time point when the request may execute; the caller sleeps until then. 5. **`MSBackpressureHandler`** — Orchestrator that wires all components together: - `on_ms_busy()`: Called when `retry_rpc` receives `MAX_QPS_LIMIT`. Consults coordinator for cooldown, builds QPS snapshot from registry, feeds to state machine, applies resulting actions to throttler. - `before_rpc()` / `after_rpc()`: Called around each load-related RPC for throttle enforcement and QPS recording. - Background tick thread: Runs every 1 second, advances coordinator by 1000 ticks. Triggers downgrade when enough time has passed without MS_BUSY. **Upgrade/Downgrade lifecycle example:** ``` Time 0s: MS returns MAX_QPS_LIMIT → Upgrade level 1: top-2 tables (A: 100 qps, B: 80 qps) → A limited to 50 qps, B limited to 40 qps Time 2s: MS returns MAX_QPS_LIMIT again (cooldown 5s not passed) → Skipped Time 6s: MS returns MAX_QPS_LIMIT (cooldown passed) → Upgrade level 2: top-2 tables now (A: 50 qps, C: 60 qps) → A limited to 25 qps, C limited to 30 qps Time 11s: No MS_BUSY for 5s → Downgrade: undo level 2 → A restored to 50 qps, C limit removed Time 16s: No MS_BUSY for 5s → Downgrade: undo level 1 → A limit removed, B limit removed ``` **Configuration:** | Config | Default | Mutable | Description | |--------|---------|---------|-------------| | `enable_ms_backpressure_handling` | `false` | Yes | Global enable/disable switch | | `ms_rpc_table_qps_window_sec` | `3` | No | bvar time window for QPS calculation | | `ms_backpressure_upgrade_interval_ms` | `3000` | Yes | Minimum cooldown between upgrades | | `ms_backpressure_upgrade_top_k` | `2` | Yes | Number of top tables to throttle per upgrade | | `ms_backpressure_throttle_ratio` | `0.75` | Yes | QPS decay ratio on upgrade | | `ms_rpc_table_qps_limit_floor` | `1.0` | Yes | Minimum QPS limit (won't throttle below this) | | `ms_backpressure_downgrade_interval_ms` | `3000` | Yes | Time without MS_BUSY before downgrade | **Observability (bvar metrics):** - `ms_rpc_backpressure_upgrade_count` / `_60s` — Upgrade event counts - `ms_rpc_backpressure_downgrade_count` / `_60s` — Downgrade event counts - `ms_rpc_backpressure_ms_busy_count` / `_60s` — MS_BUSY signal counts - `ms_rpc_backpressure_throttle_wait_<rpc_name>` — Per-RPC-type throttle wait latency - `ms_rpc_backpressure_throttled_tables_<rpc_name>` — Number of throttled tables per RPC type **New files:** - `be/src/cloud/cloud_throttle_state_machine.h` / `.cpp` - `be/src/cloud/cloud_ms_backpressure_handler.h` / `.cpp` - `be/test/cloud/cloud_throttle_state_machine_test.cpp` - `be/test/cloud/cloud_ms_backpressure_handler_test.cpp` **Also renamed (not part of the feature, cleanup):** - `common/cpp/s3_rate_limiter.h/.cpp` → `common/cpp/token_bucket_rate_limiter.h/.cpp` (more general naming since it's now used beyond S3) The table-level backpressure system operates transparently inside BE. When issues arise, users and DBAs have no way to inspect which tables are being throttled, what their QPS limits are, or what their current QPS is — beyond checking raw bvar metrics. Add a new system table `information_schema.backend_ms_rpc_table_throttlers` that exposes the real-time state of the `TableRpcThrottler` on each BE. This table is a **Backend-Partitioned Schema Table**, meaning each BE reports its own throttling data, and queries are distributed to all alive BEs and aggregated. **Schema:** | Column | Type | Description | |--------|------|-------------| | `BE_ID` | BIGINT | Backend ID | | `TABLE_ID` | BIGINT | Table ID being throttled | | `RPC_TYPE` | VARCHAR(64) | RPC type name (e.g., `PREPARE_ROWSET`, `COMMIT_ROWSET`) | | `QPS_LIMIT` | DOUBLE | Current QPS limit enforced on this (table, rpc) pair | | `CURRENT_QPS` | DOUBLE | Current observed QPS for this (table, rpc) pair | **Usage examples:** ```sql -- View all currently throttled tables across all BEs SELECT * FROM information_schema.backend_ms_rpc_table_throttlers; -- View throttled tables on a specific BE SELECT * FROM information_schema.backend_ms_rpc_table_throttlers WHERE BE_ID = 10001; -- Find the most severely throttled tables SELECT * FROM information_schema.backend_ms_rpc_table_throttlers ORDER BY QPS_LIMIT ASC; ``` None - Test <!-- At least one of them must be included. --> - [x] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [x] Yes. <!-- Explain the behavior change --> - Added host-level token-bucket rate limiting for all MS RPCs (enabled by default via `enable_ms_rpc_host_level_rate_limit`) - Added table-level adaptive backpressure handling triggered by MS `MAX_QPS_LIMIT` response (disabled by default via `enable_ms_backpressure_handling`)
hello-stephen
commented
Jun 11, 2026
Contributor
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
bobhan1
marked this pull request as ready for review
June 11, 2026 03:48
bobhan1
commented
Jun 11, 2026
ContributorAuthor
run buildall |
hello-stephen
commented
Jun 11, 2026
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
bobhan1
commented
Jun 11, 2026
ContributorAuthor
run feut |
bobhan1
commented
Jun 11, 2026
ContributorAuthor
run external |
1 similar comment
bobhan1
commented
Jun 11, 2026
ContributorAuthor
run external |
hello-stephen
commented
Jun 11, 2026
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
hello-stephen
commented
Jun 11, 2026
Contributor
FE Regression Coverage ReportIncrement line coverage |
hello-stephen
commented
Jun 11, 2026
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
hello-stephen
commented
Jun 11, 2026
Contributor
FE Regression Coverage ReportIncrement line coverage |
yiguolei
approved these changes
Jun 16, 2026
Uh oh!
There was an error while loading. Please reload this page.
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
Backport #60344 / eaa4ef9 to branch-4.1.
This adds BE-to-MS RPC throttling support, including:
Conflict resolution notes:
Validation
git diff HEAD^ HEAD --check./run-be-ut.sh --run --filter=HostLevelMSRpcRateLimitersTest.*:HostLevelMSRpcRateLimitersConfigTest.*:StrictQpsLimiterTest.*:TableRpcQpsRegistryTest.*:TableRpcThrottlerTest.*:MSBackpressureHandlerTest.*:LoadRelatedRpcTest.*:RpcThrottleStateMachineTest.*:RpcThrottleCoordinatorTest.*:RpcThrottleIntegrationTest.* -j32