branch-4.1: pick some prs by bobhan1 · Pull Request #61981 · apache/doris

bobhan1 · 2026-04-01T03:04:13Z

pick [Opt](cloud) Notify BE to make committed rowset visible directly after txn commit without fetching from meta service #59754
pick [Opt](freshness tolerance) Continue to capture rowsets when the rowset is not in _rowset_warm_up_states #61238

Thearas · 2026-04-01T03:04:19Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

bobhan1 · 2026-04-01T03:05:36Z

run buildall

bobhan1 · 2026-04-01T03:09:14Z

run buildall

bobhan1 · 2026-04-01T03:48:49Z

run buildall

hello-stephen · 2026-04-01T04:15:19Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	78.40% (1793/2287)
Line Coverage	64.06% (32104/50113)
Region Coverage	64.91% (16085/24781)
Branch Coverage	55.43% (8568/15456)

JNSimba · 2026-04-01T06:04:48Z

run buildall

…r txn commit without fetching from meta service (apache#59754) 1FE+3BE, 400bucket mow table, 10 concurrency stream load before: <img width="3218" height="1300" alt="image" src="https://github.com/user-attachments/assets/3d4fe51d-4fae-484b-a257-a6b5ec010970" /> after: <img width="3206" height="1304" alt="image" src="https://github.com/user-attachments/assets/e7230b3c-999d-407b-aa57-783492995e5b" /> --- Problem Summary In cloud storage mode, after a load transaction commits rowsets to Meta Service (MS), BE nodes need to fetch the committed rowset metadata from MS during subsequent sync_rowsets() operations. This introduces additional latency and MS query overhead, especially for high-frequency import scenarios. This PR implements a notification mechanism where FE/BE directly notifies BE nodes about committed rowsets after transaction commit, allowing BE to update tablet metadata immediately without fetching from MS. Solution 1. CloudCommittedRSMgr (BE): A new in-memory manager that caches committed temporary rowset metadata after they are committed to MS - Stores rowset meta with expiration time - Supports efficient lookup by (txn_id, tablet_id) - Handles empty rowset markers - Background thread for cleaning expired entries 2. Notification Flow: - When FE commits a load transaction to MS, it sends MakeCloudTmpRsVisibleTask to involved BE nodes - BE receives the task with final version/visible_ts information - BE promotes cached rowset metadata to tablet meta directly - BE can forward notification to FE if needed (for tablets not on the original BE) 3. Configuration: - FE: enable_cloud_notify_be_after_load_txn_commit (default: false) - BE: enable_cloud_make_rs_visible_on_be (default: false) - BE: cloud_make_committed_rs_visible_worker_count (default: 4) - BE: cloud_mow_sync_rowsets_when_load_txn_begin (default: true) Benefits - Reduces Meta Service query pressure by avoiding redundant rowset metadata fetches - Improves import latency by making rowsets visible faster - Optimizes cloud storage mode performance for high-frequency imports

…t is not in `_rowset_warm_up_states` (apache#61238) In the freshness tolerance query path, when a BE restarts during rowset warmup, the warmup requests from the upstream BE are lost, leaving some rowsets with no entry in `_rowset_warm_up_states`. Previously, `is_rowset_warmed_up()` returned `false` for such rowsets, treating them as "not warmed up". This becomes problematic for **compaction-produced rowsets** whose `visible_timestamp` is set at rowset builder initialization time rather than at the final transaction commit time on meta-service. Their `visible_timestamp` can be **earlier** than `startup_timepoint`, causing the `startup_timepoint` filter to NOT skip them — they then reach `is_rowset_warmed_up()` with no warmup entry. If such a rowset sits before the cumulative compaction point and base compaction never happens, returning `false` causes the version path algorithm to exclude it, leading to a **persistently low `path_max_version`**. With continuous upstream ingestion, the freshness tolerance fallback check keeps triggering, making **every query on this tablet fall back to reading all data from remote storage** — effectively defeating the cache entirely. ### Solution Change `is_rowset_warmed_up()` to return `true` (optimistically treat as warmed up) when a rowset has no entry in `_rowset_warm_up_states`. This allows the version path algorithm to include such rowsets normally. On cache miss, data is transparently read from remote storage per-segment and cached locally in 1MB blocks, so the problem **self-heals** through subsequent queries. A bvar counter (`rowset_warmup_state_missing_count`) and a throttled WARNING log are added for observability. ### Changes - `CloudTablet::is_rowset_warmed_up()`: return `true` instead of `false` when the rowset is not found in `_rowset_warm_up_states` - Add `rowset_warmup_state_missing_count` bvar for monitoring: counts the number of times a rowset's warmup state is missing from `_rowset_warm_up_states`. A non-zero value indicates that some rowsets lost their warmup entries (e.g. due to BE restart during warmup) and were optimistically treated as warmed up. Sustained growth may indicate frequent BE restarts or warmup instability. - Add `add_not_warmed_up_rowset()` test helper to explicitly mark rowsets as not warmed up (DOING state) for unit tests - Fix existing UTs that relied on absence from the warmup map to mean "not warmed up"

bobhan1 · 2026-04-01T06:14:34Z

run buildall

doris-robot · 2026-04-01T06:54:52Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	78.47% (1797/2290)
Line Coverage	64.17% (32266/50285)
Region Coverage	65.05% (16174/24863)
Branch Coverage	55.51% (8617/15524)

bobhan1 · 2026-04-01T10:31:02Z

run beut

bobhan1 · 2026-04-01T10:31:07Z

run p0

bobhan1 · 2026-04-01T10:31:13Z

run external

bobhan1 · 2026-04-01T11:51:47Z

run nonConcurrent

hello-stephen · 2026-04-01T12:16:24Z

BE UT Coverage Report

Increment line coverage 41.15% (165/401) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.92% (19789/37397)
Line Coverage	36.39% (185765/510525)
Region Coverage	32.68% (143982/440624)
Branch Coverage	33.84% (63179/186679)

hello-stephen · 2026-04-01T12:25:35Z

BE Regression && UT Coverage Report

Increment line coverage 87.68% (356/406) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.09% (26754/36605)
Line Coverage	56.49% (287441/508820)
Region Coverage	54.13% (240676/444585)
Branch Coverage	55.69% (104256/187207)

hello-stephen · 2026-04-01T14:33:50Z

BE Regression && UT Coverage Report

Increment line coverage 87.68% (356/406) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.13% (26771/36605)
Line Coverage	56.53% (287645/508820)
Region Coverage	54.19% (240904/444585)
Branch Coverage	55.73% (104337/187207)

bobhan1 requested a review from yiguolei as a code owner April 1, 2026 03:04

bobhan1 changed the title ~~branch-4.1: [Opt](cloud) Notify BE to make committed rowset visible directly after txn commit without fetching from meta service (#59754)~~branch-4.1: pick some prsApr 1, 2026

bobhan1 added 3 commits April 1, 2026 14:14

fix
7c28474

bobhan1 force-pushed the branch-4.1-pick-59754 branch from ccba3b5 to 7c28474Compare April 1, 2026 06:14

yiguolei approved these changes Apr 2, 2026
View reviewed changes

yiguolei merged commit f52cd31 into apache:branch-4.1Apr 2, 2026
25 of 28 checks passed

Uh oh!

Conversation

bobhan1 commented Apr 1, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Thearas commented Apr 1, 2026

Uh oh!

bobhan1 commented Apr 1, 2026

Uh oh!

bobhan1 commented Apr 1, 2026

Uh oh!

bobhan1 commented Apr 1, 2026

Uh oh!

hello-stephen commented Apr 1, 2026

Cloud UT Coverage Report

Uh oh!

JNSimba commented Apr 1, 2026

Uh oh!

bobhan1 commented Apr 1, 2026

Uh oh!

doris-robot commented Apr 1, 2026

Cloud UT Coverage Report

Uh oh!

bobhan1 commented Apr 1, 2026

Uh oh!

bobhan1 commented Apr 1, 2026

Uh oh!

bobhan1 commented Apr 1, 2026

Uh oh!

bobhan1 commented Apr 1, 2026

Uh oh!

hello-stephen commented Apr 1, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Apr 1, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Apr 1, 2026

BE Regression && UT Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

bobhan1 commented Apr 1, 2026•
edited
Loading