Uh oh!
There was an error while loading. Please reload this page.
[Performance](clould) reduce tablet cache read in topn query - #50803
Merged
Conversation
hello-stephen
commented
May 12, 2025
Contributor
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
HappenLee
commented
May 12, 2025
ContributorAuthor
run buildall |
doris-robot
commented
May 12, 2025
TPC-H: Total hot run time: 33804 ms |
HappenLee
commented
May 12, 2025
ContributorAuthor
run buildall |
hello-stephen
commented
May 12, 2025
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
hello-stephen
commented
May 12, 2025
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
doris-robot
commented
May 12, 2025
TPC-H: Total hot run time: 34672 ms |
doris-robot
commented
May 12, 2025
TPC-DS: Total hot run time: 193854 ms |
doris-robot
commented
May 12, 2025
ClickBench: Total hot run time: 29.4 s |
HappenLee
marked this pull request as ready for review
May 12, 2025 15:50
Contributor
PR approved by at least one committer and no changes requested. |
Contributor
PR approved by anyone and no changes requested. |
Uh oh!
There was an error while loading. Please reload this page.
eldenmoon
self-requested a review
May 13, 2025 02:44
Contributor
PR approved by at least one committer and no changes requested. |
gavinchou
approved these changes
May 26, 2025
github-actionsBot
pushed a commit
that referenced
this pull request
May 26, 2025
Topn lazy read only read the cache in cloud mode to reduce tablet cache read.
koarz pushed a commit
to koarz/doris
that referenced
this pull request
Jun 4, 2025
…50803) Topn lazy read only read the cache in cloud mode to reduce tablet cache read.
Closed
Hastyshell pushed a commit
to Hastyshell/doris
that referenced
this pull request
Jul 21, 2025
…50803) Topn lazy read only read the cache in cloud mode to reduce tablet cache read.
freemandealer added a commit
to freemandealer/doris
that referenced
this pull request
Nov 14, 2025
Topn lazy read only read the cache in cloud mode to reduce tablet cache read. fixapache#50803 failed to pass force_use_only_cached to tablet_mgr Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
freemandealer added a commit
to freemandealer/doris
that referenced
this pull request
Nov 14, 2025
Topn lazy read only read the cache in cloud mode to reduce tablet cache read. fixapache#50803 failed to pass force_use_only_cached to tablet_mgr Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
16 tasks
freemandealer added a commit
to freemandealer/doris
that referenced
this pull request
Nov 18, 2025
Topn lazy read only read the cache in cloud mode to reduce tablet cache read. fixapache#50803 failed to pass force_use_only_cached to tablet_mgr Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
freemandealer added a commit
to freemandealer/doris
that referenced
this pull request
Nov 18, 2025
Topn lazy read only read the cache in cloud mode to reduce tablet cache read. fixapache#50803 failed to pass force_use_only_cached to tablet_mgr Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
freemandealer added a commit
to freemandealer/doris
that referenced
this pull request
Nov 18, 2025
Topn lazy read only read the cache in cloud mode to reduce tablet cache read. fixapache#50803 failed to pass force_use_only_cached to tablet_mgr Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
16 tasks
dataroaring pushed a commit
that referenced
this pull request
Nov 19, 2025
Topn lazy read only read the cache in cloud mode to reduce tablet cache read. fix#50803 failed to pass force_use_only_cached to tablet_mgr Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
github-actionsBot
pushed a commit
that referenced
this pull request
Nov 19, 2025
Topn lazy read only read the cache in cloud mode to reduce tablet cache read. fix#50803 failed to pass force_use_only_cached to tablet_mgr Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
github-actionsBot
pushed a commit
that referenced
this pull request
Nov 19, 2025
Topn lazy read only read the cache in cloud mode to reduce tablet cache read. fix#50803 failed to pass force_use_only_cached to tablet_mgr Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
nagisa-kunhah pushed a commit
to nagisa-kunhah/doris
that referenced
this pull request
Dec 14, 2025
Topn lazy read only read the cache in cloud mode to reduce tablet cache read. fixapache#50803 failed to pass force_use_only_cached to tablet_mgr Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
w41ter pushed a commit
to w41ter/incubator-doris
that referenced
this pull request
Dec 26, 2025
…pache#5786) pick: apache#58044 Topn lazy read only read the cache in cloud mode to reduce tablet cache read. fixapache#50803 failed to pass force_use_only_cached to tablet_mgr Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
Hastyshell pushed a commit
to Hastyshell/doris
that referenced
this pull request
Dec 29, 2025
…pache#5887) Topn lazy read only read the cache in cloud mode to reduce tablet cache read. fixapache#50803 failed to pass force_use_only_cached to tablet_mgr Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
seawinde pushed a commit
to seawinde/doris
that referenced
this pull request
Jan 30, 2026
…pache#5887) Topn lazy read only read the cache in cloud mode to reduce tablet cache read. fixapache#50803 failed to pass force_use_only_cached to tablet_mgr Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
uchenily added a commit
to uchenily/doris
that referenced
this pull request
Aug 19, 2026
…s in topn deferred materialization In cloud mode, the old topn deferred materialization path (DeferMaterializeTopNResult, enabled when topn_lazy_materialization_threshold=0) broadcasts a multiget_data RPC to all BEs in the compute group. Each BE uses force_use_only_cached=true to only read the local tablet cache, avoiding meta-service RPCs on BEs that do not hold the tablet (optimization introduced by apache#50803, parameter-passing bug fixed by apache#58044). This optimization assumes the tablet cache is enabled and that the BE which performed the scan phase still has the tablet cached by the time the deferred-materialization RPC arrives. When this assumption breaks, every BE returns 0 rows and the coordinator reports: Miss matched return row loc count 0, expected 0, input N Two scenarios break the assumption: 1. tablet_cache_capacity=0: CloudTabletMgr falls back to DummyLRUCache (every insert and lookup is a no-op), so force_use_only_cached always misses -- even on the BE that performed the scan. This is a silent misconfiguration with no visible warning. 2. Cache eviction under a small tablet_cache_capacity: in a multi-BE cluster the scan phase loads the tablet into the scanning BE's cache, but between the scan and the deferred- materialization RPC other concurrent queries may load enough tablets to evict it. The scanning BE then misses just like the non-scanning BEs, and since it is the only BE that holds the rowset data, the coordinator cannot recover the rows. (In a normal multi-BE cluster, BEs that did not perform the scan are *expected* to miss the cache and skip -- the coordinator merges results from the scanning BE. The bug only manifests when the scanning BE itself misses, which happens with cache disabled or evicted.) This patch makes the old path resilient without giving up the apache#50803 optimization: rowid_fetcher.cpp: when force_use_only_cached=true misses, fall back to a normal get_tablet() load (force_use_only_cached=false). The cache-only fast path is still attempted first, so the broadcast-remote-read avoidance is preserved when the cache hits. The fallback only pays a meta-service RPC on cache miss, which is the same behavior as the new multiget_data_v2 path (apache#55870). Reproduction ------------ Environment: single-node cloud-mode cluster CREATE TABLE t (k int, v int) DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES("replication_num"="1"); INSERT INTO t VALUES (1, 1), (2, 2); Set tablet_cache_capacity=0 in be_custom.conf and restart BE. Before fix: mysql> SET enable_sql_cache=false; mysql> SET topn_lazy_materialization_threshold=0; mysql> SELECT k,v FROM t ORDER BY k LIMIT 10; ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[INTERNAL_ERROR] Miss matched return row loc count 0, expected 0, input 2 After fix: mysql> SET enable_sql_cache=false; mysql> SET topn_lazy_materialization_threshold=0; mysql> SELECT k,v FROM t ORDER BY k LIMIT 10; +------+------+ | k | v | +------+------+ | 1 | 1 | | 2 | 2 | +------+------+
16 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Topn lazy read only read the cache in cloud mode to reduce tablet cache read.
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)