Uh oh!
There was an error while loading. Please reload this page.
[Feature](Cloud) Support session variable disable_file_cache and enable_segment_cache in query - #37141
Merged
Merged
Conversation
doris-robot
commented
Jul 2, 2024
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
wangshuo128force-pushed
the
disable_file_cache
branch
from
July 2, 2024 08:23
2955ce5 to
39bcd74CompareContributor
clang-tidy review says "All clean, LGTM! 👍" |
3 similar comments
Contributor
clang-tidy review says "All clean, LGTM! 👍" |
Contributor
clang-tidy review says "All clean, LGTM! 👍" |
Contributor
clang-tidy review says "All clean, LGTM! 👍" |
wangshuo128force-pushed
the
disable_file_cache
branch
from
July 4, 2024 05:03
e609c4e to
d31abdfCompareContributor
clang-tidy review says "All clean, LGTM! 👍" |
wangshuo128
commented
Jul 4, 2024
ContributorAuthor
run buildall |
doris-robot
commented
Jul 4, 2024
TPC-H: Total hot run time: 39896 ms |
wangshuo128force-pushed
the
disable_file_cache
branch
from
July 4, 2024 08:28
d31abdf to
d878919CompareContributor
clang-tidy review says "All clean, LGTM! 👍" |
doris-robot
commented
Jul 4, 2024
TPC-DS: Total hot run time: 172155 ms |
wangshuo128force-pushed
the
disable_file_cache
branch
from
July 4, 2024 08:41
d878919 to
d544243Comparedoris-robot
commented
Jul 4, 2024
ClickBench: Total hot run time: 30.85 s |
Contributor
clang-tidy review says "All clean, LGTM! 👍" |
wangshuo128force-pushed
the
disable_file_cache
branch
from
July 4, 2024 11:45
d544243 to
09e731cCompareContributor
clang-tidy review says "All clean, LGTM! 👍" |
wangshuo128force-pushed
the
disable_file_cache
branch
from
July 4, 2024 12:05
09e731c to
66dedebComparewangshuo128
commented
Jul 4, 2024
ContributorAuthor
run buildall |
Contributor
clang-tidy review says "All clean, LGTM! 👍" |
doris-robot
commented
Jul 4, 2024
TPC-H: Total hot run time: 39752 ms |
doris-robot
commented
Jul 4, 2024
TPC-DS: Total hot run time: 172565 ms |
doris-robot
commented
Jul 4, 2024
ClickBench: Total hot run time: 30 s |
wangshuo128
commented
Jul 8, 2024
ContributorAuthor
run buildall |
doris-robot
commented
Jul 8, 2024
TPC-H: Total hot run time: 39910 ms |
doris-robot
commented
Jul 8, 2024
TPC-DS: Total hot run time: 173427 ms |
doris-robot
commented
Jul 8, 2024
ClickBench: Total hot run time: 30.47 s |
wangshuo128
commented
Jul 9, 2024
ContributorAuthor
run p0 |
gavinchou
commented
Aug 2, 2024
Contributor
run buildall |
gavinchou
approved these changes
Aug 2, 2024
Contributor
PR approved by at least one committer and no changes requested. |
Contributor
clang-tidy review says "All clean, LGTM! 👍" |
yagagagaga
approved these changes
Aug 2, 2024
Yukang-Lian
approved these changes
Aug 2, 2024
doris-robot
commented
Aug 2, 2024
TPC-H: Total hot run time: 41916 ms |
doris-robot
commented
Aug 2, 2024
TPC-DS: Total hot run time: 169551 ms |
doris-robot
commented
Aug 2, 2024
ClickBench: Total hot run time: 30.39 s |
gavinchou
commented
Aug 5, 2024
Contributor
run buildall |
doris-robot
commented
Aug 5, 2024
TPC-H: Total hot run time: 41931 ms |
doris-robot
commented
Aug 5, 2024
TPC-DS: Total hot run time: 168324 ms |
doris-robot
commented
Aug 5, 2024
ClickBench: Total hot run time: 30.89 s |
gavinchou added a commit
to gavinchou/doris
that referenced
this pull request
Aug 8, 2024
…apache#37141 Session variable `disable_file_cache` is processed as "disposable file cache" in beta_rowset_reader.cpp. ``` if (_read_context->runtime_state != nullptr) { _read_options.io_ctx.query_id = &_read_context->runtime_state->query_id(); _read_options.io_ctx.read_file_cache = _read_context->runtime_state->query_options().enable_file_cache; _read_options.io_ctx.is_disposable = _read_context->runtime_state->query_options().disable_file_cache; } ``` We use disposable cache to avoid IO amp and avoid large amount of eviction from the cached data ("normal cache"). We cannot set the read option cache policy to "no cache" because it may cause IO amp: every page IO will cause a remote IO, which is a performance disaster.
gavinchou added a commit
that referenced
this pull request
Aug 9, 2024
…#37141 (#39123) Session variable `disable_file_cache` is processed as "disposable file cache" in beta_rowset_reader.cpp. ``` if (_read_context->runtime_state != nullptr) { _read_options.io_ctx.query_id = &_read_context->runtime_state->query_id(); _read_options.io_ctx.read_file_cache = _read_context->runtime_state->query_options().enable_file_cache; _read_options.io_ctx.is_disposable = _read_context->runtime_state->query_options().disable_file_cache; } ``` We use disposable cache to avoid IO amp and avoid large amount of eviction from the cached data ("normal cache"). We cannot set the read option cache policy to "no cache" because it may cause IO amp: every page IO will cause a remote IO, which is a performance disaster.
wyxxxcat pushed a commit
to wyxxxcat/doris
that referenced
this pull request
Aug 14, 2024
…apache#37141 (apache#39123) Session variable `disable_file_cache` is processed as "disposable file cache" in beta_rowset_reader.cpp. ``` if (_read_context->runtime_state != nullptr) { _read_options.io_ctx.query_id = &_read_context->runtime_state->query_id(); _read_options.io_ctx.read_file_cache = _read_context->runtime_state->query_options().enable_file_cache; _read_options.io_ctx.is_disposable = _read_context->runtime_state->query_options().disable_file_cache; } ``` We use disposable cache to avoid IO amp and avoid large amount of eviction from the cached data ("normal cache"). We cannot set the read option cache policy to "no cache" because it may cause IO amp: every page IO will cause a remote IO, which is a performance disaster.
gavinchou added a commit
that referenced
this pull request
Aug 16, 2024
…le_segment_cache in query (#37141) Currently, whether to read from file cache or remote storage is controlled by the BE config `enable_file_cache` in cloud mode. This PR proposed to control the file cache behavior via session variables when executing queries in cloud mode. It's more convenient when have such a session variable, cache behavior could be controlled per query/session without changing BE configs, such as: 1. **Performance test**. Test the query performance when read from local file cache or remote storage for queries. 2. **Data correctness**. Check if it's file cache issue for certain tables or queries. The read path has three kinds of caches: segment cache, page cache and file cache. | module | cache| BE config | session variable| |------------|------|----------| ---- | | Segment | segment cache | disable_segment_cache | **enable_segment_cache** (supportted by this PR) | | PageIO | page cache | disable_storage_page_cache | enable_page_cache | | FileReader | file cache | enable_file_cache | **disable_file_cache** (supportted by this PR) | The modification of the PR: - **enable_segment_cache**: add a new session variable enable_segment_cache to control use segment cache or not. - **disable_file_cache**: disable_file_cache was for write path in cloud mode. It's supported for read path when executing queries in the PR. With this PR, data is read from remote storage without cache: ```sql set enable_segment_cache=false; set enable_page_cache=false; set disable_file_cache=true; ``` Co-authored-by: Gavin Chou <gavineaglechou@gmail.com>
gavinchou added a commit
that referenced
this pull request
Aug 16, 2024
…#37141 (#39123) Session variable `disable_file_cache` is processed as "disposable file cache" in beta_rowset_reader.cpp. ``` if (_read_context->runtime_state != nullptr) { _read_options.io_ctx.query_id = &_read_context->runtime_state->query_id(); _read_options.io_ctx.read_file_cache = _read_context->runtime_state->query_options().enable_file_cache; _read_options.io_ctx.is_disposable = _read_context->runtime_state->query_options().disable_file_cache; } ``` We use disposable cache to avoid IO amp and avoid large amount of eviction from the cached data ("normal cache"). We cannot set the read option cache policy to "no cache" because it may cause IO amp: every page IO will cause a remote IO, which is a performance disaster.
Closed
HappenLee pushed a commit
to HappenLee/incubator-doris
that referenced
this pull request
Apr 24, 2026
…le_segment_cache in query (apache#37141) Currently, whether to read from file cache or remote storage is controlled by the BE config `enable_file_cache` in cloud mode. This PR proposed to control the file cache behavior via session variables when executing queries in cloud mode. It's more convenient when have such a session variable, cache behavior could be controlled per query/session without changing BE configs, such as: 1. **Performance test**. Test the query performance when read from local file cache or remote storage for queries. 2. **Data correctness**. Check if it's file cache issue for certain tables or queries. The read path has three kinds of caches: segment cache, page cache and file cache. | module | cache| BE config | session variable| |------------|------|----------| ---- | | Segment | segment cache | disable_segment_cache | **enable_segment_cache** (supportted by this PR) | | PageIO | page cache | disable_storage_page_cache | enable_page_cache | | FileReader | file cache | enable_file_cache | **disable_file_cache** (supportted by this PR) | The modification of the PR: - **enable_segment_cache**: add a new session variable enable_segment_cache to control use segment cache or not. - **disable_file_cache**: disable_file_cache was for write path in cloud mode. It's supported for read path when executing queries in the PR. With this PR, data is read from remote storage without cache: ```sql set enable_segment_cache=false; set enable_page_cache=false; set disable_file_cache=true; ``` Co-authored-by: Gavin Chou <gavineaglechou@gmail.com>
HappenLee pushed a commit
to HappenLee/incubator-doris
that referenced
this pull request
Apr 24, 2026
…apache#37141 (apache#39123) Session variable `disable_file_cache` is processed as "disposable file cache" in beta_rowset_reader.cpp. ``` if (_read_context->runtime_state != nullptr) { _read_options.io_ctx.query_id = &_read_context->runtime_state->query_id(); _read_options.io_ctx.read_file_cache = _read_context->runtime_state->query_options().enable_file_cache; _read_options.io_ctx.is_disposable = _read_context->runtime_state->query_options().disable_file_cache; } ``` We use disposable cache to avoid IO amp and avoid large amount of eviction from the cached data ("normal cache"). We cannot set the read option cache policy to "no cache" because it may cause IO amp: every page IO will cause a remote IO, which is a performance disaster.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
Currently, whether to read from file cache or remote storage is controlled by the BE config
enable_file_cachein cloud mode.This PR proposed to control the file cache behavior via session variables when executing queries in cloud mode.
It's more convenient when have such a session variable, cache behavior could be controlled per query/session without changing BE configs, such as:
The read path has three kinds of caches: segment cache, page cache and file cache.
The modification of the PR:
With this PR, data is read from remote storage without cache: