Skip to content

[Enhancement](file-cache) Add fine-grained control for compaction file cache - #62119

Merged
yiguolei merged 4 commits into
apache:branch-4.1from
gavinchou:pick-60609-to-branch-4.1
Apr 5, 2026
Merged

[Enhancement](file-cache) Add fine-grained control for compaction file cache#62119
yiguolei merged 4 commits into
apache:branch-4.1from
gavinchou:pick-60609-to-branch-4.1

Conversation

@gavinchou

@gavinchougavinchou commented Apr 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Pick #60609 to branch-4.1

Support selective caching of index files only during compaction in cloud mode.

Changes:

  • Add two mBool configs to control index-only caching for base and cumulative compaction:

    • enable_base_compaction_output_write_index_only (default: false)
    • enable_cumu_compaction_output_write_index_only (default: false)

  • Extend RowsetWriterContext with compaction_output_write_index_only field to mark whether only index files should be cached

  • Modify get_file_writer_options() to accept is_index_file parameter:

    • When compaction_output_write_index_only=true and is_index_file=false, set write_file_cache=false to skip caching data files
    • Index files continue to be cached normally

  • Update file writer creation call sites to pass is_index_file parameter:

    • Index file writers: pass true
    • Segment (data) file writers: pass false

Benefits:

  • Reduces cache pressure by avoiding caching large data files during compaction
  • Preserves index file caching for query performance
  • Provides separate control for base and cumulative compaction strategies
  • Maintains backward compatibility with default settings

@gavinchougavinchou
[Enhancement](file-cache) Add fine-grained control for compaction fil…
64dea4f
…e cache
Support selective caching of index files only during compaction in cloud mode.
Changes:
- Add two mBool configs to control index-only caching for base and cumulative compaction:
* enable_base_compaction_output_write_index_only (default: false)
* enable_cumu_compaction_output_write_index_only (default: false)
- Extend RowsetWriterContext with compaction_output_write_index_only field
to mark whether only index files should be cached
- Modify get_file_writer_options() to accept is_index_file parameter:
* When compaction_output_write_index_only=true and is_index_file=false,
set write_file_cache=false to skip caching data files
* Index files continue to be cached normally
- Update file writer creation call sites to pass is_index_file parameter:
* Index file writers: pass true
* Segment (data) file writers: pass false
Benefits:
- Reduces cache pressure by avoiding caching large data files during compaction
- Preserves index file caching for query performance
- Provides separate control for base and cumulative compaction strategies
- Maintains backward compatibility with default settings
@gavinchougavinchou
Add UT
744e8ca
@gavinchougavinchou
Rename conf
c85c656
@gavinchou
gavinchou requested a review from yiguolei as a code ownerApril 4, 2026 09:51
@hello-stephen

hello-stephen commented Apr 4, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@gavinchou

gavinchou commented Apr 4, 2026

Copy link
Copy Markdown
ContributorAuthor

run buildall

1 similar comment
@gavinchou

gavinchou commented Apr 4, 2026

Copy link
Copy Markdown
ContributorAuthor

run buildall

gavinchou
[Fix] Fix compilation issues for branch-4.1
1ab0260
1. Update parquet API: enable_force_write_int96_timestamps -> enable_deprecated_int96_timestamps
2. Fix test include paths: olap/ -> storage/ (file location changed in branch-4.1)
@gavinchou

gavinchou commented Apr 4, 2026

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Apr 4, 2026

Copy link
Copy Markdown

BE UT Coverage Report

Increment line coverage 85.71% (30/35) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage52.95% (19808/37409)
Line Coverage36.43% (186123/510846)
Region Coverage32.72% (144275/440988)
Branch Coverage33.89% (63331/186872)

@hello-stephen

hello-stephen commented Apr 4, 2026

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (35/35) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage71.28% (26101/36617)
Line Coverage54.14% (275666/509141)
Region Coverage51.67% (229895/444949)
Branch Coverage52.96% (99255/187400)

@yiguolei

yiguolei commented Apr 5, 2026

Copy link
Copy Markdown
Contributor

skip buildall

@yiguolei
yiguolei merged commit 9617314 into apache:branch-4.1Apr 5, 2026
28 of 30 checks passed
@hubgeterhubgeter mentioned this pull request Apr 10, 2026
Merged
29 tasks
yiguolei pushed a commit that referenced this pull request Apr 11, 2026
@hubgeter
branch-4.1:[fix](parquet)fix parquet write timestamp int96 type. (2/2) (
682bcdc
#61832) (#62319)
bp #61832
### What problem does this PR solve?
Problem Summary:
pick #61832 again, since #62119 reverted the changes made to #61847.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change. - [ ] No code files have been
changed. - [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
kaka11chen pushed a commit to kaka11chen/doris that referenced this pull request Apr 11, 2026
@hubgeter@kaka11chen
branch-4.1:[fix](parquet)fix parquet write timestamp int96 type. (2/2) (
363ff5d
apache#61832) (apache#62319)
bp apache#61832
### What problem does this PR solve?
Problem Summary:
pick apache#61832 again, since apache#62119 reverted the changes made to apache#61847.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change. - [ ] No code files have been
changed. - [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguoleiyiguoleiyiguolei approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@gavinchou@hello-stephen@doris-robot@yiguolei