Skip to content

branch-4.0: [fix](cloud) Fix warm up cache not supporting packed file (#60375) - #61887

Merged
yiguolei merged 1 commit into
apache:branch-4.0from
liaoxin01:cherry-pick-60375-to-branch-4.0
Apr 15, 2026
Merged

branch-4.0: [fix](cloud) Fix warm up cache not supporting packed file (#60375)#61887
yiguolei merged 1 commit into
apache:branch-4.0from
liaoxin01:cherry-pick-60375-to-branch-4.0

Conversation

@liaoxin01

Copy link
Copy Markdown
Contributor

Pick #60375

When enable_packed_file is enabled, the warm up functions fail to
download cache data because they use storage_resource->fs (the raw
FileSystem) instead of RowsetMeta::fs() which wraps the FileSystem with
PackedFileSystem.
The PackedFileSystem correctly maps segment file paths to their actual
locations within packed files. Without this mapping, the download fails
because the segment files don't exist on S3 - they are packed into
larger packed files.
This fix changes the file_system used in:
1. download_file_cache_block (block_file_cache_downloader.cpp)
2. _submit_segment_download_task (cloud_tablet.cpp)
3. _submit_inverted_index_download_task (cloud_tablet.cpp)
All now use RowsetMeta::fs() instead of storage_resource->fs to properly
handle packed files.
CopilotAI review requested due to automatic review settings March 30, 2026 09:00
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@liaoxin01

Copy link
Copy Markdown
ContributorAuthor

run buildall

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes cloud warm-up file cache downloads to correctly support packed files by using the PackedFileSystem-wrapped filesystem from rowset metadata, and extends regression coverage to run warm-up/cache/tablet-cleanup suites with packed files toggled.

Changes:

  • Switch warm-up download paths to use RowsetMeta::fs() (PackedFileSystem-aware) instead of storage_resource->fs.
  • Update multiple cloud P0 regression suites to run with enable_packed_file enabled/disabled.
  • Add BE config injection for enable_packed_file across the impacted suites.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 18 comments.

Show a summary per file
FileDescription
regression-test/suites/cloud_p0/tablets/test_clean_tablet_when_rebalance.groovyRandomly toggles enable_packed_file and passes it into BE configs for rebalance cleanup coverage.
regression-test/suites/cloud_p0/tablets/test_clean_tablet_when_drop_force_table.groovyRandomly toggles enable_packed_file and passes it into BE configs for drop-force cleanup coverage.
regression-test/suites/cloud_p0/tablets/test_clean_stale_rs_index_file_cache.groovyRandomly toggles enable_packed_file and passes it into BE configs for stale index cache cleanup coverage.
regression-test/suites/cloud_p0/tablets/test_clean_stale_rs_file_cache.groovyRandomly toggles enable_packed_file and passes it into BE configs for stale segment cache cleanup coverage.
regression-test/suites/cloud_p0/cache/test_topn_broadcast.groovyRandomly toggles enable_packed_file and passes it into BE configs for cache behavior coverage.
regression-test/suites/cloud_p0/cache/test_load_cache.groovyRandomly toggles enable_packed_file and passes it into BE configs for load/query cache behavior coverage.
regression-test/suites/cloud_p0/balance/test_warmup_rebalance.groovyRandomly toggles enable_packed_file and applies it to BE configs for rebalance warm-up coverage.
regression-test/suites/cloud_p0/balance/test_peer_read_async_warmup.groovyRandomly toggles enable_packed_file and applies it to BE configs for peer-read warm-up coverage.
regression-test/suites/cloud_p0/balance/test_expanding_node_balance.groovyRandomly toggles enable_packed_file and applies it to BE configs across multiple clusters.
regression-test/suites/cloud_p0/balance/test_balance_warm_up_with_compaction_use_peer_cache.groovyRandomly toggles enable_packed_file and applies it to BE configs for warm-up + compaction scenario.
regression-test/suites/cloud_p0/balance/test_balance_warm_up_use_peer_cache.groovyRandomly toggles enable_packed_file and applies it to BE configs for warm-up using peer cache.
regression-test/suites/cloud_p0/balance/test_balance_warm_up_task_abnormal.groovyRandomly toggles enable_packed_file and applies it to BE configs for abnormal warm-up task behavior.
regression-test/suites/cloud_p0/balance/test_balance_warm_up_sync_global_config.groovyRandomly toggles enable_packed_file and applies it to BE configs for sync warm-up config path.
regression-test/suites/cloud_p0/balance/test_balance_warm_up.groovyRandomly toggles enable_packed_file and applies it to BE configs for baseline warm-up balance coverage.
regression-test/suites/cloud_p0/balance/test_balance_use_compute_group_properties.groovyRandomly toggles enable_packed_file and applies it to BE configs for compute group property interactions.
regression-test/suites/cloud_p0/balance/test_balance_metrics.groovyRandomly toggles enable_packed_file and applies it to BE configs for balance metrics assertions.
regression-test/suites/cloud_p0/balance/test_alter_compute_group_properties.groovyRandomly toggles enable_packed_file and applies it to BE configs for compute group alter flows.
be/src/io/cache/block_file_cache_downloader.cppUses RowsetMeta::fs() to ensure warm-up downloads resolve packed segment paths correctly.
be/src/cloud/cloud_tablet.cppUses rowset_meta->fs() when submitting warm-up download tasks so packed files are handled correctly.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@doris-robot

Copy link
Copy Markdown

BE UT Coverage Report

Increment line coverage 0.00% (0/7) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage52.95% (19211/36284)
Line Coverage36.12% (179009/495571)
Region Coverage32.74% (138757/423778)
Branch Coverage33.71% (60284/178808)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/16) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage71.31% (25331/35521)
Line Coverage54.03% (267301/494702)
Region Coverage51.66% (221136/428074)
Branch Coverage53.12% (95329/179453)

@liaoxin01liaoxin01 changed the title [fix](cloud) Fix warm up cache not supporting packed file (#60375)branch-4.0: [fix](cloud) Fix warm up cache not supporting packed file (#60375)Apr 3, 2026
@yiguolei
yiguolei merged commit d6ae1ea into apache:branch-4.0Apr 15, 2026
32 of 36 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@liaoxin01@hello-stephen@doris-robot@yiguolei