Uh oh!
There was an error while loading. Please reload this page.
branch-4.1: [fix](cloud) Fix reading packed inverted index file on file cache miss - #64410
Merged
yiguolei merged 1 commit intoJun 11, 2026
Merged
Conversation
apache#64383) When `enable_packed_file` is enabled (cloud mode), the first segment's inverted index file (`{rowset_id}_{seg}.idx`) is packed into a shared file instead of being written as a standalone object on remote storage. At read time, `Segment::_open_index_file_reader()` derived the `.idx` path prefix from `_file_reader->path()`. The remote file reader normalizes this to an **absolute** path (e.g. `s3://bucket/instance_prefix/data/{tablet}/{rowset}_{seg}.idx`). But `PackedFileSystem`'s index map is keyed by **relative** paths (`data/{tablet}/{rowset}_{seg}.idx`, exactly as recorded by `CloudRowsetWriter` at write time). The absolute lookup key therefore never matched the relative map key, so `PackedFileSystem::open_file_impl()` fell through to reading the `.idx` as a **standalone object**, which does not exist (the data lives inside the packed file). The read failed with: ``` [E-6002]CLuceneError occur when init idx file s3://.../{rowset}_{seg}.idx, error msg: read past EOF ``` (`read past EOF` is how the S3 `NOT_FOUND`/404 is surfaced by `FSIndexInput::readInternal`.) The failure was masked by the local file cache, whose key is filename based: a warm-up read (which uses the relative/packed path) populates the cache, and a subsequent query hits it. So the bug only surfaces on a **file cache miss** (cold/evicted cache). The `.dat` segment file is unaffected because it is opened directly with the relative segment path. Note: `branch-3.1` does not have this bug because there `Segment::_open_inverted_index()` derives the index path from the relative `_seg_path` member. The regression was introduced when this was switched to `_file_reader->path()`. Fix `CLuceneError ... read past EOF` when querying an inverted index whose `.idx` file is stored in a packed file and is not present in the local file cache. Store the path passed to `Segment::open()` in a new `_seg_path` member and use it (instead of `_file_reader->path()`) to derive the inverted index file path prefix, so the lookup key matches the relative keys recorded by `PackedFileSystem`. This restores the behavior `branch-3.1` already had. A regression test (`cloud_p0/packed_file/test_packed_file_inverted_index_query`) loads small data so the `.idx` is packed, clears the file cache to force a miss, then runs inverted-index-backed queries and asserts they succeed with correct results. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
hello-stephen
commented
Jun 11, 2026
Contributor
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
liaoxin01
commented
Jun 11, 2026
ContributorAuthor
run buildall |
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes a cloud-mode failure when reading packed inverted-index (.idx) files after a BE file-cache miss by ensuring the inverted index path is derived from the original segment path (relative), not from a normalized absolute path returned by the remote filesystem.
Changes:
- BE: Preserve the original segment path and use it to derive the inverted index path prefix during index-file reader initialization.
- Regression: Add a cloud P0 test that clears the BE file cache and validates inverted-index queries succeed on cache miss with packed files enabled.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| regression-test/suites/cloud_p0/packed_file/test_packed_file_inverted_index_query.groovy | New regression test reproducing the packed .idx cache-miss scenario and asserting inverted-index queries succeed. |
| be/src/storage/segment/segment.h | Adds _seg_path member to retain the segment path passed into Segment::open. |
| be/src/storage/segment/segment.cpp | Sets _seg_path during open and uses it (instead of _file_reader->path()) to derive the inverted-index path prefix. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Uh oh!
There was an error while loading. Please reload this page.
hello-stephen
commented
Jun 11, 2026
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
approved these changes
Jun 11, 2026
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pick #64383