Uh oh!
There was an error while loading. Please reload this page.
fix: block type reset after dict filter - #63168
Merged
Gabriel39 merged 2 commits intoMay 20, 2026
Merged
Conversation
hello-stephen
commented
May 12, 2026
Contributor
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
yoock
commented
May 12, 2026
ContributorAuthor
run buildall |
Gabriel39
commented
May 12, 2026
Contributor
@yoock Please add regression test cases. |
ContributorAuthor
本质上就是RowGroupReader::next_batch结束时必须调用_convert_dict_cols_to_string_cols(block);，但是这里漏了 |
Gabriel39
commented
May 13, 2026
Contributor
/review |
Contributor
There was a problem hiding this comment.
Reviewed the full PR diff and existing review context. I found one blocking issue in BE Parquet lazy-read error handling: the new call ignores a Status-returning conversion helper, which violates Doris error propagation rules and can also fail compilation because Status is [[nodiscard]].
Critical checkpoint conclusions:
- Goal/test: The PR appears intended to keep dictionary-filtered columns converted to string on an early EOF lazy-read path. The change is not safely implemented because conversion failures are not propagated; no new test is included for this edge path.
- Scope/clarity: The change is small and focused, but needs the same RETURN_IF_ERROR handling used by adjacent paths.
- Concurrency/lifecycle/config/compatibility: No new concurrency, lifecycle, config, persistence, storage-format, or FE-BE compatibility concerns found in this one-line change.
- Parallel paths: Neighboring no-row/filter-all paths already propagate _convert_dict_cols_to_string_cols errors; this new path should match them.
- Error handling: Blocking issue found; Status is ignored.
- Test coverage/results: No test or result changes were added. Given the fix is a direct error-propagation correction, existing coverage may be sufficient after updating the call, but this specific lazy EOF path is not newly covered.
- Observability/performance/memory: No additional observability, performance, or memory issues found.
- User focus: No additional user-provided review focus was specified.
| DCHECK_EQ(pre_read_rows + _cached_filtered_rows, 0); | ||
| *read_rows = 0; | ||
| *batch_eof = true; | ||
| _convert_dict_cols_to_string_cols(block); |
Contributor
There was a problem hiding this comment.
This helper returns Status and Status is [[nodiscard]], so this call must be propagated. More importantly, if _get_block_column_pos() or convert_dict_column_to_string_column() fails, this path currently returns OK with the block left in the dictionary-code representation, unlike all adjacent _convert_dict_cols_to_string_cols call sites that use RETURN_IF_ERROR. Please keep the same error handling here.
Suggested change
| _convert_dict_cols_to_string_cols(block); | |
| RETURN_IF_ERROR(_convert_dict_cols_to_string_cols(block)); |
Gabriel39
commented
May 19, 2026
Contributor
/review |
Gabriel39
commented
May 19, 2026
Contributor
/run buildall |
Gabriel39
commented
May 19, 2026
Contributor
run buildall |
hello-stephen
commented
May 19, 2026
Contributor
TPC-H: Total hot run time: 30897 ms |
hello-stephen
commented
May 19, 2026
Contributor
TPC-DS: Total hot run time: 169949 ms |
hello-stephen
commented
May 19, 2026
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
hello-stephen
commented
May 19, 2026
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Gabriel39
commented
May 20, 2026
Contributor
/review |
Gabriel39
approved these changes
May 20, 2026
Uh oh!
There was an error while loading. Please reload this page.
Contributor
PR approved by at least one committer and no changes requested. |
Contributor
PR approved by anyone and no changes requested. |
github-actionsBot
pushed a commit
that referenced
this pull request
May 20, 2026
### What problem does this PR solve? Fix parquet reader reporting: Read parquet file xxxxx.parquet failed, reason = [INTERNAL_ERROR]comparison must input two same type column or column type is decimalv3/numeric, lhs=Int32, rhs=String --------- Co-authored-by: wanglong16 <wanglong16@xiaomi.com>
github-actionsBot
pushed a commit
that referenced
this pull request
May 20, 2026
### What problem does this PR solve? Fix parquet reader reporting: Read parquet file xxxxx.parquet failed, reason = [INTERNAL_ERROR]comparison must input two same type column or column type is decimalv3/numeric, lhs=Int32, rhs=String --------- Co-authored-by: wanglong16 <wanglong16@xiaomi.com>
yiguolei pushed a commit
that referenced
this pull request
May 20, 2026
zhaorongsheng pushed a commit
to zhaorongsheng/doris
that referenced
this pull request
Jun 4, 2026
### What problem does this PR solve? Fix parquet reader reporting: Read parquet file xxxxx.parquet failed, reason = [INTERNAL_ERROR]comparison must input two same type column or column type is decimalv3/numeric, lhs=Int32, rhs=String --------- Co-authored-by: wanglong16 <wanglong16@xiaomi.com>
morningman pushed a commit
that referenced
this pull request
Jun 24, 2026
### What problem does this PR solve? Fix parquet reader reporting: Read parquet file xxxxx.parquet failed, reason = [INTERNAL_ERROR]comparison must input two same type column or column type is decimalv3/numeric, lhs=Int32, rhs=String --------- Co-authored-by: wanglong16 <wanglong16@xiaomi.com>
morningman added a commit
that referenced
this pull request
Jun 26, 2026
Closed
wenzhenghu added a commit
to HYDCP/hy-doris
that referenced
this pull request
Aug 31, 2026
* [fix](be) Reset parquet dictionary columns on lazy-read EOF ### What problem does this PR solve? Issue Number: N/A Related PR: apache#63168 Problem Summary: When Parquet lazy materialization reaches EOF before creating a filter map, dictionary-filter columns remain Int32 in the returned block. Restore them to String before returning to prevent downstream bad casts and comparison failures. ### Release note Fix a BE crash caused by an incorrect Parquet dictionary-filter column type on the lazy-read EOF path. ### Check List (For Author) - Test: Static check - `./build-support/clang-format.sh` - `git diff --check` - BE UT was not run for this one-line semantic backport - Behavior changed: Yes. The Parquet reader now restores dictionary-filter columns to their logical String type before returning EOF. - Does this need documentation: No * [test](regression) Add parquet dict predicate lazy-read EOF case * [fix](regression) Align Parquet lazy-read fixture path ### What problem does this PR solve? Issue Number: None Related PR: #78 Problem Summary: The regression wrote the Parquet fixture under output/be/tmp while local() resolves relative paths under user_files_secure_path. Use /tmp consistently for OUTFILE, local(), and cleanup in the p0 regression environment. ### Release note None ### Check List (For Author) - Test: No need to test (static-only fix requested; git diff --check passed) - Behavior changed: No. Only the regression fixture path is aligned. - Does this need documentation: No
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
修复读取parquet时，当上一个row group有dict filter时未将block的类型从Int32重置为String，导致报如下错误：
Read parquet file xxxxx.parquet failed, reason = [INTERNAL_ERROR]comparison must input two same type column or column type is decimalv3/numeric, lhs=Int32, rhs=String
在有些情况下会触发如下代码导致be core：