Uh oh!
There was an error while loading. Please reload this page.
branch-4.1: [fix](multi-catalog) Preserve external partition metadata - #66012
Conversation
hello-stephen
commented
Jul 24, 2026
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
suxiaogang223
commented
Jul 24, 2026
run buildall |
hello-stephen
commented
Jul 24, 2026
FE UT Coverage ReportIncrement line coverage |
hello-stephen
commented
Jul 24, 2026
FE Regression Coverage ReportIncrement line coverage |
19633ca to
eb977fcComparesuxiaogang223
commented
Jul 30, 2026
run buildall |
hello-stephen
commented
Jul 30, 2026
FE Regression Coverage ReportIncrement line coverage |
suxiaogang223
commented
Jul 31, 2026
run buildall |
hello-stephen
commented
Jul 31, 2026
FE UT Coverage ReportIncrement line coverage |
hello-stephen
commented
Jul 31, 2026
FE Regression Coverage ReportIncrement line coverage |
suxiaogang223
commented
Aug 3, 2026
/review |
There was a problem hiding this comment.
Request changes: this static review found four issues in the new partition-metadata paths: bulk load changes the established path NULL sentinel into an explicit non-null value; Iceberg UUID metadata is incompatible with VARBINARY mapping; Iceberg TIMESTAMPTZ metadata drops the zone required by both BE parsers; and stable Iceberg key discovery is recomputed across all historical specs for every file split. The classic and Nereids range branches otherwise remain aligned, Hive cache conversion operates on copied values, and the spec-ID-aware Iceberg cache/common-source-ID logic is conservative across partition evolution. The added tests cover basic parsing and ordering but not these compatibility/type-mapping cases or per-file scaling. I followed the static-review-only constraint and did not run builds. There were no existing review threads and no extra user focus; the required repository code-review skill was not present, so the explicit review prompt governed this run.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
suxiaogang223
commented
Aug 3, 2026
run buildall |
suxiaogang223
commented
Aug 3, 2026
/review |
There was a problem hiding this comment.
Request changes: one P1 correctness issue remains in Iceberg identity-partition metadata. Explicit snapshot/branch scans pin their descriptor schema, but the new helpers resolve names and types through the latest table schema. After rename/name reuse, that can bind a pinned regular slot to a different partition source ID; both the legacy scanner and FileScannerV2 then return the wrong constant.
The current head otherwise resolves the four existing review concerns: bulk-load \N keeps its legacy NULL contract, UUID and TIMESTAMPTZ stay file-backed under incompatible mappings, and ordered Iceberg keys are cached once per scan. Hive/Hudi and classic/Nereids paths keep values, keys, and NULL flags aligned; spec-aware cache identity and common-versus-per-spec scanner behavior otherwise look sound. The added unit tests cover the basic NULL, mapped-type, and ordering cases, but not the pinned-schema rename/name-reuse case above.
No additional user focus was provided. This was a static review under the workflow constraint, so no builds were run. No required AGENTS.md files were listed, and the required repository code-review skill was not present; the explicit review prompt governed the review.
Uh oh!
There was an error while loading. Please reload this page.
hello-stephen
commented
Aug 3, 2026
FE Regression Coverage ReportIncrement line coverage |
2f32410 to
97f5678Comparesuxiaogang223
commented
Aug 4, 2026
run buildall |
Backport the remaining external partition fixes to branch-4.1 while adapting them to its FileScannerV2 infrastructure. Preserve explicit NULL metadata for Hive-style scans, Hudi, and load paths, and make Iceberg identity partition metadata stable and safe across partition spec evolution.
97f5678 to
b5dae17Comparesuxiaogang223
commented
Aug 4, 2026
run buildall |
hello-stephen
commented
Aug 4, 2026
FE UT Coverage ReportIncrement line coverage |
hello-stephen
commented
Aug 4, 2026
FE Regression Coverage ReportIncrement line coverage |
Uh oh!
There was an error while loading. Please reload this page.
…6835) ### What problem does this PR solve? Issue Number: DORIS-27947 Related PR: #66012, #66481 Problem Summary: On branch-4.1, Iceberg identity partition columns are classified as partition keys and are therefore omitted from the Parquet file-column schema tree. A predicate on such a column still reaches the Parquet file-level predicate probe. `StructNode::children_column_exists()` assumed that every probed column had a schema-tree entry, so a partition predicate could trigger a DCHECK and abort the BE. This change makes the existence probe return `false` when the schema tree has no entry for the column. Parquet then skips file-level predicate pushdown for that partition column while the existing partition-value fill and upper-level predicate evaluation remain in effect. The added unit test covers present, known-missing, and absent schema-tree children. ### Release note Fix a BE crash when querying Iceberg branch data with an identity partition predicate. ### Validation - Formatted the changed files with clang-format 16.0.6. - `git diff --check` passed. - BE compilation and unit tests were not run locally because of their cost. ### Behavior changed Yes. A column absent from the file schema tree is now treated as unavailable for file-level predicate pushdown instead of terminating the BE.
What problem does this PR solve?
Issue Number: None
Related PR: #62821, #65581, #66010
Problem Summary: branch-4.1 already backported the Paimon and BE partition metadata changes through #65581, but Hive-style scans, Hudi, Iceberg, and file load paths still had incomplete FE metadata propagation. Hive default partitions were encoded as
\N, which lost the distinction between NULL and a literal\N. Iceberg attached partition metadata only when runtime partition pruning was enabled, used unstable per-map ordering, and did not safely handle files written with different partition specs.This change:
\Nas data while representing Hive default partitions as NULL;Release note
Fix partition-column materialization, NULL handling, and Iceberg partition evolution compatibility for external table scans and file loads on branch-4.1.
Check List (For Author)
./run-fe-ut.sh --run org.apache.doris.common.util.BrokerUtilTest,org.apache.doris.datasource.hive.source.HiveScanNodeTest,org.apache.doris.datasource.iceberg.IcebergUtilsTest,org.apache.doris.datasource.iceberg.source.IcebergScanNodeTest,org.apache.doris.datasource.paimon.source.PaimonScanNodeTest