Uh oh!
There was an error while loading. Please reload this page.
[regression](test) expand Iceberg and Paimon partition evolution coverage - #65992
Conversation
hello-stephen
commented
Jul 24, 2026
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Gabriel39
commented
Jul 24, 2026
run buildall |
Gabriel39
commented
Jul 24, 2026
/review |
There was a problem hiding this comment.
Requesting changes. The patch is test/document-only and the fixtures are broad, but several matrix cells can pass without exercising the behavior they claim.
Critical checkpoint conclusions:
- Goal and proof: not yet met. The current fixtures miss post-drop position/DV data, attributable add-identity/bucket/temporal RF pruning, cross-spec temporal matches, complex-field bindings, and proven native/DV paths.
- Scope/focus: source changes are confined to seven external-table suites and one coverage document. The scope is coherent, but the document overstates several transform/delete/format/reader cross-products.
- Concurrency: no Doris product concurrency was changed. The two profile-polling suites do have a test-harness race because they run in parallel while discovering profiles through the bounded FE-wide list.
- Lifecycle/cleanup: container-process completion, catalog refresh, snapshot capture, unique fixture names, and suite-local JDBC cleanup look sound. No static-initialization or product object-lifecycle change applies.
- Configuration: no new config item was added. Session-variable changes are suite-local because SuiteContext closes each connection; the issue is whether the RF is retained and observed, not cross-suite leakage.
- Compatibility/persistence/data writes: no product protocol, storage-format, EditLog, or transaction path changed. External snapshot/tag ordering is sound, but one position/DV fixture writes its victim before the partition-field drop.
- Parallel reader/scanner paths: the explicit Iceberg V1/V2 toggles are real where present, but the full transform timelines do not cover the documented format/scanner cross-product. Paimon's second read may use native or CPP and does not prove a physical DV or a historical evolved projection.
- Test results: I did not run builds or tests, as required by the review contract. The live compile check passes; external/P0/nonConcurrent regressions are still pending. Fixed SQL results also need named qt/order_qt checks with generated .out files.
- Observability: the sole positive Iceberg RF profile is attributable to category identity pruning. The delete-aware RF checks lack plan/profile proof, and the Paimon reader/DV path lacks explain/metadata proof.
- Performance: no product hot path changed. The main execution concern is nondeterministic profile retention under parallel suite traffic.
- User focus: no additional user-provided focus was supplied.
The PR title checker is also failing because test: expand Iceberg and Paimon partition evolution coverage does not match the required [type](module) form. Suggested title: [test](regression) Expand Iceberg and Paimon partition evolution coverage.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
### What problem does this PR solve? Issue Number: None Related PR: apache#65992 Problem Summary: Complete the Iceberg and Paimon partition-evolution review matrix with deterministic expected outputs, positive runtime-filter pruning evidence, post-evolution delete artifacts, complex-schema historical projections, native/JNI split assertions, and full Parquet/ORC scanner coverage. ### Release note None ### Check List (For Committer) - Test: local P0 regression on one FE and two BEs (8 suites, 0 failed, 0 fatal, 0 skipped)
Gabriel39
commented
Jul 24, 2026
run buildall |
Gabriel39
commented
Jul 24, 2026
/review |
Codex automated review failed and did not complete. Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Jul 30th, 2026 7:58 AM. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
PR approved by at least one committer and no changes requested. |
Uh oh!
There was an error while loading. Please reload this page.
…rage (apache#65992) ## What changed - add five Iceberg P0 suites for partition evolution with static/runtime filters, snapshot/tag/branch reads, position deletes, equality deletes, deletion vectors, and a complete format/scanner transform timeline - add three Paimon P0 suites for fixed-partition complex schema evolution, partitioned primary-key delete/upsert/compaction, and atomic rejection of unsupported partition-key mutations - keep independent dimensions in separate Groovy suites for parallel execution, while marking four profile-dependent suites `nonConcurrent` - add deterministic runner-generated `.out` files and update the Iceberg/Paimon schema evolution and time-travel coverage matrix ## Coverage highlights - Iceberg add/drop/replace partition fields and identity, bucket, truncate, year/month/day/hour transforms - complex STRUCT/MAP/ARRAY changes combined with partition filtering and historical references - runtime-filter pruning with attributable positive profile-counter assertions on supported identity partition paths - Parquet/ORC, file scanner V1/V2, Iceberg v2 position/equality deletes, and v3 deletion vectors - Paimon Parquet/ORC, asserted JNI/native split paths, physical DV metadata, primary-key upsert/delete/compaction, snapshot/tag/branch - topology-independent execution on a distributed Doris cluster ## Validation - master: `f1460f89230441bc1b6b1872d66bd32a526b25b1` - one FE and two live BEs - eight partition-evolution extension suites: 8 passed, 0 failed, 0 fatal, 0 skipped - documented P0 matrix: 20 suites, including 8 partition-evolution extensions - no Doris production code changed - no additional Doris product issue was reproduced
…rage (apache#65992) ## What changed - add five Iceberg P0 suites for partition evolution with static/runtime filters, snapshot/tag/branch reads, position deletes, equality deletes, deletion vectors, and a complete format/scanner transform timeline - add three Paimon P0 suites for fixed-partition complex schema evolution, partitioned primary-key delete/upsert/compaction, and atomic rejection of unsupported partition-key mutations - keep independent dimensions in separate Groovy suites for parallel execution, while marking four profile-dependent suites `nonConcurrent` - add deterministic runner-generated `.out` files and update the Iceberg/Paimon schema evolution and time-travel coverage matrix ## Coverage highlights - Iceberg add/drop/replace partition fields and identity, bucket, truncate, year/month/day/hour transforms - complex STRUCT/MAP/ARRAY changes combined with partition filtering and historical references - runtime-filter pruning with attributable positive profile-counter assertions on supported identity partition paths - Parquet/ORC, file scanner V1/V2, Iceberg v2 position/equality deletes, and v3 deletion vectors - Paimon Parquet/ORC, asserted JNI/native split paths, physical DV metadata, primary-key upsert/delete/compaction, snapshot/tag/branch - topology-independent execution on a distributed Doris cluster ## Validation - master: `f1460f89230441bc1b6b1872d66bd32a526b25b1` - one FE and two live BEs - eight partition-evolution extension suites: 8 passed, 0 failed, 0 fatal, 0 skipped - documented P0 matrix: 20 suites, including 8 partition-evolution extensions - no Doris production code changed - no additional Doris product issue was reproduced
…rage (apache#65992) ## What changed - add five Iceberg P0 suites for partition evolution with static/runtime filters, snapshot/tag/branch reads, position deletes, equality deletes, deletion vectors, and a complete format/scanner transform timeline - add three Paimon P0 suites for fixed-partition complex schema evolution, partitioned primary-key delete/upsert/compaction, and atomic rejection of unsupported partition-key mutations - keep independent dimensions in separate Groovy suites for parallel execution, while marking four profile-dependent suites `nonConcurrent` - add deterministic runner-generated `.out` files and update the Iceberg/Paimon schema evolution and time-travel coverage matrix ## Coverage highlights - Iceberg add/drop/replace partition fields and identity, bucket, truncate, year/month/day/hour transforms - complex STRUCT/MAP/ARRAY changes combined with partition filtering and historical references - runtime-filter pruning with attributable positive profile-counter assertions on supported identity partition paths - Parquet/ORC, file scanner V1/V2, Iceberg v2 position/equality deletes, and v3 deletion vectors - Paimon Parquet/ORC, asserted JNI/native split paths, physical DV metadata, primary-key upsert/delete/compaction, snapshot/tag/branch - topology-independent execution on a distributed Doris cluster ## Validation - master: `f1460f89230441bc1b6b1872d66bd32a526b25b1` - one FE and two live BEs - eight partition-evolution extension suites: 8 passed, 0 failed, 0 fatal, 0 skipped - documented P0 matrix: 20 suites, including 8 partition-evolution extensions - no Doris production code changed - no additional Doris product issue was reproduced
…rage (apache#65992) ## What changed - add five Iceberg P0 suites for partition evolution with static/runtime filters, snapshot/tag/branch reads, position deletes, equality deletes, deletion vectors, and a complete format/scanner transform timeline - add three Paimon P0 suites for fixed-partition complex schema evolution, partitioned primary-key delete/upsert/compaction, and atomic rejection of unsupported partition-key mutations - keep independent dimensions in separate Groovy suites for parallel execution, while marking four profile-dependent suites `nonConcurrent` - add deterministic runner-generated `.out` files and update the Iceberg/Paimon schema evolution and time-travel coverage matrix ## Coverage highlights - Iceberg add/drop/replace partition fields and identity, bucket, truncate, year/month/day/hour transforms - complex STRUCT/MAP/ARRAY changes combined with partition filtering and historical references - runtime-filter pruning with attributable positive profile-counter assertions on supported identity partition paths - Parquet/ORC, file scanner V1/V2, Iceberg v2 position/equality deletes, and v3 deletion vectors - Paimon Parquet/ORC, asserted JNI/native split paths, physical DV metadata, primary-key upsert/delete/compaction, snapshot/tag/branch - topology-independent execution on a distributed Doris cluster ## Validation - master: `f1460f89230441bc1b6b1872d66bd32a526b25b1` - one FE and two live BEs - eight partition-evolution extension suites: 8 passed, 0 failed, 0 fatal, 0 skipped - documented P0 matrix: 20 suites, including 8 partition-evolution extensions - no Doris production code changed - no additional Doris product issue was reproduced
## Proposed changes Backport the requested changes to `branch-4.1` in master merge order, skipping changes already present in this PR: 1. #62438 2. #65329 (merged prerequisite for the nested-schema cases) 3. #65960 4. #65965 5. #65972 6. #65998 7. #66002 8. #65992 9. #66021 10. #66036 11. #66008 12. #66073 13. #66056 (explicitly requested; current open-PR head, appended after the merged sequence) The branch-specific compatibility commits preserve the selected master behavior on `branch-4.1`, including master wire IDs for file formats and the merged Paimon test helper prerequisite. No regression expected output, test assertion, or test input was changed to make validation pass. ## Validation - Full BE ASAN build passed. - Full FE build passed. - Targeted BE ASAN unit tests: 332 tests from 13 suites passed. - Targeted FE Iceberg unit tests: 41 passed, 0 failed. - Iceberg write regression: 20 suites, 0 failed, 0 fatal. - `PaimonScanNodeTest`: all 16 test bodies completed with 0 assertion failures; the class reports one Mockito teardown error because #66008 left four now-unused stubs on master. The still-open #65867 contains the upstream test-only cleanup commit. This PR intentionally does not alter those test cases. - Final rebase against the latest `branch-4.1` completed; the branch was already up to date. - Working-tree, formatting, and sensitive-information audits completed. Existing EOF blank lines in picked regression output files are preserved. --------- Co-authored-by: daidai <changyuwei@selectdb.com> Co-authored-by: Mingyu Chen (Rayner) <yunyou@selectdb.com>
…rage (apache#65992) ## What changed - add five Iceberg P0 suites for partition evolution with static/runtime filters, snapshot/tag/branch reads, position deletes, equality deletes, deletion vectors, and a complete format/scanner transform timeline - add three Paimon P0 suites for fixed-partition complex schema evolution, partitioned primary-key delete/upsert/compaction, and atomic rejection of unsupported partition-key mutations - keep independent dimensions in separate Groovy suites for parallel execution, while marking four profile-dependent suites `nonConcurrent` - add deterministic runner-generated `.out` files and update the Iceberg/Paimon schema evolution and time-travel coverage matrix ## Coverage highlights - Iceberg add/drop/replace partition fields and identity, bucket, truncate, year/month/day/hour transforms - complex STRUCT/MAP/ARRAY changes combined with partition filtering and historical references - runtime-filter pruning with attributable positive profile-counter assertions on supported identity partition paths - Parquet/ORC, file scanner V1/V2, Iceberg v2 position/equality deletes, and v3 deletion vectors - Paimon Parquet/ORC, asserted JNI/native split paths, physical DV metadata, primary-key upsert/delete/compaction, snapshot/tag/branch - topology-independent execution on a distributed Doris cluster ## Validation - master: `f1460f89230441bc1b6b1872d66bd32a526b25b1` - one FE and two live BEs - eight partition-evolution extension suites: 8 passed, 0 failed, 0 fatal, 0 skipped - documented P0 matrix: 20 suites, including 8 partition-evolution extensions - no Doris production code changed - no additional Doris product issue was reproduced
What changed
nonConcurrent.outfiles and update the Iceberg/Paimon schema evolution and time-travel coverage matrixCoverage highlights
Validation
f1460f89230441bc1b6b1872d66bd32a526b25b1