Skip to content

[test](regression) Add external correctness P0 coverage - #66002

Merged
yiguolei merged 1 commit into
apache:masterfrom
Gabriel39:agent/correctness-special-p0
Jul 24, 2026
Merged

[test](regression) Add external correctness P0 coverage#66002
yiguolei merged 1 commit into
apache:masterfrom
Gabriel39:agent/correctness-special-p0

Conversation

@Gabriel39

@Gabriel39Gabriel39 commented Jul 24, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Problem Summary:

This PR fills P0 regression gaps found while auditing the external-table correctness issues under JIRA:

  • bind Iceberg predicates and runtime filters correctly across rename/drop/type evolution;
  • scan, filter, and aggregate files written with multiple Iceberg partition specs;
  • keep an Iceberg write atomic after a pipeline error and make a corrected retry visible exactly once;
  • mask S3 and OAuth credentials in persisted audit statements.

The audit also found this, including a nested required-field reproduction. This test-only PR does not change production code or attempt to fix that issue.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

    Local verification:

    ./run-regression-test.sh --run -s test_iceberg_schema_evolution_filter_binding,test_iceberg_multi_spec_filter_aggregate,test_iceberg_failed_write_atomicity_retry,test_external_catalog_credential_masking

    Result: 4 suites passed, 0 failed.

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

hello-stephen commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39

Gabriel39 commented Jul 24, 2026

Copy link
Copy Markdown
ContributorAuthor

run buildall

@Gabriel39

Gabriel39 commented Jul 24, 2026

Copy link
Copy Markdown
ContributorAuthor

/review

@Gabriel39
Gabriel39 marked this pull request as ready for review July 24, 2026 08:02
@github-actions

github-actionsBot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Jul 30th, 2026 7:58 AM.
Workflow run: https://github.com/apache/doris/actions/runs/30077439974

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@Gabriel39
Gabriel39force-pushed the agent/correctness-special-p0 branch from a5cff33 to 102f49dCompareJuly 24, 2026 08:29
@Gabriel39

Gabriel39 commented Jul 24, 2026

Copy link
Copy Markdown
ContributorAuthor

run buildall

@Gabriel39

Gabriel39 commented Jul 24, 2026

Copy link
Copy Markdown
ContributorAuthor

/review

@github-actions

github-actionsBot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Jul 30th, 2026 7:58 AM.
Workflow run: https://github.com/apache/doris/actions/runs/30079125938

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@Gabriel39

Gabriel39 commented Jul 24, 2026

Copy link
Copy Markdown
ContributorAuthor

/review

@github-actions

github-actionsBot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Jul 30th, 2026 7:58 AM.
Workflow run: https://github.com/apache/doris/actions/runs/30081192781

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@yiguolei
yiguolei merged commit 2232784 into apache:masterJul 24, 2026
29 of 30 checks passed
github-actionsBot pushed a commit that referenced this pull request Jul 24, 2026
@Gabriel39Your Name
[test](regression) Add external correctness P0 coverage (#66002)
2f72ef7
### What problem does this PR solve?
Problem Summary:
This PR fills P0 regression gaps found while auditing the external-table
correctness issues under JIRA:
- bind Iceberg predicates and runtime filters correctly across
rename/drop/type evolution;
- scan, filter, and aggregate files written with multiple Iceberg
partition specs;
- keep an Iceberg write atomic after a pipeline error and make a
corrected retry visible exactly once;
- mask S3 and OAuth credentials in persisted audit statements.
The audit also found this, including a nested required-field
reproduction. This test-only PR does not change production code or
attempt to fix that issue.
### Release note
None
### Check List (For Author)
- Test
- [x] Regression test
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Local verification:
`./run-regression-test.sh --run -s
test_iceberg_schema_evolution_filter_binding,test_iceberg_multi_spec_filter_aggregate,test_iceberg_failed_write_atomicity_retry,test_external_catalog_credential_masking`
Result: 4 suites passed, 0 failed.
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
@github-actionsgithub-actionsBot mentioned this pull request Jul 24, 2026
Merged
@Gabriel39Gabriel39 mentioned this pull request Jul 25, 2026
Merged
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Jul 25, 2026
@Gabriel39
[test](regression) Add external correctness P0 coverage (apache#66002)
561ab78
### What problem does this PR solve?
Problem Summary:
This PR fills P0 regression gaps found while auditing the external-table
correctness issues under JIRA:
- bind Iceberg predicates and runtime filters correctly across
rename/drop/type evolution;
- scan, filter, and aggregate files written with multiple Iceberg
partition specs;
- keep an Iceberg write atomic after a pipeline error and make a
corrected retry visible exactly once;
- mask S3 and OAuth credentials in persisted audit statements.
The audit also found this, including a nested required-field
reproduction. This test-only PR does not change production code or
attempt to fix that issue.
### Release note
None
### Check List (For Author)
- Test
- [x] Regression test
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Local verification:
`./run-regression-test.sh --run -s
test_iceberg_schema_evolution_filter_binding,test_iceberg_multi_spec_filter_aggregate,test_iceberg_failed_write_atomicity_retry,test_external_catalog_credential_masking`
Result: 4 suites passed, 0 failed.
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
0AyanamiRei pushed a commit to 0AyanamiRei/doris that referenced this pull request Jul 27, 2026
@Gabriel39@0AyanamiRei
[test](regression) Add external correctness P0 coverage (apache#66002)
3def2e5
### What problem does this PR solve?
Problem Summary:
This PR fills P0 regression gaps found while auditing the external-table
correctness issues under JIRA:
- bind Iceberg predicates and runtime filters correctly across
rename/drop/type evolution;
- scan, filter, and aggregate files written with multiple Iceberg
partition specs;
- keep an Iceberg write atomic after a pipeline error and make a
corrected retry visible exactly once;
- mask S3 and OAuth credentials in persisted audit statements.
The audit also found this, including a nested required-field
reproduction. This test-only PR does not change production code or
attempt to fix that issue.
### Release note
None
### Check List (For Author)
- Test
- [x] Regression test
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Local verification:
`./run-regression-test.sh --run -s
test_iceberg_schema_evolution_filter_binding,test_iceberg_multi_spec_filter_aggregate,test_iceberg_failed_write_atomicity_retry,test_external_catalog_credential_masking`
Result: 4 suites passed, 0 failed.
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Jul 27, 2026
@Gabriel39
[test](regression) Add external correctness P0 coverage (apache#66002)
905cb52
### What problem does this PR solve?
Problem Summary:
This PR fills P0 regression gaps found while auditing the external-table
correctness issues under JIRA:
- bind Iceberg predicates and runtime filters correctly across
rename/drop/type evolution;
- scan, filter, and aggregate files written with multiple Iceberg
partition specs;
- keep an Iceberg write atomic after a pipeline error and make a
corrected retry visible exactly once;
- mask S3 and OAuth credentials in persisted audit statements.
The audit also found this, including a nested required-field
reproduction. This test-only PR does not change production code or
attempt to fix that issue.
### Release note
None
### Check List (For Author)
- Test
- [x] Regression test
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Local verification:
`./run-regression-test.sh --run -s
test_iceberg_schema_evolution_filter_binding,test_iceberg_multi_spec_filter_aggregate,test_iceberg_failed_write_atomicity_retry,test_external_catalog_credential_masking`
Result: 4 suites passed, 0 failed.
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Jul 28, 2026
@Gabriel39
[test](regression) Add external correctness P0 coverage (apache#66002)
0a9e11f
### What problem does this PR solve?
Problem Summary:
This PR fills P0 regression gaps found while auditing the external-table
correctness issues under JIRA:
- bind Iceberg predicates and runtime filters correctly across
rename/drop/type evolution;
- scan, filter, and aggregate files written with multiple Iceberg
partition specs;
- keep an Iceberg write atomic after a pipeline error and make a
corrected retry visible exactly once;
- mask S3 and OAuth credentials in persisted audit statements.
The audit also found this, including a nested required-field
reproduction. This test-only PR does not change production code or
attempt to fix that issue.
### Release note
None
### Check List (For Author)
- Test
- [x] Regression test
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Local verification:
`./run-regression-test.sh --run -s
test_iceberg_schema_evolution_filter_binding,test_iceberg_multi_spec_filter_aggregate,test_iceberg_failed_write_atomicity_retry,test_external_catalog_credential_masking`
Result: 4 suites passed, 0 failed.
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
Gabriel39 added a commit that referenced this pull request Jul 28, 2026
@Gabriel39@hubgeter@morningman
[branch-4.1] Backport Iceberg, Paimon, and file scanner changes (#66032)
16a773b
## Proposed changes
Backport the requested changes to `branch-4.1` in master merge order,
skipping changes already present in this PR:
1. #62438
2. #65329 (merged prerequisite for the nested-schema cases)
3. #65960
4. #65965
5. #65972
6. #65998
7. #66002
8. #65992
9. #66021
10. #66036
11. #66008
12. #66073
13. #66056 (explicitly requested; current open-PR head, appended after
the merged sequence)
The branch-specific compatibility commits preserve the selected master
behavior on `branch-4.1`, including master wire IDs for file formats and
the merged Paimon test helper prerequisite. No regression expected
output, test assertion, or test input was changed to make validation
pass.
## Validation
- Full BE ASAN build passed.
- Full FE build passed.
- Targeted BE ASAN unit tests: 332 tests from 13 suites passed.
- Targeted FE Iceberg unit tests: 41 passed, 0 failed.
- Iceberg write regression: 20 suites, 0 failed, 0 fatal.
- `PaimonScanNodeTest`: all 16 test bodies completed with 0 assertion
failures; the class reports one Mockito teardown error because #66008
left four now-unused stubs on master. The still-open #65867 contains the
upstream test-only cleanup commit. This PR intentionally does not alter
those test cases.
- Final rebase against the latest `branch-4.1` completed; the branch was
already up to date.
- Working-tree, formatting, and sensitive-information audits completed.
Existing EOF blank lines in picked regression output files are
preserved.
---------
Co-authored-by: daidai <changyuwei@selectdb.com>
Co-authored-by: Mingyu Chen (Rayner) <yunyou@selectdb.com>
yiguolei pushed a commit that referenced this pull request Jul 30, 2026
@github-actions@Gabriel39
branch-4.1: [test](regression) Add external correctness P0 coverage#…
70bbb1c
…66002 (#66023)
Cherry-picked from #66002
Co-authored-by: Gabriel <liwenqiang@selectdb.com>
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Aug 17, 2026
@Gabriel39@wyxxxcat
[test](regression) Add external correctness P0 coverage (apache#66002)
cfd94bd
### What problem does this PR solve?
Problem Summary:
This PR fills P0 regression gaps found while auditing the external-table
correctness issues under JIRA:
- bind Iceberg predicates and runtime filters correctly across
rename/drop/type evolution;
- scan, filter, and aggregate files written with multiple Iceberg
partition specs;
- keep an Iceberg write atomic after a pipeline error and make a
corrected retry visible exactly once;
- mask S3 and OAuth credentials in persisted audit statements.
The audit also found this, including a nested required-field
reproduction. This test-only PR does not change production code or
attempt to fix that issue.
### Release note
None
### Check List (For Author)
- Test
- [x] Regression test
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Local verification:
`./run-regression-test.sh --run -s
test_iceberg_schema_evolution_filter_binding,test_iceberg_multi_spec_filter_aggregate,test_iceberg_failed_write_atomicity_retry,test_external_catalog_credential_masking`
Result: 4 suites passed, 0 failed.
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguoleiyiguoleiyiguolei approved these changes

Assignees

No one assigned

Labels

dev/4.1.4-merged

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Gabriel39@hello-stephen@yiguolei