Skip to content

Implement column projection - #1443

Merged
Fokko merged 11 commits into
apache:mainfrom
gabeiglio:specPartitionIdentity
Feb 10, 2025
Merged

Implement column projection #1443
Fokko merged 11 commits into
apache:mainfrom
gabeiglio:specPartitionIdentity

Conversation

@gabeiglio

@gabeigliogabeiglio commented Dec 18, 2024•
edited by Fokko
Loading

Copy link
Copy Markdown
Contributor

This is a fix for issue #1401. In which table scans needed to infer partition column by following the column projection rules

Fixes#1401

@Fokko
Fokko self-requested a review December 18, 2024 20:38
@gabeiglio
gabeiglio marked this pull request as ready for review December 19, 2024 15:12
kevinjqliu
kevinjqliu reviewed Dec 19, 2024
View reviewed changes

@kevinjqliukevinjqliu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a few comments, please take a look! The PR looks great already. Thanks for working on this!

Comment threadpyiceberg/io/pyarrow.py Outdated
Comment threadtests/io/test_pyarrow.py Outdated
Comment threadpyiceberg/io/pyarrow.py Outdated
Fokko
Fokko reviewed Dec 20, 2024
View reviewed changes
Comment threadpyiceberg/io/pyarrow.py Outdated
Fokko
Fokko reviewed Dec 20, 2024
View reviewed changes
Comment threadpyiceberg/io/pyarrow.py Outdated
@kevinjqliu
kevinjqliu self-requested a review December 23, 2024 19:04
Gabriel Igliozzi
Removed file_schema check and initial-default logic, separated projec…
fee24ab
…tion logic to helper method, changed test to use high-level table scan
Fokko
Fokko reviewed Dec 29, 2024
View reviewed changes
Comment threadpyiceberg/io/pyarrow.py Outdated
Fokko
Fokko reviewed Dec 29, 2024
View reviewed changes
Comment threadpyiceberg/io/pyarrow.py Outdated
Fokko
Fokko reviewed Dec 29, 2024
View reviewed changes
Comment threadpyiceberg/io/pyarrow.py Outdated
Fokko
Fokko reviewed Dec 29, 2024
View reviewed changes
Comment threadpyiceberg/io/pyarrow.py Outdated
kevinjqliu
kevinjqliu reviewed Jan 3, 2025
View reviewed changes
Comment threadtests/io/test_pyarrow.py
Comment threadpyiceberg/io/pyarrow.py Outdated
Comment threadpyiceberg/io/pyarrow.py Outdated
kevinjqliu
kevinjqliu reviewed Jan 7, 2025
View reviewed changes
Comment threadpyiceberg/io/pyarrow.py Outdated
@Fokko
Fokko self-requested a review January 13, 2025 12:48
Fokko
Fokko reviewed Jan 13, 2025
View reviewed changes
Comment threadpyiceberg/io/pyarrow.py Outdated
kevinjqliu
kevinjqliu reviewed Jan 20, 2025
View reviewed changes

@kevinjqliukevinjqliu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

generally LGTM! I added a few nit comments and some clarifying questions on testing.

thanks for working on this!

Comment threadtests/io/test_pyarrow.py
Comment threadpyiceberg/io/pyarrow.py Outdated
Comment threadpyiceberg/io/pyarrow.py Outdated
Comment threadpyiceberg/io/pyarrow.py Outdated
Comment threadtests/io/test_pyarrow.py Outdated
Comment threadtests/io/test_pyarrow.py
kevinjqliu
kevinjqliu reviewed Feb 1, 2025
View reviewed changes

@kevinjqliukevinjqliu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like CI caught an interesting case where a new identity partition is added after data files were written. The accessor then cannot find the proper partition record... We need to do something like this

Comment threadtests/io/test_pyarrow.py Outdated
Comment threadtests/io/test_pyarrow.py Outdated
Comment threadtests/io/test_pyarrow.py
Fokko
Fokko reviewed Feb 4, 2025
View reviewed changes

@FokkoFokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there!

Comment threadpyiceberg/io/pyarrow.py Outdated
Comment threadpyiceberg/io/pyarrow.py Outdated
Comment threadpyiceberg/io/pyarrow.py Outdated
kevinjqliu
kevinjqliu approved these changes Feb 5, 2025
View reviewed changes

@kevinjqliukevinjqliu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks @gabeiglio !!

@kevinjqliu
kevinjqliu requested a review from FokkoFebruary 5, 2025 02:16
@Fokko
Fokko merged commit d9b1c03 into apache:mainFeb 10, 2025
@Fokko

Fokko commented Feb 10, 2025

Copy link
Copy Markdown
Contributor

Thanks for working on this @gabeiglio, and thanks for the review @kevinjqliu 🙌

@kevinjqliukevinjqliu mentioned this pull request Mar 1, 2025
Closed
@kevinjqliukevinjqliu mentioned this pull request Jul 26, 2025
Closed
@kevinjqliukevinjqliu mentioned this pull request Aug 6, 2025
Merged
@kevinjqliukevinjqliu mentioned this pull request Dec 23, 2025
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@FokkoFokkoFokko approved these changes
@kevinjqliukevinjqliukevinjqliu approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

API table.scan does not conform to Iceberg spec for identity partition columns

3 participants

@gabeiglio@Fokko@kevinjqliu