Skip to content

Deletion Vectors read support - #1516

Merged
Fokko merged 16 commits into
apache:mainfrom
Fokko:fd-add-deletion-vectors
Mar 26, 2025
Merged

Deletion Vectors read support#1516
Fokko merged 16 commits into
apache:mainfrom
Fokko:fd-add-deletion-vectors

Conversation

@Fokko

@FokkoFokko commented Jan 15, 2025

Copy link
Copy Markdown
Contributor

First see if we can produce some deletion vectors.

Resolves#1549

First see if we can produce some deletion vectors.
Comment threaddev/provision.py Outdated
@kevinjqliu

Copy link
Copy Markdown
Contributor

Sidenote, we dont have a way to read puffin files in python. I saw this PR from iceberg-rust, apache/iceberg-rust#892
Perhaps this will be a good candidate for pyiceberg_core

@Fokko

Copy link
Copy Markdown
ContributorAuthor

I want to try using the Python roaringbitmap API first, otherwise, we can go the Iceberg-Rust route.

@kevinjqliu

Copy link
Copy Markdown
Contributor

Looks like V3 is not supported on the pyiceberg side

=========================== short test summary info ============================
FAILED tests/integration/test_reads.py::test_read_table_with_deletion_vector[session_catalog_hive] - pyiceberg.exceptions.ValidationError: 1 validation error for TableMetadataWrapper
Input tag '3' found using 'format_version' | 'format-version' does not match any of the expected tags: 1, 2 [type=union_tag_invalid, input_value={'format-version': 3, 'ta...c474ec.metadata.json'}]}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.10/v/union_tag_invalid
FAILED tests/integration/test_reads.py::test_read_table_with_deletion_vector[session_catalog] - pydantic_core._pydantic_core.ValidationError: 1 validation error for TableResponse
metadata
Input tag '3' found using 'format_version' | 'format-version' does not match any of the expected tags: 1, 2 [type=union_tag_invalid, input_value={'format-version': 3, 'ta...26ab8a.metadata.json'}]}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.10/v/union_tag_invalid
==== 2 failed, 970 passed, 8 skipped, 2757 deselected in 346.14s (0:05:46) =====
make: *** [Makefile:55: test-integration] Error 1
Error: Process completed with exit code 2.

@FokkoFokko mentioned this pull request Jan 20, 2025
@Fokko

Copy link
Copy Markdown
ContributorAuthor

@kevinjqliu That's correct, I've split that out here: #1540

@Fokko

Copy link
Copy Markdown
ContributorAuthor

@kevinjqliu PR is out here: #1554 :)

@kevinjqliu

Copy link
Copy Markdown
Contributor

do we want to include this as part of 0.9.0? the ability to read puffin files would be great!

@Fokko

Fokko commented Feb 3, 2025

Copy link
Copy Markdown
ContributorAuthor

@kevinjqliu I agree, but I don't think there is a lot of value in just supporting Puffin without having the ability to actually understand the content of the Puffin files. For the CI we're blocked on Iceberg-Java 1.8.0, until then I don't think there is much value of getting this in

@Fokko
Fokko marked this pull request as ready for review February 17, 2025 10:24
@FokkoFokko changed the title WIP: Deletion vectorsDeletion vectorsFeb 17, 2025
@FokkoFokko changed the title Deletion vectorsDeletion Vectors read supportFeb 17, 2025
@Fokko
Fokko requested a review from kevinjqliuMarch 4, 2025 09:25
@FokkoFokko added this to the PyIceberg 0.10.0 milestone Mar 4, 2025

@kevinjqliukevinjqliu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Excited for puffin files

Comment threadpyiceberg/table/puffin.py Outdated
Comment threadpyiceberg/table/puffin.py
@FokkoFokko mentioned this pull request Mar 20, 2025
14 tasks
@kevinjqliu

Copy link
Copy Markdown
Contributor

except for the MAX_JAVA_SIGNED variable, everything else LGTM!

@Fokko
Fokko merged commit 62191ee into apache:mainMar 26, 2025
gabeiglio pushed a commit to Netflix/iceberg-python that referenced this pull request Aug 13, 2025
First see if we can produce some deletion vectors.
Resolvesapache#1549
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support Deletion Vectors

2 participants

@Fokko@kevinjqliu