Skip to content

feat: Add support for writing bloom filters - #3265

Closed
renaudb wants to merge 7 commits into
apache:mainfrom
renaudb:renaudb-add-bloom-filters-write
Closed

feat: Add support for writing bloom filters#3265
renaudb wants to merge 7 commits into
apache:mainfrom
renaudb:renaudb-add-bloom-filters-write

Conversation

@renaudb

@renaudbrenaudb commented Apr 21, 2026

Copy link
Copy Markdown

Closes#850

Note: This PR is currently held back by boto requiring pyarrow<=23.1 as bloom filter write support was added in pyarrow 24.

Rationale for this change

Add support for writing bloom filters to parquet files. This changes leverages the new bloom_filter_optionswrite_parquet argument in pyarrow 24.

Are these changes tested?

Added tests for the metadata parsing. Added a very basic test for the writing path (there is currently no way to test for the existence of a bloomfilter in a parquet file using pyarrow).

Are there any user-facing changes?

N/A

@renaudb
renaudb marked this pull request as draft April 21, 2026 21:40
@Fokko

Copy link
Copy Markdown
Contributor

Thanks for working on this @renaudb. I've noticed that in the uv.lock we're behind some versions of boto3/botocore, not sure why this. Maybe we can bump this with uv sync --upgrade manually (in a separate PR).

@renaudb

Copy link
Copy Markdown
Author

@Fokko the issue is with Bodo, not Boto. Bodo forces pyarrow>=23.0,<23.1. I saw a bunch of issues filed on their hand about how this is too restrictive, but it looks like they are limited in changing it.

https://github.com/bodo-ai/Bodo/blob/main/pyproject.toml#L9

@renaudb
renaudbforce-pushed the renaudb-add-bloom-filters-write branch from b86268b to efd56e8CompareMay 14, 2026 18:44
@renaudb
renaudb marked this pull request as ready for review May 14, 2026 19:04
@renaudb

Copy link
Copy Markdown
Author

@Fokko this should be ready for review.

@Fokko
Fokko self-requested a review May 21, 2026 20:10
Comment threadpyiceberg/io/pyarrow.py Outdated
Comment threadpyiceberg/io/pyarrow.py Outdated
Comment threadpyiceberg/io/pyarrow.py Outdated
Comment threadtests/integration/test_writes/test_writes.py
@github-actions

Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions

Copy link
Copy Markdown

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

@renaudb

Copy link
Copy Markdown
Author

Pyarrow 25 was released so the test for bloomfilter existence should now pass.

@renaudb

Copy link
Copy Markdown
Author

@abnobdoss do you have the ability to reopen the PR? I don't see to have the permission to do it.

@abnobdoss

abnobdoss commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Hi @renaudb, I don't believe I can either.

@kevinjqliu / @Fokko / @geruh would you be able to re-open this?

@raphaelauv

Copy link
Copy Markdown

@renaudb otherwise you could open a new PR ( using the same branch ). Thanks again for all this works

@renaudb

Copy link
Copy Markdown
Author

Reopened here: #3757

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

support bloom-filter writing

4 participants

@renaudb@Fokko@abnobdoss@raphaelauv