GH-49376: [Python][Parquet] Add ability to write Bloom filters from pyarrow by etseidl · Pull Request #49377 · apache/arrow

etseidl · 2026-02-23T20:41:15Z

Fixes#49376

Rationale for this change

Adds ability to enable the writing of Parquet Bloom filters via pyarrow.

What changes are included in this PR?

Adds bloom_filter_options to parquet.write_table.

Are these changes tested?

Yes, new tests are added.

Are there any user-facing changes?

Adds an option (defaults to None) to parquet.write_table.

GitHub Issue: [Python][Parquet] Add options to control writing of Bloom filters to parquet.write_table #49376

github-actions · 2026-02-23T20:41:46Z

⚠️ GitHub issue #49376has been automatically assigned in GitHub to PR creator.

raulcd · 2026-02-24T09:07:12Z

+ elif isinstance(_bloom_opts, bool):
+ # if false do nothing, if true then just pass defaults
+ if not _bloom_opts:
+ continue


I understand that bloom filter is disabled by default but should we be explicit here calling props.disable_bloom_filter(tobytes(column))?
Otherwise exposing Builder* disable_bloom_filter(const c_string& path) on libparquet.pxd seems slightly unnecessary as is never being used.

Yes, I agree it would be better to be explicit here. Thanks!

mapleFU · 2026-02-25T03:06:36Z

+ The keys of the `dict` are column paths. For each path, the value can be either:
+
+ - A boolean, with ``True`` indicating that a Bloom filter should be produced with
+ the default values of `NDV=1048576` and `FPP=0.05`.


lower case?

mapleFU · 2026-02-25T03:06:52Z

+ - A boolean, with ``True`` indicating that a Bloom filter should be produced with
+ the default values of `NDV=1048576` and `FPP=0.05`.
+ - A dictionary, with keys `ndv` and `fpp`. `ndv` must be a positive integer, and
+ `fpp` must be a float between 0.0 and 1.0. Default values will be used for any


Unexists ndv and fpp is regarded as 1048576/0.05 ?

Yes. I've reworked the docs to hopefully make this clearer.

mapleFU

General LGTM. Sorry for late review!

mapleFU · 2026-03-24T13:32:47Z

+ if isinstance(fpp, float):
+ if fpp <= 0.0 or fpp >= 1.0:
+ raise ValueError(
+ f"'fpp' for column '{column}' must be in (0.0, 1,0), got {fpp}")


1,0 typo here?

mapleFU · 2026-03-24T13:33:13Z

+ if "ndv" in _bloom_opts:
+ ndv = _bloom_opts["ndv"]
+ if isinstance(ndv, int):
+ if ndv < 0:


I'm not sure if NDV can be 0 or not...I can't seem to find validation in either the C++ nor rust implementations. This test just ensures it's a positive value. Honestly, I don't see why you'd need a bloom filter for a column with no values, so I'm fine following your suggestion.

mapleFU · 2026-03-24T13:39:31Z

+ bloom_opts.ndv = ndv
+ else:
+ raise TypeError(
+ f"'ndv' for column '{column}' must be an int")


should we notice they're in bloom filter options?

mapleFU

cc @pitrou @raulcd would you mind take a look? Otherwise I'll merge this next week

mapleFU · 2026-03-25T02:11:36Z

+ f"'bloom_filter_options:ndv' for column '{column}' must be an int")
+ if "fpp" in _bloom_opts:
+ fpp = _bloom_opts["fpp"]
+ if isinstance(fpp, float):


I don't know whether casting to float is a better way

I don't think so. We don't want to accept strings, for example.

raulcd

LGTM, just a minor question about defining the defaults vs trying to reuse the ones defined on the C++ struct, so we don't diverge if at any point we change the defaults.
But overall all looks good!

pitrou · 2026-03-25T09:22:48Z


+ # bloom filters
+ if bloom_filter_options is not None:
+ if isinstance(bloom_filter_options, dict):


Can we factor this out into a helper function? This function is becoming much too large IMHO.

remove constants

raulcd

Thanks @etseidl

PyArrow 23.0.1 has no API for writing Parquet bloom filters. The C++ layer supports it (apache/arrow#37400) but Python bindings are still in draft (apache/arrow#49377). Row group size (2000) and sorting metadata still work and provide the main pushdown benefit. Bloom filters are kept as commented-out TODOs for when PyArrow 24+ ships.

mapleFU · 2026-04-01T07:47:21Z

I would merge if no objection next monday

raulcd · 2026-04-01T07:59:35Z

I would merge if no objection next monday

Sounds good to me. I've added the 24.0.0 milestone to the original issue so I don't forget to pick it unless there are other concerns being worked on.

mapleFU · 2026-04-06T04:16:02Z

Thanks all, pr is merged

conbench-apache-arrow · 2026-04-06T10:36:38Z

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 5fd30d3.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 4 possible false positives for unstable benchmarks that are known to sometimes produce them.

…from pyarrow (apache#49377) Fixesapache#49376 ### Rationale for this change Adds ability to enable the writing of Parquet Bloom filters via pyarrow. ### What changes are included in this PR? Adds `bloom_filter_options` to `parquet.write_table`. ### Are these changes tested? Yes, new tests are added. ### Are there any user-facing changes? Adds an option (defaults to `None`) to `parquet.write_table`. * GitHub Issue: apache#49376 Authored-by: seidl <seidl2@llnl.gov> Signed-off-by: mwish <maplewish117@gmail.com>

etseidl added 3 commits February 23, 2026 10:14

add bloom filter write options to pyarrow
a9ca4d7

add more options and tests and clean up docs
c0fb063

docs
176fe77

etseidl requested review from AlenkaF, raulcd and rok as code owners February 23, 2026 20:41

github-actionsBot added Component: Python awaiting committer review Awaiting committer review labels Feb 23, 2026

raulcd reviewed Feb 24, 2026
View reviewed changes

github-actionsBot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Feb 24, 2026

make disabling explicit
c74fa69

github-actionsBot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Feb 24, 2026

etseidl changed the title ~~GH-49376: [Python] Add ability to write Bloom filters from pyarrow~~GH-49376: [Python][Parquet] Add ability to write Bloom filters from pyarrowFeb 24, 2026

etseidl added Component: Parquet labels Feb 24, 2026

mapleFU reviewed Feb 25, 2026
View reviewed changes

make the docs a little clearer
c67af90

github-actionsBot added awaiting changes Awaiting changes and removed Component: Parquet awaiting change review Awaiting change review labels Feb 25, 2026

etseidl requested a review from mapleFU March 3, 2026 16:10

mapleFU reviewed Mar 24, 2026
View reviewed changes

implement suggestions from review
1b39f04

github-actionsBot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Mar 24, 2026

mapleFU approved these changes Mar 25, 2026
View reviewed changes

raulcd reviewed Mar 25, 2026
View reviewed changes

Comment threadpython/pyarrow/_parquet.pyx Outdated

github-actionsBot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Mar 25, 2026

pitrou reviewed Mar 25, 2026
View reviewed changes

refactor setting of bloom filter options to separate function
965729a
remove constants

github-actionsBot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Mar 25, 2026

raulcd approved these changes Mar 27, 2026
View reviewed changes

github-actionsBot added awaiting merge Awaiting merge and removed awaiting change review Awaiting change review labels Mar 27, 2026

mapleFU merged commit 5fd30d3 into apache:mainApr 6, 2026
17 checks passed

mapleFU removed the awaiting merge Awaiting merge label Apr 6, 2026

mapleFU mentioned this pull request Apr 6, 2026
[Python][Parquet] Add options to control writing of Bloom filters to parquet.write_table#49376
Closed

renaudb mentioned this pull request Apr 6, 2026
support bloom-filter writing apache/iceberg-python#850
Open

Uh oh!

Conversation

etseidl commented Feb 23, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actionsBot commented Feb 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mapleFU left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mapleFU left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

Uh oh!

mapleFU commented Apr 1, 2026

Uh oh!

raulcd commented Apr 1, 2026

Uh oh!

Uh oh!

mapleFU commented Apr 6, 2026

Uh oh!

conbench-apache-arrowBot commented Apr 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

etseidl commented Feb 23, 2026•
edited
Loading