GH-50007: [C++][Parquet] Add bloom filter folding to automatically size SBBF filters by HuaHuaY · Pull Request #50008 · apache/arrow

HuaHuaY · 2026-05-21T10:18:59Z

Rationale for this change

This PR follows apache/arrow-rs#9628. It supports optimizing the disk usage of the Bloom filter. So specifying an ndv value larger than the actual value will not affect disk usage.

Bloom filters now support folding mode: allocate a conservatively large filter (sized for worst-case NDV), insert all values during writing, then fold down at flush time to meet a target FPP. This eliminates the need to guess NDV upfront and produces optimally-sized filters automatically.

What changes are included in this PR?

BloomFilterBuilder will try to fold the bloom filter before writing it to the output stream.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes.

The type of ndv in BloomFilterOptions is changed from int32_t to std::optional<int64_t>. And the argument type of OptimalNumOfBytes and OptimalNumOfBits in BlockSplitBloomFilter is changed from uint32_t ndv to uint64_t ndv.

Add a new field fold in BloomFilterOptions and default value is true.

GitHub Issue: [C++][Parquet] Add bloom filter folding to automatically size SBBF filters #50007

github-actions · 2026-05-21T10:20:34Z

⚠️ GitHub issue #50007has been automatically assigned in GitHub to PR creator.

HuaHuaY · 2026-05-21T12:56:23Z

@wgtmac @alamb @etseidl @emkornfield Please take a look.

alamb · 2026-05-21T18:11:16Z

I am not likely to have time to review C++ code in the arrow repository unfortunately

wgtmac

Thanks @HuaHuaY for adding this quickly!

wgtmac · 2026-05-29T02:26:19Z

 std::to_string(bloom_filter_options.fpp));
 }
+ if (bloom_filter_options.ndv.has_value() && bloom_filter_options.ndv.value() < 0) {
+ throw ParquetException("Bloom filter number of distinct values must be >= 0, got " +


What is the expected behavior of 0?

It will create a smallest bloom filter.

wgtmac · 2026-05-29T03:53:17Z

cc @mapleFU @adamreeve

wgtmac

Generally LGTM. I left some nits.

mapleFU

Generally LGTM

HuaHuaY · 2026-06-02T11:17:12Z

@pitrou @mapleFU Please take a look.

mapleFU · 2026-06-02T13:59:55Z

+ (static_cast<double>(num_blocks) * kBytesPerFilterBlock * 8);
+ const auto max_folds = static_cast<uint32_t>(std::countr_zero(num_blocks));
+
+ if (avg_fill == 0.0) {


I little bit forgot would this really happens when writing a parquet file?

If all values in a column chunk are null, avg_fill will be 0.

Does it still need a BF or fold in this scenerio? Or this path would lead to zero cost folding?

I think there are differences between "not have a bloom filter" and "bloom filter has no values". The latter can filter every not null values. And there is currently no way to indicate that a bloom filter exists but has no value through metadata.

What about
(1) without folding, just replace to a smallest one without any copying
And there is currently no way to indicate that a bloom filter exists but has no value through metadata.
In theory you're right. In production, I believe null_count == num_values works?

without folding, just replace to a smallest one without any copying
Good idea. I will change the code soon.
I believe null_count == num_values works
I don't think it's good to mix two separate components. Also, null_count is an optional value and may not actually exist.

I have updated the commits and now only fold when total_set_bits is not equal to 0.

mapleFU · 2026-06-02T14:16:53Z

+ const auto* bitset32 = reinterpret_cast<const uint32_t*>(data_->data());
+ const uint32_t num_words = num_bytes_ / static_cast<uint32_t>(sizeof(uint32_t));
+ for (uint32_t i = 0; i < num_words; ++i) {
+ total_set_bits += static_cast<uint64_t>(std::popcount(bitset32[i]));


I don't know whether internal::CountSetBits easy to understand here ( though popcount is right and a bit faster)

I have changd to internal::CountSetBits. internal::CountSetBits may be faster because it counts once every 64 bits.

HuaHuaY · 2026-06-05T06:36:16Z

@pitrou Please take a look again.

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

pitrou

Thanks a lot @HuaHuaY !

conbench-apache-arrow · 2026-06-11T16:07:56Z

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 108240f.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 5 possible false positives for unstable benchmarks that are known to sometimes produce them.

…als-test on test-conda-cpp-valgrind (#50213) ### Rationale for this change Fix#50212 `Test #89: parquet-internals-test .......................***Failed 14.96 sec` ### What changes are included in this PR? Add custom `PrintTo` for parameter `BloomFilterBuilderFoldingTestCase` introduced with pr #50008, so gtest prints fields instead of raw bytes (error `Use of uninitialised value`). Similar to the past solution in commit 1b7e396 but with additional field names/debug output. ### Are these changes tested? Yes, by CI. ### Are there any user-facing changes? No. * GitHub Issue: #50212 Authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org>

HuaHuaY requested a review from wgtmac as a code owner May 21, 2026 10:19

github-actionsBot added the awaiting review Awaiting review label May 21, 2026

github-actionsBot added Component: Parquet Component: C++ labels May 21, 2026

HuaHuaY commented May 21, 2026
View reviewed changes

Comment threadcpp/src/parquet/bloom_filter_writer.cc Outdated

github-actionsBot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels May 21, 2026

wgtmac reviewed May 29, 2026
View reviewed changes

HuaHuaY force-pushed the sbbf_filters branch from 4495c53 to 9565196Compare May 29, 2026 07:55

wgtmac approved these changes May 29, 2026
View reviewed changes

Comment threadcpp/src/parquet/properties.h Outdated
Comment threadcpp/src/parquet/bloom_filter.cc Outdated
Comment threadcpp/src/parquet/properties.h

mapleFU reviewed May 29, 2026
View reviewed changes

Comment threadcpp/src/parquet/bloom_filter_writer.cc Outdated
Comment threadcpp/src/parquet/bloom_filter.cc Outdated

pitrou reviewed Jun 2, 2026
View reviewed changes

mapleFU approved these changes Jun 2, 2026
View reviewed changes

mapleFU reviewed Jun 2, 2026
View reviewed changes

Comment threadcpp/src/parquet/bloom_filter_reader_writer_test.cc

HuaHuaY force-pushed the sbbf_filters branch 2 times, most recently from d26cb2e to d57601bCompare June 4, 2026 09:34

wgtmac approved these changes Jun 5, 2026
View reviewed changes

pitrou reviewed Jun 9, 2026
View reviewed changes

Comment threadcpp/src/parquet/bloom_filter_reader_writer_test.cc Outdated
Comment threadcpp/src/parquet/bloom_filter.cc

HuaHuaYand others added 6 commits June 9, 2026 22:47

add bloom filter folding to automatically size SBBF filters
ae6871c

fix review
e3cfb18

fix review
0d9c10b

Update cpp/src/parquet/bloom_filter_reader_writer_test.cc
a2cda43
Co-authored-by: Antoine Pitrou <pitrou@free.fr>

Update cpp/src/parquet/bloom_filter.cc
8427dfb
Co-authored-by: Antoine Pitrou <pitrou@free.fr>

fix ci
be0a0bc

HuaHuaY force-pushed the sbbf_filters branch from 9569e5a to be0a0bcCompare June 9, 2026 14:47

pitrou reviewed Jun 9, 2026
View reviewed changes

Comment threadcpp/src/parquet/bloom_filter_reader_writer_test.cc

fix review
f0c2bc8

HuaHuaY requested a review from pitrou June 11, 2026 07:45

pitrou approved these changes Jun 11, 2026
View reviewed changes

pitrou merged commit 108240f into apache:mainJun 11, 2026
52 checks passed

pitrou removed the awaiting committer review Awaiting committer review label Jun 11, 2026

github-actionsBot added the awaiting committer review Awaiting committer review label Jun 11, 2026

pitrou mentioned this pull request Jun 11, 2026
[C++][Parquet] Add bloom filter folding to automatically size SBBF filters #50007
Closed

HuaHuaY deleted the sbbf_filters branch June 11, 2026 08:42

tadeja mentioned this pull request Jun 17, 2026
GH-50212: [C++][Parquet] Add PrintTo to fix failure in parquet-internals-test on test-conda-cpp-valgrind #50213
Merged

dmatth1 mentioned this pull request Jun 18, 2026
GH-50026: [C++][Parquet] SIMD-accelerate SBBF probe via branchless autovec #50030
Merged

Uh oh!

Conversation

HuaHuaY commented May 21, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actionsBot commented May 21, 2026

Uh oh!

Uh oh!

HuaHuaY commented May 21, 2026

Uh oh!

alamb commented May 21, 2026

Uh oh!

wgtmac left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wgtmac commented May 29, 2026

Uh oh!

wgtmac left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mapleFU left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

HuaHuaY commented Jun 2, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HuaHuaYJun 4, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HuaHuaYJun 4, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

HuaHuaY commented Jun 5, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pitrou left a comment

HuaHuaY commented May 21, 2026•
edited
Loading

HuaHuaYJun 4, 2026•
edited
Loading

HuaHuaYJun 4, 2026•
edited
Loading