docs(performance): document Parquet bloom filters and separate them from the bloom index - #19590
Conversation
hudi-agent
left a comment
There was a problem hiding this comment.
Thanks for the docs update! This adds a well-organized Parquet Bloom Filters section to performance.md, clearly stating the per-column write keys, the spark.hadoop. forwarding, sizing guidance, and two easy-to-miss caveats (the name collision with hoodie.parquet.bloom.filter.enabled and the swallowed reflection failure). My main concern is that the section is placed under "Read Path" but only documents the write-side keys — it doesn't mention what a reader needs to actually consult these filters. Please have a Hudi committer confirm the read-path behavior before merging.
234eced to
32cacba
Compare
…rom the bloom index
Parquet's per-column bloom filters have been passthrough-configurable in Hudi since 0.14.0, but the
only trace of them in the docs was an auto-generated row in configurations.md - and that row is for
a different feature (see below). Nothing told a user the keys exist, that they are per column, or
that they are read from the Hadoop config.
Adds a "Parquet Bloom Filters" subsection to performance.md under Read Path, next to Data Skipping,
since that is the gap it fills: column stats prune on ranges and are least useful for an equality
predicate on a high-cardinality column, which is exactly what a bloom filter covers.
parquet.bloom.filter.enabled#<column> write a bloom filter for that column
parquet.bloom.filter.expected.ndv#<column> expected distinct values, sizes the filter
HoodieBaseParquetWriter.handleParquetBloomFilters scans the Hadoop configuration for those prefixes
and forwards each to the Parquet writer builder, so from Spark they are set with the spark.hadoop.
prefix. The spelling matches TestHoodieParquetBloom, which sets exactly these keys on
jsc.hadoopConfiguration.
Two things are called out because both are easy to get wrong:
- hoodie.parquet.bloom.filter.enabled is NOT this feature, despite the name. It controls whether
Hudi writes a bloom filter of RECORD KEYS into the footer for the bloom index during upserts; it
defaults to true, applies only when meta fields are populated, and is implied when
hoodie.index.type names a BLOOM index (HoodieFileWriterFactory.enableBloomFilter). Setting it
does not enable per-column Parquet filters.
- the settings are applied by reflection and the NoSuchMethodException is swallowed, so on a
Parquet without withBloomFilterEnabled / withBloomFilterNDV they are silently ignored.
Applied to the current docs and to every supported versioned copy - 1.2.0, 1.1.1, 1.0.2, 1.0.1 and
1.0.0 - because the feature ships in all of them: handleParquetBloomFilters is absent at
release-0.13.1 and present from release-0.14.0 onwards, and the hoodie.parquet.bloom.filter.enabled
config it is contrasted against reads defaultValue(true)/sinceVersion(0.15.0) identically at
release-1.0.0, 1.0.2, 1.1.1 and 1.2.0, so the text is accurate for each. That five-version fan-out
matches how other cross-version docs fixes have been applied (apache#19555, apache#19459). version-1.0.0 has no
"Related Resources" heading, so there the section is appended at the end of the page instead of
before it; the inserted text is byte-identical in all six files.
The 0.15.x and 0.14.x copies are left alone: they have had no or nearly no doc commits in the last
six months and are effectively frozen.
Review follow-up: says what the read side needs, since the section sits under Read Path while the
keys it documents are write-side. On Spark 3.x nothing extra is configured - reading back through the
Hudi datasource consults the filters given a pushed-down equality predicate, and
TestHoodieParquetBloomFilter pins that by disabling the statistics and column-index filter levels so
only a bloom can cause a skip, then asserting zero row groups scanned for an absent value against one
for a present value. parquet.filter.bloom.enabled is left at its parquet-mr default and Hudi never
sets it. Also renamed the example column from uuid, which read like a record key, and said outright
that <column> is any data column filtered by equality rather than the record key.
Closes apache#16063
32cacba to
41a3a0d
Compare
hudi-agent
left a comment
There was a problem hiding this comment.
Thanks for the docs update! This adds a well-sourced "Parquet Bloom Filters" section to performance.md (and the versioned copies), covering the per-column parquet.bloom.filter.* write-time keys, expected.ndv sizing, the read-side behavior on Spark 3.x, and a clear note distinguishing these from Hudi's own record-key hoodie.parquet.bloom.filter.enabled. I verified each claim against master — the write-path keys and reflection behavior in HoodieBaseParquetWriter, and the record-key config's true default / meta-fields / BLOOM-index conditions in HoodieStorageConfig and HoodieFileWriterFactory — and all are accurate; the round-1 concerns were already addressed. Please have a Hudi committer or PMC member do a final pass before merge.
cc @yihua
|
Thank you @rangareddy! +1 |
Describe the issue this Pull Request addresses
Closes #16063 (HUDI-6456), "Add parquet blooms documentation".
Hudi forwards Parquet's per-column bloom filter settings through to the Parquet writer, but the only trace of
bloom filters in the docs was one auto-generated row in
configurations.md— and that row is for a differentfeature with a near-identical name. So nothing told a user that these keys exist, that they are per column,
or that they are read from the Hadoop configuration:
Summary and Changelog
Adds a Parquet Bloom Filters subsection to
performance.md, underRead Pathnext toData Skipping,because that is precisely the gap it fills: column stats prune on ranges and so help least for an equality
predicate on a high-cardinality column whose min-max spans nearly every file — which is what a bloom filter
covers.
The section states the two keys, that they are per column, how to set them from Spark (
spark.hadoop.prefix),how to size
expected.ndv, that it is a write-time decision so an existing table only picks them up as it isrewritten, and — added after review — what the read side needs: nothing, on Spark 3.x, beyond an equality
predicate that can be pushed down.
Two caveats are called out, because both are easy to get wrong and neither was written down anywhere:
hoodie.parquet.bloom.filter.enabledis not this feature. It controls whether Hudi writes a bloomfilter of record keys into the footer for the bloom index during
upserts. It defaults to
true, applies only when meta fields are populated, and is implied anyway whenhoodie.index.typenames aBLOOMindex. Setting it does not enable per-column Parquet filters. Giventhe names differ by one prefix, a user reading
configurations.mdcould reasonably conclude the opposite.withBloomFilterEnabled/withBloomFilterNDV, the keys are silently ignored rather than rejected.Verification
Docs change, so no test to add — said plainly rather than implied. Every claim was read off master
(
3ba31dd37fff):HoodieBaseParquetWriter.handleParquetBloomFilters,:98-119— iterates theConfigurationand splits the column off each keyHoodieBaseParquetWriter:52-53, andTestHoodieParquetBloom.scala:36-37setsparquet.bloom.filter.enabled#bloom_col/expected.ndv#bloom_colonjsc.hadoopConfigurationwithBloomFilterEnabled/withBloomFilterNDV:105and:114, invoked by reflection:107-109and:116-118—NoSuchMethodExceptionis caught and skippedhoodie.parquet.bloom.filter.enabledis about record-key blooms for the bloom indexHoodieStorageConfig:252-258(defaulttrue, since 0.15.0) andHoodieFileWriterFactory.enableBloomFilter:141-146(populateMetaFields && (thisConfig ‖ index.type contains BLOOM))TestHoodieRowDataParquetConfigInjector:100-101TestHoodieParquetBloomFilter(hudi-spark): sets the write keys, disablesparquet.filter.columnindex.enabledandparquet.filter.stats.enabledso only a bloom can skip, then asserts 0 row groups scanned for an absent value vs 1 for a present one, reading viaspark.read.format("hudi"), over BULK_INSERT / INSERT / UPSERT / INSERT_OVERWRITE on COWparquet.filter.bloom.enabledleft at its defaultgrepfinds Hudi setting it nowhereVersion coverage. The feature ships in every currently-supported release, so the section is added to the
current docs and to all five supported versioned copies:
website/docs/versioned_docs/version-1.2.0,1.1.1,1.0.2,1.0.1,1.0.0Checked rather than assumed:
handleParquetBloomFiltersis absent atrelease-0.13.1and present fromrelease-0.14.0onward, and thehoodie.parquet.bloom.filter.enabledconfig it is contrasted against readsdefaultValue(true)/sinceVersion("0.15.0")identically atrelease-1.0.0,1.0.2,1.1.1and1.2.0— so every claim in the text is accurate for each version it lands in. The five-version fan-out matches how
other cross-version docs fixes have been applied (#19555, #19459). The 0.15.x and 0.14.x copies are left
alone: they have had no, or nearly no, doc commits in the last six months and are effectively frozen.
version-1.0.0/performance.mdhas no## Related Resourcesheading — its page ends after Data Skipping —so there the section is appended at the end of the file rather than before that anchor. Everywhere else it is
inserted before it.
Markdown checked by running
markdownlintover all six files before and after and comparing ruleclasses rather than counts, since these files already carry findings:
Also checked across all six: every
<column>occurrence sits inside inline code, so nothing new is exposedto the MDX parser; the
indexes.mdlink target exists in every one of the five versioned trees so therelative link resolves in each; and the inserted region is byte-identical in all six files.
Deliberately not naming
TestHoodieParquetBloomFilterin the docs page itself: a user-facing reference to atest class rots silently on a rename, which is the objection @hudi-agent raised on #19486. The evidence lives
here and in the commit message instead.
Not done: the Docusaurus build (
website/node_modulesabsent, full install heavy). The section adds atable, a fenced block and two admonitions, all constructs already used on this page.
Relationship to #9056
@parisni opened #9056 ("[HUDI-6456] [DOC] Add parquet blooms documentation") for this same issue in
June 2023, and it is still open. I did not see it when I opened this PR — I checked the issue timeline for
cross-referenced PRs, and because #9056 predates the JIRA-migrated issue it never linked there, even though
hudi-bot had posted a "Linked PR(s)" comment naming it. That was my error and I have said so on both threads.
The two differ in shape: #9056 adds a dedicated
parquet_bloom.mdpage (plus a Parquet Config section inconfigurations.md), where this one adds a section toperformance.mdnext to Data Skipping. #9056 alsocarries material this PR does not — how dictionary encoding makes blooms redundant below ~40k distinct
values, and NDV rationale — while this PR carries the config name collision, the silent-reflection caveat and
the five-version fan-out that #9056 does not have. I have posted those findings on #9056 as review comments
too, so they are available whichever shape a committer prefers, and I am happy to fold this content into that
PR instead if that is the preference.
Impact
Documentation only — no code, config, API or format change. Two features whose names differ by one prefix are
now distinguishable, and a write-side knob that previously required reading
HoodieBaseParquetWritertodiscover is now documented.
Risk Level
none
Documentation Update
This is the documentation update. Targets
asf-site; applied todocs/plus the five supported versionedcopies (1.2.0, 1.1.1, 1.0.2, 1.0.1, 1.0.0), as justified in Verification above.
Contributor's checklist
asf-sitePRs do not run themastergates;markdownlintparity checked locallyas described