feat: support bloom filters in native Iceberg writes

[NikitaMatskevich]

Which issue does this PR close?

Partially addresses #5643 by lifting the native Iceberg V2 write restriction for the complete supported Bloom-filter property set:

• write.parquet.bloom-filter-enabled.column.<column>
• write.parquet.bloom-filter-fpp.column.<column>
• write.parquet.bloom-filter-ndv.column.<column>
• write.parquet.bloom-filter-max-bytes, when its value is exactly representable by the Apache Arrow Rust parquet crate

This is also part of the production-quality native Iceberg write work tracked by #5649.

This is related to #5304, which tracks missing writer-property propagation in the generic native ParquetWriterExec, but #5304 is not a blocker for this PR and is not closed by it. Iceberg V2 writes use a separate native writer path and construct their own WriterProperties.

Why is this change needed?

Before this PR, an Iceberg table that enabled a Parquet Bloom filter could not use Comet's native Iceberg writer.

Bloom-filter enablement alone is not enough for production compatibility. Iceberg exposes the requested false-positive probability (FPP), expected number of distinct values (NDV), and maximum Bloom-filter allocation. Ignoring any of these properties could silently produce a differently sized filter and regress pruning for readers.

This PR therefore translates the complete sizing decision, uses the native writer only when the result can be represented without any regression, and otherwise keeps the classic writer fallback.

Writer architecture

The write path is:

Spark Iceberg V2 write
└── JVM `IcebergWriteExec` (Comet split writer / JVM fallback)
    └── JVM `CometIcebergWriteExec` (native-execution wrapper)
        └── protobuf `Operator { iceberg_write: IcebergWrite }`
            └── Rust `execution::operators::iceberg_write::IcebergWriteExec`
                ├── Comet-built `parquet::file::properties::WriterProperties`
                └── iceberg-rust `ParquetWriterBuilder::new(WriterProperties, schema)`

It constructs WriterProperties from the Apache Arrow Rust parquet crate
and passes them into iceberg-rust: dee native/core/src/execution/operators/iceberg_write.rs.
That direct construction is useful: Comet does not have to delegate Iceberg table-property translation to iceberg-rust's from_table_properties. The pinned Apache Arrow Rust parquet crate 58.4.0 API already exposes set_column_bloom_filter_enabled, set_column_bloom_filter_fpp, and set_column_bloom_filter_ndv, so no iceberg-rust contribution is required for this work.

Sizing compatibility strategy

Iceberg's defaults are FPP 0.01 and max-bytes 1 MiB. The JVM Iceberg/Apache Parquet Java stack applies the settings in this order:

1. If NDV is absent, request the full max-bytes allocation.
2. If NDV is present, calculate the requested allocation from NDV and FPP.
3. Cap that allocation at max-bytes.

Comet reproduces this precedence.

Apache Parquet Java accepts a non-power-of-two cap and can serialize that exact used space, while the Rust crate uses power-of-two allocations and may fold a sparse filter after writing. For that reason, non-power-of-two, malformed, and out-of-range values fallback to classic writer.

The Apache Arrow Rust parquet crate 58.4.0 has no independent max-byte setter. Comet first reproduces Apache Parquet Java's allocation decision, then converts the resulting power-of-two byte target into a synthetic NDV passed to the Rust parquet crate. For a target B, the crate rounds every calculated size in (B/2, B] up to B; Comet aims at 3B/4, verifies the result using the crate's exact sizing expression, and has a binary-search fallback for unusual valid FPP values. This avoids fragile +/- 1 behavior at floating-point boundaries.

Folding: the Apache Arrow Rust parquet crate may fold a sparsely populated filter to a smaller power-of-two filter while preserving the configured FPP. This is considered an accepted improvement: it reduces file space and remains safe for every standards-compliant Parquet reader. Where folding does not occur, this PR verifies byte identity with Apache Parquet Java. Where it does occur, tests compare both used space and membership safety explicitly.

Feature	Supported by this PR	Upstream work required?
Per-column Bloom-filter enablement	Yes	No
Per-column FPP configuration	Yes	No
Per-column NDV configuration	Yes	No
Power-of-two write.parquet.bloom-filter-max-bytes in [32, 128 MiB]	Yes	No; Comet emulates it through a verified synthetic NDV
Arbitrary non-power-of-two max-byte parity	No; classic-writer fallback	A future Apache Arrow Rust parquet API would still need exact arbitrary-size and folding semantics before it could remove this fallback safely
Automatic Iceberg property translation in iceberg-rust	Not used	No; direct WriterProperties construction is intentional

What changes are included?

• The JVM driver resolves enabled Iceberg logical column names to physical Parquet leaf paths
• The native writer mirrors Apache Parquet Java's allocation decision and configures the Apache Arrow Rust parquet crate per column.

Commit structure

This PR is intentionally split into two reviewable feature commits:

1. Enable per-column Bloom filters in native Iceberg writes (d60c19f). This commit plumbs write.parquet.bloom-filter-enabled.column.<column>
2. Honor Iceberg Bloom-filter sizing settings (b544d41). This commit adds FPP, NDV, max-byte, and nested physical-path translation

Each commit is a self-contained feature increment with its own implementation and tests. Both feature states were validated with their relevant CI test suites locally and can be split into distinct PRs if reviewers prefer to review or merge enablement and sizing separately.

End-to-end writer and file tests: CometIcebergWriteActionSuite

• Configured Bloom filters exist and native execution is real.
• Binding max-bytes byte identity.
• FPP/NDV presence matrix and impossible settings.
• Underestimated explicit NDV.
• Large cap, folding, byte identity, and adjacent fallback.
• Out-of-range writer selection.

Planning tests: CometIcebergWriteDetectionSuite

JVM/protobuf tests: IcebergWriteProtoTranslationSuite

• Proves the unset defaults are exactly FPP 0.01 and max bytes 1,048,576.
• Proves explicit per-column FPP/NDV and explicit max bytes are transferred deterministically.
• Proves logical Iceberg list/map leaf names are remapped to physical Parquet paths, while properties for missing fields are skipped like Iceberg Java.

Rust unit tests: execution::operators::iceberg_write::tests

Assisted by Codex.

[alessandro-nori]

found one divergence from iceberg Java, the rest of the implementation looks good to me

[alessandro-nori]

this differs from the Java implementation for map and list columns (e.g. for a list tags.element vs tags.list.element) and the filter would be silently omitted by Java readers.

In iceberg-rust there is a schema visitor called IndexByParquetPathName but it is private.
Could we consider resolving the Parquet paths on the driver (Scala)? Or maybe making the iceberg-rust visitor public

[NikitaMatskevich]

Thanks for reviewing it! Implemented and pushed in the amended second commit 1d8376b. Test covering this was added too

[unikdahal]

Thanks for the detailed work here. The nested list/map physical-path fix looks good.

I found three remaining parquet-mr compatibility gaps in Bloom-filter property handling/sizing. Details inline.

[unikdahal]

I don't think we should intentionally change the max-bytes=32 behavior while describing this sizing logic as parquet-mr compatible.

In parquet-mr, BlockSplitBloomFilter only installs maximumBytes when it is strictly greater than the 32-byte lower bound. So a configured maximum of exactly 32 is effectively not used as the maximum when NDV/FPP request a larger filter.

For example:

NDV=1,000,000, FPP=0.0001, max-bytes=32

requests about 2.63 MiB before power-of-two rounding, and the JVM writer ends up with a 4 MiB Bloom filter because the 32-byte maximum is ignored. This implementation forcibly returns 32 bytes instead — a very large pruning-quality difference.

Could we either emulate the parquet-mr behavior here or conservatively fall back to the classic writer for max-bytes=32 when it matters? The existing binding-cap test would be stronger if it compared the JVM and native footer sizes for this boundary.

[NikitaMatskevich]

Ok, changed this behavior to comply with parquet-mr in 7d218de.

[unikdahal]

There is one more parquet-mr parity edge case here for very large but still valid NDVs.

parquet-mr calculates -8 * n using Java long arithmetic before the division/conversion to floating point. That multiplication can overflow. Here ndv is converted to f64 before multiplication, so the Java overflow behavior can never occur.

For example, with NDV = 2^61, Java's -8 * n wraps to zero and parquet-mr ends up requesting the minimum 32-byte Bloom filter, while this implementation calculates a huge value and caps it at the configured maximum (1 MiB by default).

Since planning currently accepts the full positive Java long range, this can silently select native execution with materially different output. Rather than reproducing the overflow, could we conservatively fall back for ndv > Long.MaxValue / 8 and add tests at the threshold, threshold + 1, 2^61, and Long.MaxValue?

[NikitaMatskevich]

This does not look like a feature one can actually rely upon. But I guess it doesn't hurt to have this behavior replicated here. Merged in 7d218de.

IMO so many distinct values in a column is not a reasonable usecase for bloom filters anyway, so producing a tiny 32b "placeholder" filter for such columns is even better implementation choice than actually allocating 128mb. That said, I would even lower this "NDV max limit" down from 2^60 to something more realistic if I could.

[unikdahal]

Thanks for addressing the earlier comments. I re-checked the latest changes, and those issues look resolved.

I found one remaining cross-version test issue and left an inline comment on it. Other than that, the implementation LGTM.

Also, could you please rebase against main to resolve the merge conflicts?

[unikdahal]

Could we gate NDV support separately here? This only checks the FPP property, but NDV is absent on older supported Iceberg versions (e.g. 1.8/1.10). The assumption therefore passes while production correctly falls back for explicit NDV, so NDV tests such as the overflow / false + NDV cases can incorrectly expect Compatible.

Separate FPP and NDV assumptions would also preserve FPP-only coverage on those versions.

[sunchao]

Correctness

Reviewed 7d218de7 against authoritative base 7f1e0018, with authored changes measured from merge base 719cba11. Previously, an enabled Bloom-filter property forced Iceberg writes onto the JVM path. The PR lifts that restriction, maps configured columns to Parquet paths, carries FPP/NDV/max-byte settings through protobuf, and configures the native writer. Its eligibility checks keep unsupported runtime properties, unrepresentable caps and problematic numeric values on the classic writer.

The previous discussions about enabled=false plus NDV, the 32-byte cap and overflowing NDV are addressed in the current control flow. NDV can re-enable a configured column, associated malformed values are checked even when enabled is false, and the two sizing boundaries fall back. The list/map path adjustment also addresses the previously reported missing intermediate path components.

One new [P2] remains: the new map assumes Iceberg Java's physical field names are also the native writer's names. Java sanitizes names such as order id to order_x20id, while the pinned native schema conversion preserves order id. The native Bloom-property lookup therefore misses the configured column. The inline comment requests matching the native writer's schema or falling back for renamed fields.

The maintained Spark 3.5 and 4.0 parsers accept backquoted names containing spaces and punctuation, and their schemas retain the names. This is a valid column shape, not an invalid-name input. Spark's V2 commit/abort protocol is unchanged by this PR. Bloom filters affect pruning rather than row values, so correct row results alone cannot establish that these properties were honored.

The existing cross-version test issue also remains. Both suites' shared assumption checks only the FPP constant. Iceberg 1.8.1 and 1.10.0 expose that constant but lack NDV support, so the assumption passes while production correctly rejects explicit NDV. NDV cases that expect native execution or Compatible need a separate capability assumption. I have not duplicated that inline comment.

Validation

No check runs, statuses or Actions runs were present for this head in the fresh check at 2026-09-08 04:40:59 UTC. No CI checkout, merge-tree equivalence or native artifact provenance could therefore be credited. The author's local-suite statement is not independently verified. The assigned base has advanced beyond this branch's merge base, including overlapping writer/reflection/test changes that must survive the requested rebase.

A bounded component test compiled the exact Java name-sanitizing methods, parquet-rs column-path identity/conversion and Comet path/sizing functions. Plain names matched. Three sanitized names missed the native column keys, and controls using native paths matched. It also passed 115 allocation round trips across supported powers of two and representative FPPs, plus default, underestimated-NDV, binding-cap and pathological-FPP controls. This used a simple property-map fixture and a Java precondition stub. It was not a complete Parquet-file or Spark/JNI reproduction. No local Comet build, Spark suite or performance benchmark was run. Maintained Spark 3.4/4.1 source branches were unavailable.

Performance

The default requests up to 1 MiB per configured column per row group, with larger supported caps up to 128 MiB. Allocation and hashing occur in the native writer, and fanout can multiply concurrent allocations. The sizing translation preserves the intended cap/NDV precedence in the inspected cases. The pinned parquet-rs encoder folds filters on flush, so final file size can differ from initial allocation.

The new path mismatch silently loses requested Bloom pruning for affected names. Apart from that finding, I found no material unnecessary work introduced by the translation. Schema-path reflection is performed during driver serialization, and the inverse sizing usually takes one candidate with a bounded search fallback. File-size assertions and sizing calculations do not establish write-throughput or downstream query gains.

Design

Separating eligibility, protobuf translation and native writer-property construction keeps fallback decisions before task execution. Explicit defaults prevent parquet-rs's different FPP default from leaking into Iceberg writes, and preserving absent NDV is necessary for the JVM sizing precedence. The physical-path mapping is the weak boundary: its source must agree with the schema actually emitted by the native writer.

Abstraction & complexity

The synthetic NDV is justified by the pinned API's lack of a separate byte-cap setter. The inverse calculation verifies its result, and the numeric search is bounded. The JVM representability check duplicates part of that sizing logic, but it serves a distinct purpose by rejecting unsupported settings before execution. Keeping those checks tied to dependency versions and boundary tests is necessary. Beyond correcting the path mapping, I found no actionable abstraction change.

[sunchao]

Correctness

[P2] Resolve Bloom paths against the native writer schema

Could we account for field-name sanitization before using this Java schema as the native path map? Iceberg Java converts a quoted column such as order id to physical name order_x20id, but the pinned iceberg-rust ToArrowSchemaConverter and parquet-rs writer preserve order id. With write.parquet.bloom-filter-enabled.column.order id=true, this map therefore configures ColumnPath(["order_x20id"]) while the writer looks up ["order id"]. The native write silently omits the requested Bloom filter. The same mismatch affects dotted names and names starting with a digit.

Please derive paths from the schema the native writer actually emits, or fall back when the Java conversion renames a configured field, and add a quoted-name test that checks the written footer for the filter. A bounded component using the exact name/path methods reproduced the lookup miss for three such names, with plain-name and native-path controls passing. This was not a full Spark/JNI reproduction.

[NikitaMatskevich]

Thank you for review! Fixed in "fix: fall back for sanitized Iceberg bloom paths" by falling back to java implementation when those problematic column names occur. I don't think its a big blocker for the coverage of Comet, personally I didn't see such naming in production before.

[NikitaMatskevich]

Added 2 boolean flags icebergSupportsBloomFpp and icebergSupportsBloomNdv based on reflection of Fpp and Ndv string prefixes availability in current iceberg-java. Based on these flags, now tests verify expected behaviors for each scenario. This was pushed in commit "fix: match Iceberg-version bloom property support".

• Spark 4.1 / Iceberg 1.11: complete writer suites, 115/115 passed; final parity test also rerun successfully.
• Spark 3.4 / Iceberg 1.5.2: detection and action suites passed after correcting the version-aware footer assertion.
• Spark 3.5 / Iceberg 1.8.1: full suites passed; final FPP-only test rerun successfully.
• Spark 4.0 / Iceberg 1.10.0: full suites passed; final FPP-only test rerun successfully.

[sunchao]

Correctness

Follow-up at fcef2c4f against bb9e7402, after the review of 7d218de7. The earlier sanitized-name finding is addressed: physical-path resolution compares each named ancestor and leaf with its Iceberg field ID, and an enabled column requiring Java name sanitization causes JVM fallback. That covers literal dots as well as spaces in field names by source inspection. The new action test specifically exercises order id. Maintained Spark 3.5/4.0 accept quoted identifiers, so preserving the JVM write path for these names is necessary. Their DataSource V2 commit/abort contract is unchanged by this update.

The FPP/NDV capability finding is also addressed. Iceberg 1.5.2 ignores both shape properties, 1.8.1/1.10.0 interpret FPP, and 1.11.0 additionally interprets NDV. Filtering unsupported prefixes before validation and serialization now preserves those versions' behavior, including enabled=false, malformed ignored properties, and the NDV setter re-enabling a filter when supported. The revised tests gate FPP independently, retaining the older-runtime FPP-only case. The positive-long/overflow, finite-FPP and representable-cap fallback checks remain in place. This change does not alter value conversion, null handling, or ANSI/Legacy expression semantics.

One new P2: the Bloom code still uses the pre-59 Parquet API after the rebase. The new test accesses private BloomFilterProperties.fpp/ndv fields, and the production setter is deprecated under the CI warnings-as-errors policy. The inline comment identifies both required API updates.

Validation

The exact dependency API projected into separate Rust crates reproduces three private-field errors and the deprecated-setter error. Getter/current-setter controls compile. The extracted sizing helper passes 115 allocation round trips and default, binding-cap, underestimated-NDV and extreme-FPP controls. These are component checks, not a full Comet/JNI/Spark build. Diff and Rust format checks pass. At the September 8, 10:41 UTC refresh, all four current-head workflows remain action_required, with zero check results. The recorded workflow-job inventories contain zero jobs. The author's reported Spark/Iceberg suite results are separate evidence, not independently reproduced current-head qualification. Maintained Spark 3.4/4.1 source coverage remains unavailable.

Performance

The new path traversal and capability filtering run during planning. No new per-row reflection is introduced. Writing enabled filters still adds hashing and allocates the initial configured filter per column and open row group. Parquet 59.3 folds at flush using an in-place truncation that retains vector capacity, so smaller serialized filters do not establish a corresponding memory reduction.

Please add a focused native/JVM write benchmark with Bloom disabled and enabled, including the default cap and a large cap at low and estimated cardinalities. Report write time, peak memory and filter/file bytes, then measure equality/IN row groups and bytes read with an identified Bloom-aware reader. Iceberg Java's reader has that pruning path. The inspected pinned iceberg-rust scan pipeline uses metrics/page/row filtering without loading Bloom filters. Footer byte equality and membership checks establish neither end-to-end speedup nor native-reader pruning benefit.

Design

The renamed-column fallback is an appropriate bounded fix while the bridge uses dot-separated physical paths. Comparing ancestors prevents a renamed parent from escaping a leaf-only check. Centralizing runtime capability filtering keeps eligibility and serialization consistent, while retaining the explicit JVM fallback for sizing that cannot be represented. No additional design blocker was found in the follow-up.

Abstraction & complexity

The path-resolution result earns its two fields by carrying the mapping and the fallback decision together. The synthetic-NDV adapter remains relevant because the current Parquet writer configuration sizes from NDV/FPP rather than accepting the desired byte cap directly. Its round-trip check makes that boundary explicit. Update its API calls for Parquet 59, but no further abstraction is needed for these fixes.

[sunchao]

Correctness

[P2] Use the Parquet 59 Bloom API after the rebase

The current lockfile resolves parquet 59.3.0, where BloomFilterProperties.fpp and .ndv are private. These three field accesses therefore fail with E0616 when the native tests are compiled. Use bloom.fpp() and bloom.ndv(). Also update the new setter at line 685 from set_column_bloom_filter_ndv to set_column_bloom_filter_max_ndv: the old name is deprecated since 59.0 and fails the Rust CI action's -- -D warnings check. Separate-crate projections of the exact upstream API reproduce both failures, while getter/current-setter controls compile. Please run the native Rust checks after both updates.

[NikitaMatskevich]

My bad, Codex did not re-run the tests after rebase, the fix was merged here: fix: use Parquet 59 bloom filter APIs

[sunchao]

Correctness

The Parquet 59 API finding from the previous review is addressed at 709a3604. Production now calls set_column_bloom_filter_max_ndv, and the test uses fpp() and ndv(). In the exact locked Parquet 59.3.0 source, the old setter delegates to this method and the getters return the same stored values. The update preserves sizing behavior while removing both the deprecated call and private-field accesses. No new or remaining verified P1/P2 was found.

The incremental diff from fcef2c4f contains only those three changed lines. The authoritative base remains bb9e7402. The other eight authored files, lockfile and Rust CI policy are unchanged. The earlier sanitized-name fallback and independent FPP/NDV capability fixes remain intact, including the FPP-only test path. Maintained Spark 3.5/4.0 quoted-name and V2 write semantics remain the compatibility reference. This API adjustment does not change column names, null/value handling, errors, expression modes or commit/abort behavior.

Validation

Diff and Rust format checks pass. The exact API source and preserved positive controls support the fix, but no new local native/JNI/Spark execution is credited. At the September 8, 12:26 UTC refresh, all three current-head workflows are action_required. Their subsequent job inventories contain zero jobs, including the CI run. The reported merge has the assigned head/base as parents and exactly the head tree, but it has not supplied an executed Rust check. The author's latest reply acknowledges that tests were not rerun after the earlier rebase and links this fix. Maintained Spark 3.4/4.1 source coverage remains unavailable.

Performance

No new timing, peak-memory or reader-pruning results were supplied. The API replacement adds no work beyond the operation already performed by the deprecated alias. The previous performance qualification remains open: folded serialized size does not establish lower retained allocation, writer speedup or native-reader pruning. The requested matched native/JVM enabled/disabled writer and Bloom-aware reader measurements remain the evidence needed for those claims.

Design

Calling the supported dependency API directly is the simplest fix. The existing eligibility, path mapping and JVM fallback boundaries are unchanged. No additional design concern was found in this update.

Abstraction & complexity

The change adds no wrapper, compatibility shim or new abstraction. Keeping the dependency-specific API usage inside the existing writer-property construction and its test preserves the current structure without adding indirection.

[sunchao]

Correctness

No new or remaining verified P1/P2 at 298cdc76. Relative to the previous approval on 709a3604, this update only adjusts Markdown table padding and removes two unnecessary Scala s prefixes. Both literals contain neither interpolation nor escape sequences, so the fallback message and generated SQL are unchanged. The authoritative base remains bb9e7402.

The six other authored files, native dependency lock and Rust CI policy are byte-identical to the previously reviewed revision. The Parquet 59 public setter/getter fix, renamed-path fallback, independent FPP/NDV capability gates and regression coverage remain intact. This update preserves the previously checked Spark 3.5/4.0 write behavior.

Validation

Authored, base and incremental diff checks and the Rust format check pass. All discussion was reread and reconciled with the supplied 2026-09-08 14:19:11 UTC snapshot. The only new discussion is the previous approval. There are no new author test results. At the September 8, 14:26 UTC refresh, all three current-head workflows are action_required, and subsequent inventories show zero jobs, including the CI run. The reported merge has the assigned parents and the head tree, but supplies no executed test result. No current native/JNI/Spark execution is credited. Maintained Spark 3.4/4.1 source coverage remains unavailable.

Performance

There is no new benchmark evidence or change to writer, sizing or reader logic. The earlier measurement requests remain outstanding: matched native/JVM writes with filters enabled and disabled, peak and retained allocation, serialized filter bytes, and equality/IN pruning measured through an identified Bloom-aware reader. Folding retains vector capacity, so smaller serialized filters alone establish neither lower retained memory nor a native-reader speedup. This formatting follow-up adds no measured performance claim.

Design

The edit keeps the existing eligibility and JVM fallback boundaries intact. Removing interpolation from fixed text is appropriate and leaves the adjacent SQL interpolation for the table and NDV unchanged. No further design issue was found.

Abstraction & complexity

No helper, compatibility layer or new abstraction is introduced. The two literals become simpler without changing the surrounding control flow or test scope.

[NikitaMatskevich]

Thank you for approval! We synced with @alessandro-nori and his approval is not required. If the CI is green we can merge

[jordepic]

I cross-checked this against Iceberg 1.11 (plus the 1.5.2, 1.8.1 and 1.10.0 tags), parquet-java 1.17, arrow-rs 59.3.0 (what the lockfile resolves) and iceberg-rust at the pinned rev. The sizing math is a faithful mirror of BlockSplitBloomFilter, including the & ~256 rounding quirk, the strict-inequality bug that turns a 32-byte cap into a no-op, and the long overflow of -8 * n. Iceberg 1.11 keys the column map by canonical findColumnName, so the canonical-only resolution here is right. iceberg-rust names list elements element and map structs key_value/key/value, and parquet-rs keeps those names with coerce_types off, so the physical leaf paths line up with TypeToMessageType. The new write.parquet.bloom-filter-adaptive-enabled property falls back through the vetted-key gate. Nice work on all of that.

The one place I think parity actually breaks is the folding behaviour, and it is broader than the description suggests. Details inline. A few smaller things I would also like to see addressed:

• Boolean leaves diverge. parquet-java's ColumnValueCollector.write(boolean) never touches the filter, so Java emits an all-zero filter at max-bytes. parquet-rs inserts bools through AsBytes for bool and then folds. Harmless for readers, but not identical. Could we either skip boolean leaves or fall back for them?
• Older Iceberg runtimes do not behave the way Comet emulates. Iceberg 1.5.2 passes the logical name straight to parquet-java, so Java writes no filter for a nested leaf while Comet writes a correct one. Iceberg 1.8.1 resolves through schema.findField, which accepts short names such as tags.a for a list of structs, and the canonical-only map here would skip those. Is it worth gating nested paths on the runtime the same way FPP and NDV are gated, or at least noting the difference in the docs?
• The branch currently conflicts with main and no checks have run on 298cdc7, so the write suites have not exercised this revision in CI yet.

[jordepic]

This is where the parity story breaks, and I do not think it is limited to the no-NDV case the description and tests frame it as.

In parquet 59.3.0, ColumnValueEncoderImpl::flush_bloom_filter unconditionally calls Sbbf::fold_to_target_fpp, and BloomFilterProperties has no way to turn that off. Folding is driven purely by observed fill, so it happens whenever a row group is sparser than the design density, regardless of whether NDV was set. With an explicit NDV that is typically the table-wide cardinality, most row groups hold fewer distinct values than that, so they fold too. The byte-identity tests pass because they insert exactly the design cardinality.

The effect on readers is the opposite of what the PR description argues for. Java keeps the full allocation, so its real false-positive rate on a low-cardinality column is close to zero. Comet folds until the estimated rate approaches the configured FPP, so a point lookup on such a column prunes up to about 1% fewer row groups than it would on a Java-written file. That is within the user's configured contract, but it is a measurable pruning regression against the classic writer, which is exactly what the sizing work here set out to avoid.

The bitset content depends only on size and inserted values, which the folded-versus-JVM test in the action suite demonstrates nicely. So an upstream BloomFilterProperties toggle to skip folding would give exact parity in every case. Until that exists, could we gate this behind a Comet config that defaults to the JVM path when any bloom column is enabled, or at minimum make the docs state plainly that native filters are smaller and have a higher realised FPP than Java's for sparse row groups? The current iceberg-writes.md wording of "preserving the requested FPP" reads as if nothing observable changes.

[andygrove]

Measured this on Spark 4.1 with Iceberg 1.11 to put a number on it. A column configured enabled=true, fpp=0.01, ndv=1000, with a single distinct value inserted: the native writer emits a 32-byte filter and parquet-mr emits 2048, a 64x difference on an explicit-NDV column. Membership holds on both.

That matches what you and sunchao derived from source, and it confirms the divergence is not confined to the no-NDV case the description frames it as.

[jordepic]

These links, the ones at line 749 and the PR description all cite arrow-rs 58.4.0, but the lockfile resolves parquet 59.3.0. That matters here because 59 is exactly where max_ndv semantics and post-insert folding arrived, and the doc comments on parquet_rs_bloom_filter_bytes describe the 58 sizing behaviour without mentioning that the allocation is then folded. Could we point these at 59.3.0 and note the fold?

[andygrove]

There is a third one outside the Rust file worth catching in the same pass. The scaladoc on requireNativeSupportedBloomFilterProperties in CometIcebergNativeWrite.scala also opens with "parquet-rs 58.x represents Bloom filters as a power-of-two number of bytes", and synthetic_ndv_for_bloom_filter_bytes says "using parquet-rs 58.x's public NDV/FPP setters" while the code now calls the 59 setter.

[jordepic]

(0.0..1.0).contains(&fpp) accepts an FPP of exactly 0.0, while the message says strictly between 0 and 1 and parquet-java's withBloomFilterFPP rejects it. The JVM gate makes this unreachable today, but the native check is the last line of defence if that gate ever changes, so I would make it fpp > 0.0 && fpp < 1.0.

[andygrove]

Agreeing, and adding why this one has more teeth than it looks. In parquet 59.3.0 set_column_bloom_filter_fpp does not return an error for an out-of-range value, it panics. ColumnProperties::set_bloom_filter_fpp calls validate_bloom_filter_fpp, which rejects !(fpp > 0.0 && fpp < 1.0), and the caller does panic!("{msg}") (properties.rs:1570 and :1663). So the failure mode is an abort inside the native library across JNI, not a differently sized filter.

It is latent today. I traced the only shape that reaches the setter with fpp == 0.0: the denominator collapses to -0.0, so synthetic_ndv_for_bloom_filter_bytes returns Err for every target except 32 bytes, which also needs max-bytes=32, and the JVM gate rejects an explicit zero before either. Still, fpp > 0.0 && fpp < 1.0 costs nothing and it is the difference between a DataFusionError and a panic if that gate ever moves.

[jordepic]

This rejects a non-power-of-two write.parquet.bloom-filter-max-bytes even when no column has a bloom filter enabled, in which case the value never influences the written file. Could the check be skipped when configured is empty so those tables stay on the native path?

[jordepic]

Folding is not specific to the no-NDV case. parquet-rs folds on observed fill, so the explicit-NDV cases avoid it only because the test inserts exactly the design cardinality. It might be worth adding a case with an explicit NDV and fewer inserted distinct values, and asserting the fold there, so the divergence from Java is documented by a test rather than hidden by data choice.

[sunchao]

Correctness

I rechecked the unchanged 298cdc76 source after jordepic's review. One newly verified [P2] remains in the reported Iceberg 1.8.1 nested-name case. For tags ARRAY<STRUCT<a: INT>>, write.parquet.bloom-filter-enabled.column.tags.a=true is valid in that runtime: its writer calls schema.findField("tags.a"), which resolves the short name to the same field ID as tags.element.a, then configures tags.list.element.a. Comet builds its map from findColumnName(id), which returns only the canonical name, and its exact lookup drops tags.a. The write can remain native with the requested filter silently absent. The caller preserves the table-property spelling, and neither the numeric gate nor the renamed-field fallback catches this case.

Please preserve that runtime's accepted names through field-ID resolution, or fall back for configurations the native path cannot reproduce, and add a Spark 3.5/Iceberg 1.8.1 footer comparison using a short nested name. Iceberg 1.10/1.11 use canonical-name maps, so unconditionally accepting aliases on every runtime would introduce the opposite mismatch. This confirms an issue already raised in the linked review, so I have not duplicated it as an inline comment. The current approval remains recorded. This comment supplies the additional finding without dismissing or replacing it.

The other version/type observations need narrower conclusions. Iceberg 1.5.2 passes configured names directly to Parquet, so the new logical-to-physical translation can add filters that its Java path omits. For boolean leaves, the inspected Java collector does not insert boolean values, while parquet-rs inserts their byte representation and folds. Their filter bytes therefore differ, but the inspected Iceberg Bloom reader does not prune BOOLEAN columns. Neither observation demonstrates incorrect row results. The native FPP check accepts zero syntactically, but the current JVM gate rejects an interpreted zero before serialization. Runtimes that ignore FPP remove that property and use the positive default. I found no reachable FPP=0 failure in the current write path.

Validation

All nine authored files match the previously approved head, and the incremental diff is empty. The maintained Spark 3.5/4.0 V2 append and commit/abort paths were checked and are unchanged. The finding concerns honoring a writer property, not Spark expression values or commit semantics. The Spark 3.5 profile pins Iceberg 1.8.1. Maintained Spark 3.4/4.1 source coverage remains unavailable. This review uses pinned dependency source and a source-derived folding arithmetic control, not a full writer or Spark/JNI execution. All three current-head workflows remain action_required with zero jobs, including CI. The PR is conflicting and has no current synthetic merge SHA to credit as executed test provenance.

Performance

The folding observation is confirmed in locked Parquet 59.3.0: flush always invokes fold_to_target_fpp, and the fold selector uses observed fill without testing whether NDV was explicit. A source-derived example with configured NDV 1,000, FPP 0.01 and one actual distinct value starts both writers at 2,048 bytes. The native selector folds to 32 bytes while non-adaptive Java retains 2,048. OR folding preserves membership, but a smaller filter can have a higher realized FPP than the oversized Java filter. The external approximate pruning percentage is not a measured regression, and meeting a target-FPP estimate does not establish equal reader performance.

The tests explicitly hold the ordinary explicit-NDV cases at their estimated cardinality. Other byte-identity cases use a binding cap or an underestimated NDV, so it would be inaccurate to say every identity test inserts the configured NDV. They do not cover an overestimated explicit NDV with sparse actual input. The documentation already acknowledges sparse folding generally. Its allocation-parity statement must not be read as serialized-byte or realized-FPP parity. The remaining 58.x source citations do not describe the locked dependency version. Existing requests for matched write-time, memory, stored-byte and Bloom-aware reader measurements remain unfulfilled. Folding truncates length while retaining vector capacity, so smaller files alone establish no memory or speedup result.

Design

The empty-config max-byte rejection in the existing inline is real: validation runs before checking for configured columns. For a valid non-power-of-two cap and no enabled filter, that is conservative and does not protect any emitted Bloom data. However, the authoritative base already rejected that property through the unvetted-key gate, so this is not a newly introduced fallback regression. I have not elevated it to another P2. Any relaxation must still preserve Java's parsing behavior and the supported-runtime NDV setter's ability to enable a filter after enabled=false.

Abstraction & complexity

There is no new source abstraction in this discussion-only follow-up. The actionable gap belongs at the existing logical-name-to-field-ID boundary. Extending that boundary with the runtime's actual lookup semantics, while retaining the existing physical-path and renamed-field checks, is more focused than adding a second writer or duplicating the entire property translator. No additional abstraction change is requested.

[andygrove]

The folding divergence jordepic flagged is real, and it is worth putting a number on it before this lands. I wrote a throwaway test on Spark 4.1 with Iceberg 1.11: a column configured enabled=true, fpp=0.01, ndv=1000, with one actual distinct value inserted. The native writer emits a 32-byte filter and parquet-mr emits 2048. Membership holds on both, so this is a pruning-quality difference rather than a correctness one, but it is a 64x difference on an explicit-NDV column, which is exactly the case the description and the byte-identity tests present as unaffected. The existing identity tests only avoid it because they insert precisely the design cardinality. Could we either add a test that pins this divergence down, so it is documented rather than hidden by the data choice, or reconsider whether an enabled bloom column should stay on the JVM path until parquet-rs can turn folding off?

Separately, no CI has run on any revision of this branch. I checked it out and ran what I could locally: the three Iceberg write suites are green at 144/144, cargo test iceberg_write is 28/28 including all five new bloom tests, and clippy is clean with -D warnings. That confirms the parquet 59 API fix holds. It only covers Spark 4.1 with Iceberg 1.11 though, and the runtime gating and the 1.8.1 short-name case both live on the older profiles, so please rebase and let CI run before merging. The one rebase we know about introduced a compile error that only review caught, so results from before it do not say much about the current head.

The rest of my comments are inline and none of them are large. I did check a fair amount of the sizing work against the pinned sources rather than taking it on faith, and it holds up. parquet_mr_bloom_filter_bytes genuinely reproduces optimalNumOfBits plus initBitset, including the & ~256 quirk and the min then power-of-two then cap ordering. Splitting the path on . matches parquet-mr exactly, since ColumnProperty runs the configured name through ColumnPath.fromDotString. The renamed-field fallback does cover literal dots, because TypeToMessageType sanitizes every name through AvroSchemaUtil.makeCompatibleName. And the synthetic NDV encoding is more robust than I expected: I swept 14 FPP values across every supported target size and the tightest case still leaves about 26 percent slack in the NDV window, so floating point differences between the JVM gate and the native writer are not a practical concern. That is good work.

[andygrove]

The encoding here is sound, so this is about drift rather than a bug. bloomFilterSizesRepresentable and parquetMrRequestedBloomFilterBytes reimplement the same arithmetic as parquet_rs_bloom_filter_bytes and parquet_mr_bloom_filter_bytes on the native side, and nothing ties the two together. The Rust #[test]s cover the Rust half, and these two are private with no unit test at all, only indirect coverage through the detection suite. If someone edits one side and the gate ends up saying "representable" where synthetic_ndv_for_bloom_filter_bytes returns Err, the write does not fall back to the classic writer, it fails inside the task.

Could we add one shared table of (ndv, fpp, max-bytes) to expected bytes and assert it from both sides? IcebergWriteProtoTranslationSuite is a plain AnyFunSuite, so it can host the Scala half cheaply if these become private[operator]. While you are there, the successful branch of the binary search is never exercised. impossible_synthetic_ndv_is_rejected does enter the loop, but only on the degenerate path where every candidate returns 32 bytes, so the Ok(low) return has no coverage.

[andygrove]

This row states the FPP and NDV rules unconditionally, but interpretedBloomFilterProperties makes them depend on the Iceberg runtime. On 1.5.2 both prefixes are dropped before validation, on 1.8.1 and 1.10.0 only NDV is, and only 1.11 interprets both. Someone on Iceberg 1.10 who sets bloom-filter-ndv.column.x gets neither the requested sizing nor a fallback, and the table as written does not predict that. Could the row name the versions that interpret each property, the way the write.parquet.shred-variants row already calls out Spark 4.x and Iceberg 1.11?

[andygrove]

insert("bloom_low_ndv_jvm") is the only JVM-side insert in the new tests that is not wrapped in withSQLConf(CometConf.COMET_ICEBERG_NATIVE_WRITE_ENABLED.key -> "false"). It works today because the conf defaults to false and the suite's sparkConf does not set it, so the comparison really is native against parquet-mr. If that default ever flips, this test quietly starts comparing Comet with Comet and passes for the wrong reason. Could it use the same explicit wrapper as its siblings?