fix: narrow Iceberg complex-null scan fallback to struct columns

[ErikBPF]

Which issue does this PR close?

Closes #5731

Rationale for this change

Spark's InferFiltersFromGenerate adds IS NOT NULL and size > 0 filters below eligible non-outer generators such as explode. Rejecting every complex-column null predicate therefore makes ordinary list/map queries fall back to Spark scans.

List/map null checks can retain native scanning without native residual pushdown. When iceberg-rust cannot bind a residual, Comet's native planner skips it. Iceberg's Spark scan builder retains the post-scan predicates needed for row-level correctness and includes their columns in the projection. The existing direct struct-column null-check fallback remains conservative and unchanged.

What changes are included in this PR?

• Narrow the scan-rule guard to top-level struct columns. Struct elements inside lists/maps do not trigger this guard.
• Correct comments, fallback diagnostics and the Iceberg user guide to distinguish scan eligibility from residual predicate pushdown.
• Update the existing list/map scan expectations and expand the regression coverage using the established Spark-answer and native-scan helpers.
• Align declared and runtime GetArrayStructFields element nullability with containsNull || field.nullable, including required fields inside nullable struct elements. Preserve the input list's outer nullability.

How are these changes tested?

CometFuzzIcebergSuite covers IS NULL and IS NOT NULL for lists, maps and structs. Its fixture distinguishes null collections, empty collections, null elements/values and non-null structs containing null fields. Predicates project only row IDs, with explicit expected rows and complete Spark-result comparisons. List/map scans must remain native; direct struct null checks must fall back.

The same regression compares complete results and requires native scans for array/map explode and explode_outer, covering Spark's inferred null predicates and preserved outer rows.

CometIcebergNativeSuite retains the existing primitive collection, element/key access and struct fallback coverage. CometScanRuleSuite exercises the scan-rule regression surface. These suites are already registered in the PR workflow.

The required-child regression creates an Iceberg list with nullable struct elements and a required integer field. SELECT id, l.a FROM t WHERE l IS NOT NULL compares Spark results, checks explicit expected rows for [NULL], mixed elements and empty lists, and requires native scanning and projection.

The Rust regression covers 16 combinations of list/element/field nullability across List and LargeList. It reproduces Arrow 59.3.0's non-nullable-list-field panic on the original implementation and passes with the fix.

Local validation on Spark 4.0 / Iceberg Java 1.10.0: both Iceberg suites passed, 112 succeeded, 0 failed, 1 existing cancellation requiring Iceberg 1.11 schema-history support. Rust regression and formatting passed.

From the reactor root:

(cd native && cargo test -p datafusion-comet-spark-expr --lib get_array_struct_fields_nullability)
make core
./mvnw package -Pspark-4.0 -Dtest=none \
  -Dsuites=org.apache.comet.CometIcebergNativeSuite,org.apache.comet.CometFuzzIcebergSuite

[sunchao]

Correctness

Reviewed 68f2f47eabee08ee10c9ef09caf7735907c343b7 against authoritative base b5a39c33a1b3bbe3693a0cf630007cd4c784358c. The change removes list/map columns from an existing whole-scan fallback for complex-column null predicates. It retains the top-level struct guard and adds a query test requiring native list/map scans and struct fallback. One P2 remains: the existing CometIcebergNativeSuite still asserts fallback for five affected list/map queries, including both direct IS NULL cases. Those expectations need to change with the behavior.

The actual safety path is different from the PR rationale. At pinned iceberg-rust 665c64e48e8d33797ecb1a421f327edd9b024879, schema accessors are still not built for list/map fields, and Reference::bind requires an accessor. Comet catches that bind failure and skips the residual. Thus this restores native scanning while the filter above the scan enforces the predicate. It does not establish working list/map predicate evaluation in iceberg-rust's Arrow visitor. Nested dotted references are also omitted by the existing residual serializer. I found no additional data-correctness defect in this narrowing.

The maintained Spark 3.5 and 4.0 branches evaluate IS NULL/IS NOT NULL from the value's nullness. Empty containers and containers holding null elements are non-null. Their InferFiltersFromGenerate rule does introduce IS NOT NULL for eligible array/map generator inputs. The matching Iceberg Java dependencies retain post-scan filters unless partition selection fully guarantees them, and include filter columns in the read projection. Normal list/map reads preserve the container arrays, including nested struct validity, through the unchanged reader path. Null struct parents, nested field access, field-ID projection, missing optional columns, partition metadata, and unsupported-type/default guards gain no new transformation in this PR. The retained direct-struct fallback is conservative and pre-existing. No new ANSI, overflow, or coercion branch is introduced. Maintained Spark 3.4 and 4.1 sources were unavailable, so source-level compatibility is not claimed for those versions.

Validation

The new test executes Spark and Comet queries, compares results, and inspects the executed native scan plan. Its three queries cover IS NOT NULL for a nullable list, map, and struct, with one populated row and one all-null row. It does not test IS NULL, empty containers, nested null parents, schema evolution, or prove native residual pushdown. Existing direct IS NULL and element/key tests provide useful coverage once their obsolete fallback assertions are corrected.

At the complete discussion/CI cutoff of 2026-09-06 02:39:49 UTC, there were no reviews, inline comments, or executed checks. CI, CodeQL, and Delta Contrib Build Gate were all action_required. I ran no local build, Scala/JNI tests, or benchmark and did not approve or rerun workflows. The merge commit has the assigned head/base parents and identical reviewed/supporting file blobs, but there is no executed CI checkout to credit.

Performance

The gate remains a plan-time schema/filter check, with a smaller candidate-column set and no new per-row work. Restoring native scans can avoid the broader fallback chain reported in issue #5731. The reported SF1 derived TPC-H improvement from 20.63 s to 16.93 s is author evidence that I did not reproduce or independently qualify. List/map residuals can still be skipped during binding, so native scan eligibility should not be presented as improved native predicate pruning. No new expression kernel warrants a generic expression microbenchmark.

Design

The localized gate change restores scan eligibility without altering filter enforcement, file-task planning, or reader ownership. Keeping the existing binding-failure path is essential to the source-level safety argument. The change should be paired with the existing integration tests' new expected behavior. The retained struct guard and string-based predicate matching are existing conservatism, not newly introduced abstractions.

Abstraction & complexity

The production change adds no helper, registry, or type hierarchy. A top-level StructType check is simple to follow, and recursive rejection of every struct inside a list/map would incorrectly conflate container nullness with element nullness. The new test reuses the established Spark-comparison and native-plan helpers. I found no additional abstraction or complexity issue.

[sunchao]

Correctness

[P2] Update the existing Iceberg fallback tests

Could you update the affected cases in CometIcebergNativeSuite along with this gate? The existing array/map IS NULL tests still call checkIcebergNativeScanFallback (lines 2358 and 2469), whose helper explicitly asserts that no CometIcebergNativeScanExec exists. Those assertions now reject the native scans this change enables. The array-element, whole-array equality, and map-key cases also retain fallback assertions based on the implicit NOT NULL restriction being removed here. This suite runs alongside CometFuzzIcebergSuite in the reader CI group, so adding the new test does not replace the conflicting expectations. Please convert the affected cases to native-scan assertions while retaining their Spark-result comparisons, and run that suite. This is a source-verified assertion conflict, not an observed CI failure: the current workflows are still action_required.

[ErikBPF]

Done — converted the five affected assertions (array IS NULL, array-element filter, whole-array equality, map IS NULL, map-key access) to checkIcebergNativeScan in 37509cb, so they now assert the scan stays native while the post-scan Comet filter enforces the predicate. Struct IS NULL and whole-struct equality remain as fallback cases with updated reasons (struct-only gate; Iceberg Java does not push whole-struct equality). CometIcebergNativeSuite: 99 succeeded, 0 failed (1 pre-existing version-gated cancel).

[sunchao]

Correctness

Re-reviewed d288b4679986148751f67f5391b731627d4121d6 against authoritative base b5a39c33a1b3bbe3693a0cf630007cd4c784358c. The prior P2 is addressed: all five affected CometIcebergNativeSuite cases now use checkIcebergNativeScan, which compares complete Spark results and asserts exactly one native Iceberg scan. Direct struct null checks retain their fallback assertions.

The expanded regression adds six IS NULL/IS NOT NULL queries over lists, maps and structs. Its five-row fixture distinguishes null containers, empty containers, null elements/values and non-null structs containing null fields. The predicates project only IDs and check explicit expected IDs as well as Spark parity. Four array/map explode and explode_outer queries compare complete results and require native scans, covering inferred non-outer null filters and preserved outer rows. These expectations agree with the maintained Spark 3.5/4.0 null-expression and generator-inference implementations.

I rechecked the retained-filter argument against the unchanged dependency pins and supporting source. iceberg-rust still creates no list/map accessors, and unary binding requires the accessor before predicate simplification. Comet catches that failure and skips the residual. Iceberg Java retains predicates requiring row-level evaluation and includes their columns in the projection; the scan substitution does not remove that filter. The revised comments and guide now describe this mechanism accurately. No new or remaining P1/P2 found.

At the complete discussion/CI cutoff of 2026-09-08 01:17:39 UTC, all four current-head workflows, including CI, were action_required, with no executed checks. The author reports 99 passing native-suite tests, but that report predates the final fixture expansion and I did not independently verify it. I ran source checks, not a local build, Scala/JNI suite or benchmark. The fetched merge's first parent differs from the assigned base, so it supplies no current-base integration proof. Maintained Spark 3.4/4.1 sources remain unavailable; no source-compatibility claim is made for those versions.

Performance

This follow-up changes tests, documentation, local names and diagnostic wording; it adds no scan-time or per-row work. The updated explanation correctly separates native scan eligibility from residual pruning. It makes no new measured performance claim, and there is no new expression kernel requiring a microbenchmark.

Design

The revisions make the existing boundary explicit: collection null predicates may retain native scanning while the filter above the scan enforces them; direct struct checks keep the conservative guard. ID-only projections exercise retention of filter columns, and the generator cases check the user-visible behavior that motivated the change. No additional execution path or ownership change is introduced.

Abstraction & complexity

The tests reuse the existing Spark-result and executed-plan helpers. Two small loops cover the null predicates and generator variants, with query-specific failure context. Renaming the guard's local variables to refer to structs clarifies its scope without introducing another abstraction. I found no simplification needed before merge.

[sunchao]

Correctness

Re-reviewed 3700467a3ca8cab7528fbfddcf8351bf7da0a93b against fefee03d94045ecd0ac5d3a1edb98a555f5ff21d, also the merge base. All four authored patches retain the prior added and removed lines. The 147-file increment comes from merging main. The five corrected native-scan assertions and the null-container/generator fixture remain intact. The prior test-expectation P2 stays addressed.

[P2] A required field inside a nullable array element still needs protection. Consider a nullable list l whose elements may be null, with a struct element containing required integer field a. For SELECT l.a FROM t WHERE l IS NOT NULL, a row containing [NULL] must yield [NULL]. Maintained Spark 3.5/4.0 explicitly combines the array's element nullability with the field's nullability for this extraction. Iceberg Java 1.10.0 preserves the required field in the scan schema, and Comet serializes that flag unchanged. The narrowed gate admits this list and the native extraction unions the struct-parent nulls into a, but then passes the original non-nullable a field to GenericListArray::new. Locked Arrow 59.3.0 rejects that combination and unwraps the error. The retained IS NOT NULL filter cannot remove this row because the list itself is non-null. The inline proposes correcting the extraction's declared and runtime element nullability, or retaining fallback for this case, with a required-child regression. This call chain is source-verified. I did not execute the full query.

The rebase's ordinary struct projection still propagates parent nulls. Its generator fast path excludes padding, non-contiguous views and populated null slots, preserving the existing outer-row path. The Variant adapter changes are gated by Variant metadata. Missing optional-column materialization and missing required-column errors remain in the unchanged locked Iceberg reader.

Validation

At September 9, 04:40 UTC, CI, CodeQL and the Delta build gate were action_required, with zero jobs. Only labeling succeeded, and its log proves a base checkout. Synthetic merge 3e5f581ce58b89dc22af2969efbde6b6868e9c5d has the assigned base/head parents and the head tree, but no build or test execution is established. Source/diff checks passed. No local build, native/JNI suite, query probe or benchmark ran. Maintained Spark 3.4/4.1 sources remain unavailable. This follow-up is a COMMENT because the P2 above needs to be addressed. I have not changed the earlier review record.

Performance

The merge adds no new authored scan work. The guard remains a planning-time type/filter check. Inherited generator changes can slice contiguous child buffers instead of gathering them, with the existing gather path handling padding. This review establishes no timing benefit. The required-field correction should change nullability metadata while preserving shared value buffers, avoiding an additional scan or value copy.

Design

The retained post-scan filter remains the correct fallback for unbound Iceberg residuals, but it cannot repair a downstream extraction's invalid array type. Correcting that expression's output nullability is more precise than rejecting every list containing a struct. The native reader's file planning and missing-column policy need no change for this case.

Abstraction & complexity

The update adds no authored abstraction. The shared parent-null helper has a clear role, but its callers must also describe the resulting nullability correctly. Reusing the existing extraction and Spark/native comparison helpers is sufficient for the fix and regression.

[sunchao]

Correctness

[P2] Preserve nullable elements when projecting a required struct field

Could this narrowing also fix the native GetArrayStructFields nullability, or retain fallback for that case? A nullable list l can have nullable struct elements with required integer field a. With l = [NULL], SELECT l.a FROM t WHERE l IS NOT NULL should return [NULL]. The old complex-column guard kept this query on Spark, while this gate admits it with native Iceberg enabled and the default Spark-to-columnar fallback disabled.

Iceberg Java preserves a as non-nullable, and the scan serde preserves that flag. GetArrayStructFields then unions the struct-parent nulls into the extracted child but still uses the original non-nullable field in both its declared list type and GenericListArray::new (declared type, array construction). Arrow 59.3.0 rejects the now-nullable values and new unwraps the error. The retained filter cannot help because [NULL] is a non-null list. Spark explicitly uses containsNull || field.nullable here.

Please align the declared and runtime element nullability and add this required-child case to the native-scan regression. The new fixture declares a nullable, so it does not cover it. This is verified from the maintained Spark 4.0, Iceberg Java and locked Arrow call chain, not a full query reproduction.

[ErikBPF]

Addressed in 266c540.

GetArrayStructFields now uses containsNull || field.nullable for both its declared list element field and runtime list construction, and derives outer nullability from the input list. The existing parent-null helper still shares value buffers.

Added the required-child Iceberg regression with [NULL], mixed null/non-null elements, empty lists and a null list, asserting Spark-equivalent results plus native scan and projection. A Rust regression covers 16 nullability combinations across List and LargeList; it reproduces the exact Arrow 59.3.0 panic on the original implementation and passes with the fix.

Validation: native build and JVM packaging passed. On Spark 4.0 / Iceberg Java 1.10.0, the full CometIcebergNativeSuite and CometFuzzIcebergSuite passed: 112 succeeded, 0 failed, 1 existing Iceberg 1.11-gated cancellation. Rust regression, Rust formatting and Scala formatting passed. Commands are in the updated PR description.

[ErikBPF]

@sunchao, thanks for catching the required-field/null-array-element case. I'm working on aligning the extraction's declared and runtime nullability and adding a required-child regression.

Would you be willing to share the review prompt or checklist you use, plus any validation commands and source/dependency checks? I'd like to run the same checks locally before submitting future contributions and catch more of these bugs upfront.

[sunchao]

Correctness

Re-reviewed e072305f3e3e5e39ff500e35bd5a311b679c500c against 4eeb1f80f0541f72389a11e6e2d0ee269d648c23. The required-field/null-array-element P2 is addressed. child_field now combines the input element's nullability with the selected field's nullability, matching containsNull || field.nullable in the maintained Spark 3.5/4.0 branches. Both data_type and the List/LargeList evaluation paths use that widened field. After the existing helper unions struct-parent nulls into the child, Arrow's list constructor therefore receives a field that permits those nulls. The expression's outer nullability now follows the input list, matching Spark's UnaryExpression, while runtime construction preserves its offsets and outer null mask.

The Iceberg regression creates a with Types.NestedField.required inside optional list elements and verifies its stored requiredness after inserting the data. Its filtered projection checks [NULL], mixed elements and empty lists against explicit expected rows and Spark results, then requires native scanning and projection. The Rust regression checks all 16 combinations of list, element and field nullability across List and LargeList, including declared/runtime type agreement and preservation of outer nulls. The earlier five corrected native-scan expectations remain intact, with direct struct fallback retained. I found no new or remaining P1/P2.

I verified the locked Arrow 59.3.0 constructor, field-update and parent-null-union chain from checksum-matched sources. The six inherited files in this update exactly match the base advancement, and the synthetic merge has the reviewed head's complete tree. At the September 9, 11:31 UTC final check, CI, CodeQL and the Delta gate awaited approval. Their job lists are empty. Only labeling ran, on the base commit. The author reports 112 passing Iceberg-suite tests, one existing cancellation, and a passing Rust regression that fails on the original code. That report predates the final merge and I did not independently verify its execution. I ran source checks, not a local build, native/JNI test or query reproduction. Maintained Spark 3.4/4.1 sources remain unavailable.

Performance

The correction adds field-metadata construction and schema lookup per evaluated batch. It preserves the existing shared value buffers, offsets and outer validity, and leaves the parent-null helper's no-work path intact. It adds no per-element value copy or additional scan. The existing extraction benchmark remains available for integer, string and nested-list fields. I did not run it, and this update establishes no measured speedup.

Design

Using child_field for both declared type and runtime construction removes the mismatch behind the prior failure. Deriving outer nullability separately from the input list keeps container nullness distinct from element nullness. Rebuilding through with_new_children recomputes the field from the replacement child, so the change introduces no cached schema that could become stale.

Abstraction & complexity

The fix reuses the existing extraction helper and parent-null propagation. The Rust test helper covers the schema combinations without another production abstraction. The programmatic Iceberg fixture is justified here because it explicitly establishes and checks required nested-field metadata that the earlier SQL fixture did not provide.

[sunchao]

Re-reviewed 296a8a434f02 against 4eeb1f80f054, including the increment from e072305f3e3e. The only new change replaces list.clone() with an explicit Arc::<GenericListArray<O>>::clone(&list) in the Rust regression. Production code and the Iceberg fixtures are unchanged. Both earlier P2 fixes remain intact, and I found no new or remaining P1/P2.

CI now supplies execution evidence that was unavailable at the previous review. The Rust job passed 1,264 tests, including get_array_struct_fields_nullability, with five skipped. The Spark 4.0 scan job passed both the collection-null scan regression and the required-field projection regression. Its full run had 491 successes, no failures, nine cancellations and one ignored test.

Both jobs checked out merge c0aae385050e1, which includes newer main. I verified that all five authored files and the dependency lockfile match this reviewed head. The update preserves the Spark 3.5/4.0 nullability behavior checked previously and adds no production allocation, execution path or abstraction. I ran source/equivalence checks and inspected CI logs, without a local build or benchmark. Maintained Spark 3.4/4.1 sources remain unavailable.