Spark: Add support for 4.2.0

[manuzhang]

This PR adds support for Apache Spark 4.2.0.

Release publishing: Spark 4.2 artifacts are excluded from dev/stage-binaries.sh and will not be published by default.

The first two commits rename the Spark 4.1 module to 4.2 and then restore 4.1 so Git retains file history. The remaining commits contain the Spark 4.2 compatibility changes and review follow-ups.

Changes

• Adds and registers the Spark 4.2 modules in Gradle, settings, packaging, benchmarks, CI, and CVE scanning.
• Integrates Spark 4.2's RelationCatalog and View APIs while preserving Iceberg property-only updates for view properties, including cached and session catalogs.
• Updates view validation and metadata handling for Spark-owned view expansion, catalog-qualified cycle detection and output, compatibility with existing Iceberg views, metric-view dependencies, and de-duplicated table/view listings.
• Converts geospatial values between Spark's BinaryView representation and Iceberg's pure WKB Parquet representation, including StructInternalRow and nested values.
• Updates Spark 4.2 tests, benchmarks, APIs, and error expectations, and resolves the released Spark 4.2.0 artifacts from Maven Central.

Geospatial compatibility details

• Validate WKB during Spark conversion: Spark 4.2 parses and validates WKB when constructing Geometry and Geography values. It accepts the seven OGC base geometry types, including Z/M variants, and enforces longitude and latitude bounds for geography. Existing Iceberg data with extended geometry types or out-of-bounds geography coordinates can now fail with wkbParseError instead of passing through as it did in Spark 4.1. POINT EMPTY remains accepted. Iceberg uses Spark's public conversion API and cannot bypass this validation.
• Keep nested geospatial conversion in the Spark 4.2 module: Spark 4.2 requires nested geometry and geography values to use its BinaryView physical representation, so arrays, maps, and structs must convert Iceberg WKB values explicitly. Older Spark versions use different physical types and can be handled separately.

View compatibility details

• Adapt to Spark 4.2's view and relation APIs: Spark 4.2 requires catalogs that provide both tables and views to implement RelationCatalog. It also represents query text, current catalog and namespace, schema mode, SQL configs, dependencies, and properties through its new View model. SparkCatalog and SparkSessionCatalog translate that model to and from Iceberg view metadata and implement create, replace, create-or-replace, load, drop, and rename through the new signatures.
• Use Spark-owned view expansion:ResolveRelations now calls RelationCatalog.loadRelation before extension resolution rules, and BaseCatalog.loadRelation falls back from tables to views. Spark therefore expands Iceberg views into logical View nodes and applies GetViewColumnByNameAndOrdinal plus schema-mode-driven casting. This replaces Iceberg 4.1's positional expansion. The unreachable ResolveViews relation expansion and its identifier rewrites have been removed; Spark function resolution now owns references such as system.bucket.
• Route Iceberg commands around session-catalog fallbacks: Spark's ResolveSessionCatalog exits early or selects V1 handling for several view commands. RewriteViewCommands recognizes only Iceberg-backed V2 view catalogs before that happens, while preserving local and global temporary-view behavior, and routes create, drop, rename, describe, show-create, show-properties, and show-views commands to Iceberg-aware logical and physical nodes. Resolved commands carry the already-loaded view into inspection planning, avoiding a second catalog lookup.
• Restore validation and cycle detection: Rewriting CreateView means Spark's native CheckViewReferences no longer sees the original node. CheckViews explicitly reuses Spark's checks for temporary objects, generated aliases, and column counts, and traverses Spark logical View nodes for fully qualified recursive-view detection. Iceberg retains its complete nested-expression traversal because Spark's helper only matches a SubqueryExpression at the root of an expression.
• Preserve Spark-only metadata across Iceberg commits: Iceberg view metadata does not have dedicated fields for every Spark value. SparkView round-trips query column names as JSON (with legacy comma-separated read compatibility), SQL configs, schema-binding mode, metric-view dependencies, engine versions, and non-default view types through reserved Iceberg properties. Views without persisted schema-mode metadata use BINDING to preserve Spark 4.1's strict behavior, while Spark 4.2-created views persist their native mode explicitly; type-promotion coverage verifies the legacy fallback. Metric-view dependencies remain persisted because Iceberg has no native dependency field and Spark must receive the structured list again when loading the view.
• Keep updates compatible with Iceberg metadata: Iceberg-specific execution nodes keep ALTER VIEW SET/UNSET TBLPROPERTIES as property-only metadata commits because Spark 4.2 removed ViewCatalog.alterView, and its replacement path would discard non-Spark SQL representations. These commands translate identifiers through the owning catalog and invalidate cached plans. Rename preserves an existing cache entry. CREATE OR REPLACE VIEW removes omitted properties on a best-effort follow-up commit because Iceberg's ViewBuilder cannot yet express atomic property removals. The owning Spark catalog remains persisted in portable default-catalog metadata; the load path only falls back to the adapter's catalog name for older metadata where it is absent.
• Match Spark SQL inspection behavior: Iceberg-specific describe, show-create, and show-properties execution preserves property redaction, emits collation and schema mode in their DDL clauses, keeps metric-view type information, fully quotes catalog-qualified identifiers, and uses one reserved-property list so collation and internal metadata do not leak through another inspection command. View listing retains temporary views and avoids duplicate table/view entries without loading every view in the namespace.
• Generated aliases are now validated: Aligning with Spark's persistent-view checks means statements such as CREATE VIEW v AS SELECT id + 1 FROM t are rejected unless the expression has an explicit alias. spark.sql.legacy.allowAutoGeneratedAliasForView=true provides the Spark compatibility escape hatch.

Verification

• Focused TestViews coverage for legacy schema binding, recursive cycles, rename behavior, property replacement, and property ALTER across the existing catalog configurations.
• TestViewCatalogCache across named and session catalogs, covering replace, property removal, ALTER, rename, and cache invalidation.
• TestSparkCatalogIdentifierMapping, TestSparkCatalogOperations, TestSparkSessionCatalog, TestSparkParquetReader, TestSparkParquetWriter, TestAlterTable, and TestTimestampWithoutZone.
• Spark 4.2 core and extensions Spotless and Checkstyle checks for main and test sources.
• git diff --check.

---

AI Disclosure

• Model: GPT-5 Codex
• Platform/Tool: Codex
• Human Oversight: partially reviewed
• Prompt Summary: Add Spark 4.2 support and address compatibility issues in views, catalogs, relation listings, geospatial values, tests, and CI coverage.

[manuzhang]

This failure from testing Spark 4.2.0-preview2 is caused by apache/spark#53788, after which an AnalysisException would be thrown on Iceberg metadata tables like default.table.partitions.

TestAddFilesProcedure > addPartitionsWithNullValueShouldAddFilesToNullPartition() > catalogName = spark_catalog, implementation = org.apache.iceberg.spark.SparkSessionCatalog, config = {type=hive, default-namespace=default, parquet-enabled=true, cache-enabled=false}, formatVersion = 2 FAILED
org.apache.spark.sql.AnalysisException: [REQUIRES_SINGLE_PART_NAMESPACE] spark_catalog requires a single-part namespace, but got `default`.`table`. SQLSTATE: 42K05

[manuzhang]

Failed tests after upgrading to Spark 4.2.0-preview3-rc1

1. testJoinsHourToDays() in TestStoragePartitionedJoins.java:

Assertion failed: "SPJ should not change query output: number of results should match"
The actual and expected query result sizes differ, indicating that either the join logic or test data setup causes a mismatch.

2. readFromViewReferencingTempFunction() in TestViews.java:

Assertion failed: Expected a specific routine not found error, but got an AnalysisException with different message details.

[manuzhang]

apache/spark#54884 has been opened to fix the first failure.

[manuzhang]

I will update HourToDaysReducer following interface changes from apache/spark#54884 in next preview release. All other test failures have been fixed.

[szehon-ho]

leave a note here to implement the new method in the Reducer once apache/spark#54884 is in (next Spark 4.2 preview)

Sorry just saw, it is the same comment

[manuzhang]

The failed tests in 4.2.0-preview3 have been fixed in 4.2.0-preview4.

[manuzhang]

@szehon-ho Thanks for the thorough review. I've addressed all the comments, either fixing as recommended, or updating comments, or opening follow-up issues. Please take another look at your convenience. We are not in a rush to get this in now, thanks for @nssalian's heads-up.

[szehon-ho]

Thank you @manuzhang for persistence, sorry about the previous review if its not so clear. I'll try to make it more clear in the future. It's just a huge pr so hard to review as other folks mention.

I'm a bit swamped but try to take a look in next day or two.

By the way, @nssalian would one option maybe, we merge some version and then fix different parts in smaller prs? For the release, we will have the source code, but this won't make any release artifact, would that work?

[huan233usc]

Thank you @manuzhang for persistence, sorry about the previous review if its not so clear. I'll try to make it more clear in the future. It's just a huge pr so hard to review as other folks mention.

I'm a bit swamped but try to take a look in next day or two.

By the way, @nssalian would one option maybe, we merge some version and then fix different parts in smaller prs? For the release, we will have the source code, but this won't make any release artifact, would that work?

+1 to merging a version first and then fixing the individual parts in smaller PRs — right now it does feel like we're reviewing a moving target.

IIUC about the changes -- the bulk of the real logic is the view migration to Spark 4.2's RelationCatalog (Spark now owns view parsing/resolution, so ResolveViews changes a lot, plus the new Iceberg*V2View exec nodes and catalog wiring). The geo side looks mostly like a mechanical GeometryVal/GeographyVal → BinaryView migration.

We could do additional follow-ups items and check like cross-version view testing to guard against regressions.

[manuzhang]

@szehon-ho No worry. The review comments are valuable. It's just that how we can collaborate more efficiently on such a huge PR involving both AI and human.

By the way, @nssalian would one option maybe, we merge some version and then fix different parts in smaller prs? For the release, we will have the source code, but this won't make any release artifact, would that work?

+1. I can remove the codes to publish the 4.2 artifacts.

[nssalian]

By the way, @nssalian would one option maybe, we merge some version and then fix different parts in smaller prs? For the release, we will have the source code, but this won't make any release artifact, would that work?

This makes more sense. Let's get the baseline there and then we can add features on top of it.
@manuzhang is there a good baseline PR you can make if haven't already that we can aim for the 1.12 release?

[manuzhang]

@nssalian I think this PR is already in good shape after many rounds of reviews. The only remaining work is to remove the release codes.

[manuzhang]

This excludes Spark 4.2 support from source-release tarball.

[nssalian]

CC: @huaxingao@szehon-ho PTAL at the latest changes

[szehon-ho]

Thanks, the compatibility changes look sound overall. I found two remaining SparkSessionCatalog correctness issues in the inline comments.

For documentation, the user-visible behavior changes for Iceberg-backed views in Spark 4.2 are:

• View columns are matched by recorded name and occurrence ordinal instead of absolute position.
  • Example: after (id, data) is reordered to (data, id), Spark 4.2 preserves the view output as (id, data); Spark 4.1 may mis-map or fail.
• CREATE VIEW ... WITH SCHEMA { BINDING | COMPENSATION | TYPE EVOLUTION | EVOLUTION } is supported. The default is COMPENSATION.
• Views capture and replay relevant creation-time SQL settings.
  • Example: a view created with ANSI mode enabled still raises invalid-cast errors when queried from a session with ANSI mode disabled.
• Persistent views require explicit aliases for generated expressions.
  • Example: SELECT id + 1 now fails during view creation; use SELECT id + 1 AS new_id. spark.sql.legacy.allowAutoGeneratedAliasForView=true restores the old behavior.
• Persistent views referencing temporary variables are rejected.
• CREATE OR REPLACE VIEW removes user properties omitted from the replacement.
  • Example: replacing a view with TBLPROPERTIES (purpose=reporting) without specifying purpose removes that property; Spark 4.1 retained it.
• SparkSessionCatalog.listViews combines Iceberg and session-catalog views, so SHOW VIEWS may return more entries.
• listTableSummaries returns tables only. The new listRelationSummaries returns tables and views, and SHOW TABLES uses the combined listing.
• Recursive-view detection covers cross-catalog views, multi-level namespaces, V1 views, and nested expressions.
  • Example: if v2 references v1, CREATE OR REPLACE VIEW v1 AS SELECT * FROM v2 is rejected during analysis with a recursive-view error.
• View output column names containing commas now round-trip correctly.
  • Example: aliasing id to the single column name id,data preserves it as one name; previously it could be interpreted as two names.
• Replacing a view or changing its properties invalidates its cached plan. Renaming a cached view preserves the cache under the new name.
• View inspection commands (SHOW CREATE TABLE, SHOW TBLPROPERTIES, and DESCRIBE EXTENDED) produce updated output.

I think there's no way around it as we are moving from Iceberg extension to Spark view behavior.

[GGraziadei]

According to the conversation and considering the last commit, LGTM!

[szehon-ho]

The follow-up changes, including the schema-mode correction, look good. There is just one more small production correctness problem in the cross-backend collision handling, covered by the two inline comments.

[szehon-ho]

lgtm. I think from https://lists.apache.org/thread/821bjvx2jym709yvt56xl6vm733z742t that @aokolnychyi wanted to take another look as well before we officially release Spark 4.2 support (I think he will soon return from vacation). But it looks ok from my side and now we are not going to release a jar for 1.12

[szehon-ho]

Merged, lets continue iterating on the code. Thanks @manuzhang for the persistence and everyone for additional reviews

[anuragmantri]

Thanks @manuzhang for the PR and everyone for the thorough reviews.