Spark: Add support for Spark 4.1 by manuzhang · Pull Request #14155 · apache/iceberg

manuzhang · 2025-09-22T16:13:34Z

No description provided.

manuzhang · 2025-09-23T17:00:04Z

Created apache/spark#52423 at Spark side to fix failed tests for Spark 4.1.0-preview1.

025-09-22T16:49:13.6462602Z TestCreateActions > testAddColumnOnMigratedTableAtEnd() > catalogName = spark_catalog, implementation = org.apache.iceberg.spark.SparkSessionCatalog, config = {type=hive, default-namespace=default, parquet-enabled=true, cache-enabled=false}, type = hive FAILED
2025-09-22T16:49:13.6466422Z org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 431.0 failed 1 times, most recent failure: Lost task 0.0 in stage 431.0 (TID 3270) (localhost executor driver): java.lang.NullPointerException: Cannot invoke "org.apache.spark.sql.types.DataType.transformRecursively(scala.PartialFunction)" because "type" is null
2025-09-22T16:49:13.6469913Z at org.apache.spark.sql.vectorized.ColumnVector.<init>(ColumnVector.java:341)
2025-09-22T16:49:13.6471238Z at org.apache.iceberg.spark.data.vectorized.ConstantColumnVector.<init>(ConstantColumnVector.java:41)

…ith null DataType ### What changes were proposed in this pull request? Check whether the parameter DataType is null in ColumnVector constructor before transforming it ### Why are the changes needed? A subclass of ColumnVector, e.g. Iceberg's [ConstantColumnVector](https://github.com/apache/iceberg/blob/main/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ConstantColumnVector.java#L41), could be created with null `DataType`. It throws NPE after #51349, which can be verified by failed tests in [integrating Spark 4.1.0-preview1 in Iceberg](apache/iceberg#14155) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? UT. ### Was this patch authored or co-authored using generative AI tooling? No. Closes#52423 from manuzhang/SPARK-53678. Authored-by: manuzhang <owenzhang1990@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

manuzhang · 2025-11-18T08:40:16Z

 qualifyTableIdentifiers(
 qualifyFunctionIdentifiers(
-SubstituteUnresolvedOrdinals.apply(CTESubstitution.apply(plan)),
+ CTESubstitution.apply(plan),


SubstituteUnresolvedOrdinals removed and not needed after apache/spark#50606

raphaelauv · 2025-12-17T01:32:12Z

will it be part of iceberg 1.10.1 ? ( since spark 4.1.0 was release today ) thanks

nastra · 2025-12-17T06:52:11Z

will it be part of iceberg 1.10.1 ? ( since spark 4.1.0 was release today ) thanks

No this won't be part of 1.10.1 since 1.10.1 is a patch release

pan3793 · 2025-12-19T07:18:48Z

Spark 4.1.0 jars are available in Maven Central, do you guys have a plan to move this forward?

manuzhang · 2025-12-19T14:11:01Z

@pan3793 Since Spark 4.1.0 has known issues and Spark 4.1.1 will be released soon, we can wait for 4.1.1 RC and test again.

aokolnychyi · 2025-12-22T20:32:26Z

@manuzhang, what about getting this in now? Can you update the PR to include any changes that went in while it was open?

The only open question I had was related to MERGE behavior but we are reconsidering that in 4.1.1, so we will test it separately.

aokolnychyi · 2025-12-22T20:33:45Z

By the way, thanks for working on this, @manuzhang. I know it is tiring to keep updating such PRs.

aokolnychyi · 2025-12-22T20:39:35Z

Let's just update to the official 4.1.0, not RC. Forgot to ask.

aokolnychyi · 2025-12-23T02:56:57Z

Double checked the last commit, looking good. Waiting for the tests.

aokolnychyi · 2025-12-23T03:15:35Z

Thanks, @manuzhang @pan3793!

pan3793 · 2025-12-23T03:19:52Z

Great, thanks for @manuzhang's work and @aokolnychyi's merging!

github-actionsBot added spark INFRA build labels Sep 22, 2025

manuzhang force-pushed the spark4.1-preview branch 2 times, most recently from f506107 to 7c1ebb6Compare September 23, 2025 16:00

manuzhang mentioned this pull request Sep 23, 2025
[SPARK-53678][SQL] Fix NPE when subclass of ColumnVector is created with null DataType apache/spark#52423
Closed

manuzhang changed the title ~~Spark: Add support for Spark 4.1 (4.1.0-preview1)~~Spark: Add support for Spark 4.1Sep 24, 2025

manuzhang force-pushed the spark4.1-preview branch 5 times, most recently from 7df730b to 030d9e8Compare September 26, 2025 02:05

manuzhang mentioned this pull request Sep 26, 2025
AWS, Spark, Flink: Remove org.jetbrains.annotations#14192
Merged

manuzhang force-pushed the spark4.1-preview branch 3 times, most recently from cac12c7 to ad82343Compare September 30, 2025 15:08

manuzhang added the not-stale label Oct 19, 2025

manuzhang force-pushed the spark4.1-preview branch from ad82343 to 5020976Compare October 19, 2025 16:37

manuzhang force-pushed the spark4.1-preview branch 3 times, most recently from c3f856b to 13d315bCompare October 27, 2025 09:11

manuzhang mentioned this pull request Oct 27, 2025
[SPARK-53656][SS] Refactor MemoryStream to use SparkSession instead of SQLContext apache/spark#52402
Closed

manuzhang force-pushed the spark4.1-preview branch from 13d315b to 22d51beCompare November 4, 2025 06:12

manuzhang force-pushed the spark4.1-preview branch 4 times, most recently from 0e2436b to 612a72aCompare November 18, 2025 08:32

manuzhang commented Nov 18, 2025
View reviewed changes

pan3793 reviewed Dec 1, 2025
View reviewed changes

Comment threadspark/v4.1/spark-extensions/src/jmh/java/org/apache/iceberg/spark/PlanningBenchmark.java Outdated

manuzhang force-pushed the spark4.1-preview branch from 4f994d6 to e24f002Compare December 1, 2025 11:53

manuzhang requested a review from pan3793 December 2, 2025 01:08

manuzhang force-pushed the spark4.1-preview branch 2 times, most recently from 11a7ba2 to 6ef2c11Compare December 10, 2025 02:00

manuzhang force-pushed the spark4.1-preview branch from 6ef2c11 to d2468a1Compare December 12, 2025 01:46

aokolnychyi approved these changes Dec 22, 2025
View reviewed changes

manuzhang added 3 commits December 23, 2025 09:34

Spark: Move 4.0 as 4.1
347c3fc

Spark: Copy back 4.1 as 4.0
c26c965

Spark: Initial support for 4.1.0
3b52e92

manuzhang force-pushed the spark4.1-preview branch from d2468a1 to 3b52e92Compare December 23, 2025 01:49

aokolnychyi merged commit 3179684 into apache:mainDec 23, 2025
47 checks passed

openinx mentioned this pull request Jan 8, 2026
[Build] Set the Spark 4.1.0 as the default version. delta-io/delta#5798
Merged
5 tasks

HeartSaVioR mentioned this pull request Feb 2, 2026
[WIP] Use CollectMetrics for numOutputRows in streaming sources HeartSaVioR/spark#11
Open

stevenzwu mentioned this pull request May 19, 2026
Docs: Add release notes for 1.11.0 #16431
Merged

dongjoon-hyun mentioned this pull request May 21, 2026
[SPARK-56985] Upgrade Iceberg example to use Iceberg 1.11.0 apache/spark-kubernetes-operator#681
Closed

sameerz mentioned this pull request May 22, 2026
[FEA] Add support for Apache Iceberg 1.11 NVIDIA/cudf-spark#14853
Closed
2 tasks

res-life mentioned this pull request May 26, 2026
Iceberg 1.11 support for Spark 411, part (2/3): add iceberg-1-11-x module NVIDIA/cudf-spark#14882
Merged
8 tasks

huaxingao mentioned this pull request Jun 1, 2026
Site: Add 1.11.0 release blog post #16516
Merged

rahulsmahadev mentioned this pull request Jun 10, 2026
Spark: Add Spark 4.2 module (copy of Spark 4.1) #16751
Closed

Navigation Menu

Uh oh!

Spark: Add support for Spark 4.1 - #14155

Spark: Add support for Spark 4.1#14155
aokolnychyi merged 3 commits into
apache:mainfrom
manuzhang:spark4.1-preview

manuzhang commented Sep 22, 2025

Uh oh!

manuzhang commented Sep 23, 2025•
edited
Loading

Uh oh!

manuzhangNov 18, 2025

Uh oh!

Uh oh!

raphaelauv commented Dec 17, 2025

Uh oh!

nastra commented Dec 17, 2025

Uh oh!

pan3793 commented Dec 19, 2025

Uh oh!

manuzhang commented Dec 19, 2025

Uh oh!

aokolnychyi commented Dec 22, 2025•
edited
Loading

Uh oh!

aokolnychyi commented Dec 22, 2025

Uh oh!

aokolnychyi commented Dec 22, 2025•
edited
Loading

Uh oh!

aokolnychyi commented Dec 23, 2025

Uh oh!

Uh oh!

aokolnychyi commented Dec 23, 2025

Uh oh!

pan3793 commented Dec 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

manuzhang commented Sep 22, 2025

Uh oh!

manuzhang commented Sep 23, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

manuzhangNov 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

raphaelauv commented Dec 17, 2025

Uh oh!

nastra commented Dec 17, 2025

Uh oh!

pan3793 commented Dec 19, 2025

Uh oh!

manuzhang commented Dec 19, 2025

Uh oh!

aokolnychyi commented Dec 22, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aokolnychyi commented Dec 22, 2025

Uh oh!

aokolnychyi commented Dec 22, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aokolnychyi commented Dec 23, 2025

Uh oh!

Uh oh!

aokolnychyi commented Dec 23, 2025

Uh oh!

pan3793 commented Dec 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

manuzhang commented Sep 23, 2025•
edited
Loading

aokolnychyi commented Dec 22, 2025•
edited
Loading

aokolnychyi commented Dec 22, 2025•
edited
Loading