[#8771] feat(spark-connector): Add Spark 4.0 support, drop 3.3 and 3.4 by LuciferYang · Pull Request #12414 · apache/gravitino

LuciferYang · 2026-08-11T03:59:09Z

What changes were proposed in this pull request?

This revision replaces what this branch previously proposed. The earlier one added Spark 4.0 on top of the 3.3/3.4/3.5 matrix; review here asked for the 3.x matrix to shrink first and, per @diqiu50, for the initial submission to cover two versions so the approach and its maintenance cost can be judged from something concrete. The final matrix is Spark 3.5 and 4.0. Spark 4.1 exists locally as a follow-up and is not part of this PR.

Three commits, each self-contained:

1. Support Spark 3.5 only for the 3.x line. Spark 3.3 and 3.4 are past their maintenance windows; 3.5.x is on extended LTS through November 2027, so it is the only 3.x line that will see another release. With them gone, adaptations that existed only to bridge 3.3/3.4 collapse back into spark-common: the TimestampNTZType mapping and UpdateColumnDefaultValue conversion move into SparkTypeConverter and SparkTableChangeConverter, loadTable(Identifier, Set<TableWritePrivilege>) moves into BaseCatalog, and four converter subclasses plus the two version modules go away. SparkHiveTable and HiveGravitinoOperationOperator switch to the plural PartitionsAlreadyExistException, because SupportsPartitionManagement.createPartition changed its throws clause in Spark 3.4.

2. Compile spark-common as a shared source set. spark-common stops being a Gradle module. Each version module composes its own source set from spark-common/src/main/java, spark-common/src/main/spark<NN> and its own src/main/java. The test tree works the same way, replacing the testArtifacts jar the version modules used to consume.

Compiling the shared code once against one Spark version and running it on another is how a version-specific bug hides: the compiler only ever sees the API the pinned version exposes. Per-version compilation means each line's compiler checks the shared code against its own API, and the shared tests run on each line rather than only on the pinned one. Commit 3 shows this paying off immediately.

3. Add Spark 4.0 support. New v4.0/spark and v4.0/spark-runtime, publishing gravitino-spark-4.0_2.13 and gravitino-spark-connector-runtime-4.0_2.13. Spark 4 is Scala 2.13 only and needs JDK 17, so both modules pin 2.13 rather than reading -PscalaVersion, and they opt out of the repository's JDK 8 target.

Each version module owns its GravitinoSparkPlugin, at the same fully-qualified name, so spark.plugins is unchanged for users. It binds the classes its own Spark version needs and passes them to the shared driver plugin as SparkBindings, keyed by a shared SparkCatalogKind enum. The bindings are compile-time class references, so a renamed or missing class fails the build rather than the session, and adding a Spark version no longer touches shared code. The authorization parser is one of those bindings, because Spark 4 added an abstract parseRoutineParam to ParserInterface, so each module carries its own copy.

The one shared flavor directory left is spark35, holding the Paimon package: Paimon publishes no paimon-spark-4.x artifact at the version this repository pins, and no paimon-spark-3.5_2.13 either, so both the Spark 4 build and the Scala 2.13 build compile it out. The Paimon catalog is therefore the one binding named by string rather than by class reference, added only when the class is present, and the driver plugin skips the Paimon session extension when no Paimon catalog is bound rather than failing SparkSession construction.

Iceberg's loadProcedure moves out of the shared GravitinoIcebergCatalog into the per-version catalog subclasses. On 3.x ProcedureCatalog is a class Iceberg ships and loadProcedure returns Iceberg's Procedure; on 4.x Spark ships its own and the return type is UnboundProcedure. implements is declaration-level, so this one belongs in the version modules rather than in a flavor file.

Two fixes to the shared code, both surfaced by compiling the shared tests against Spark 4 for the first time:

SparkTypeConverter and SparkPartitionUtils checked instanceof StringType before VarcharType and CharType. Spark 4 made both extend StringType, so that branch swallowed them and silently dropped the length. Three shared tests caught it. Subtypes are now checked first, which is correct on every supported version.
the shared Iceberg IT built columns with new Column(Literal.create(...)). Spark 4 rebuilt Column on ColumnNode and dropped that constructor; it now uses functions.lit(), available on both.

The 4.0 ITs pin Jersey, HK2 and jakarta.validation back to the javax flavor on the test runtime classpath: the embedded Gravitino server serves REST on Jetty 9 with Jersey 2, and Spark 4 brings the jakarta flavor transitively via spark-hive. Spark's own web UI cannot coexist with that, so the ITs disable it. Both are test-only; the runtime jar bundles neither Jersey nor the server.

Why are the changes needed?

Spark 4.0 is GA and the connector topped out at 3.5, leaving Spark 4 users with no supported path. The VarcharType bug above is the concrete argument for per-version compilation: it was reachable from any Spark 4 session and invisible to a build that only ever compiled the shared code against 3.x.

Fix: #8771

Does this PR introduce any user-facing change?

Yes. Two removals, one addition, and one behavior change on the JDBC PostgreSQL catalog.

gravitino-spark-connector-runtime-3.3 and -3.4 are no longer published. Spark 3.3 and 3.4 users stay on an earlier Gravitino release.
gravitino-spark-common is no longer published. It was an implementation detail of this repository's own version modules, which now compile its sources directly; nothing outside the repository consumed it.
gravitino-spark-4.0_2.13 and gravitino-spark-connector-runtime-4.0_2.13 are new. On Spark 4 the connector requires Scala 2.13 and JDK 17, and the Paimon catalog is unavailable. Glue resolves a catalog class on 4.0 but is verified on 3.5 only, because the patched Hive JARs it needs are published for Spark 3 alone.
GravitinoPostgreSqlCatalogSpark35 extends the shared JDBC catalog directly rather than a *Spark34 subclass, so it never picked up the three version-specific overrides the other five catalogs had. Folding those into the shared classes gives it all three at once: timestamp_ntz columns are accepted instead of rejected; ALTER TABLE … ALTER COLUMN … SET DEFAULT works instead of failing in the connector (the PostgreSQL backend already supported it); and loadTable(Identifier, Set<TableWritePrivilege>) now routes through the write path, so write operations on a PostgreSQL catalog are checked against write privileges where previously they were not. A user whose grants cover reads but not writes can start seeing denials. This aligns jdbc-postgresql with jdbc-mysql, which has had all three since [#10181] feat(spark-connector): Support TableWritePrivilege for Spark 3.5+ authorization #10194.

How was this patch tested?

Unit tests cover the shared code on both lines, since the shared test tree compiles into each version module: type and property conversion, the provider-to-catalog dispatch, what the driver plugin writes under spark.sql.catalog.<name>, the Iceberg and Paimon opt-in flags over both catalogs and session extensions, and the binding validation. Each version module also has a test that builds its own GravitinoSparkPlugin, so a catalog a module forgot to bind fails there rather than at SparkSession startup, and v3.5 pins the string-named Paimon catalog against the class it ships.

Docker ITs run on both lines: Hive, Iceberg on the Hive backend, MySQL and PostgreSQL on 4.0, and the full 3.5 suite as a regression check. Both runtime shadow jars were built and inspected: the 4.0 jar carries its five *Spark40 catalogs and no Paimon classes, the 3.5 jar its own set plus Paimon on Scala 2.12.

github-actions · 2026-08-11T05:38:10Z

Code Coverage Report

Overall Project	68.81% `-0.34%`	🟢
Files changed	54.33%	🔴

Module	Coverage
aliyun	19.74%	🔴
api	51.62%	🟢
authorization-common	85.96%	🟢
authorization-ranger	4.38%	🔴
aws	53.54%	🟢
azure	32.1%	🔴
catalog-common	20.87%	🔴
catalog-fileset	82.17%	🟢
catalog-glue	69.24%	🟢
catalog-hive	82.96%	🟢
catalog-jdbc-common	45.69%	🟢
catalog-jdbc-doris	82.69%	🟢
catalog-jdbc-mysql	79.33%	🟢
catalog-jdbc-postgresql	83.39%	🟢
catalog-jdbc-starrocks	79.16%	🟢
catalog-kafka	76.99%	🟢
catalog-lakehouse-generic	60.55%	🟢
catalog-lakehouse-hudi	79.1%	🟢
catalog-lakehouse-iceberg	85.86%	🟢
catalog-lakehouse-paimon	84.29%	🟢
catalog-model	77.99%	🟢
cli	44.51%	🟢
client-java	77.37%	🟢
common	56.34%	🟢
core	83.83%	🟢
filesystem-hadoop3	76.48%	🟢
flink	0.0%	🔴
flink-common	52.1%	🟢
flink-runtime	0.0%	🔴
gcp	32.2%	🔴
hadoop-auth	68.0%	🟢
hadoop-common	17.84%	🔴
hive-metastore-common	53.4%	🟢
iceberg-aliyun-bundle	0.0%	🔴
iceberg-common	64.75%	🟢
iceberg-rest-server	75.96%	🟢
idp-basic	86.42%	🟢
integration-test-common	0.0%	🔴
jobs	62.92%	🟢
lance-common	32.52%	🔴
lance-rest-server	65.69%	🟢
lineage	53.02%	🟢
optimizer	83.24%	🟢
optimizer-api	21.95%	🔴
server	89.33%	🟢
server-common	80.67%	🟢
spark	56.49% `-0.19%`	🟢
tencent	81.78%	🟢
trino-connector	51.26%	🟢

Files

Module	File	Coverage
spark	GravitinoSparkPlugin.java	75.76%	🟢
	GravitinoSparkPlugin.java	75.76%	🟢
	GravitinoAuthorizationSparkSessionExtensions.java	51.43%	🔴
	GravitinoAuthorizationSparkSessionExtensions.java	51.43%	🔴
	GravitinoGlueCatalogSpark35.java	0.0%	🔴
	GravitinoHiveCatalogSpark35.java	0.0%	🔴
	GravitinoIcebergCatalogSpark35.java	0.0%	🔴
	GravitinoJdbcCatalogSpark35.java	0.0%	🔴
	GravitinoPaimonCatalogSpark35.java	0.0%	🔴
	GravitinoGlueCatalogSpark40.java	0.0%	🔴
	GravitinoHiveCatalogSpark40.java	0.0%	🔴
	GravitinoIcebergCatalogSpark40.java	0.0%	🔴
	GravitinoJdbcCatalogSpark40.java	0.0%	🔴
	GravitinoPostgreSqlCatalogSpark40.java	0.0%	🔴
spark-common	SparkCatalogKind.java	100.0%	🟢
	SparkHiveTypeConverter.java	100.0%	🟢
	SparkBindings.java	100.0%	🟢
	SparkTypeConverter.java	95.35%	🟢
	AuthorizationTable.java	95.24%	🟢
	SparkTableChangeConverter.java	92.75%	🟢
	SparkPartitionUtils.java	88.17%	🟢
	SparkJdbcTypeConverter.java	83.33%	🟢
	GravitinoDriverPlugin.java	65.9%	🟢
	GravitinoIcebergCatalog.java	34.26%	🔴
	BaseCatalog.java	21.09%	🔴
	SparkHiveTable.java	0.0%	🔴
	HiveGravitinoOperationOperator.java	0.0%	🔴

LuciferYang · 2026-08-11T06:32:34Z

Thank you @roryqi

diqiu50 · 2026-08-11T09:36:44Z

Since we are going to refactor this part, I think the architecture should follow two principles:

spark-common should not depend on any specific Spark version. All the suported spark vertion can build it.
Each supported Spark version, including minor versions, should depend only on spark-common and should not depend on other version-specific modules. using sourceset dependency is better than binary jars

The tests should follow the same structure as well. This will keep each Spark version independent and make it much easier to deprecate or remove old versions in the future.

LuciferYang · 2026-08-11T10:33:13Z

@diqiu50 Thanks for your suggestions. I'll take a look and get back to you later.

LuciferYang · 2026-08-11T13:46:04Z

Agreed on both principles, and I am happy to take the refactor as a separate PR ahead of this one, then rebase this one on top. It changes how all of 3.x builds, so it deserves review on its own. Below is what I found while sizing it, and a layout to check against before I start.

Where the version boundaries actually fall. I tried compiling spark-common against 3.5 and it fails on SparkHiveTable: SupportsPartitionManagement.createPartition declares the singular PartitionAlreadyExistsException in 3.3 and the plural PartitionsAlreadyExistException in 3.4, 3.5 and 4.0. So that split is 3.3 versus 3.4+. ParserInterface splits differently: 3.3, 3.4 and 3.5 all require the same 8 methods, and 4.0 adds parseRoutineParam. And Paimon is a third axis: paimon-spark-4.0 first appears in Paimon 1.3.0, and we pin 1.2.0 (libs.versions.toml:75), so no Spark 4 Paimon artifact exists at the version we build against.

Three different boundaries means the layout has to be additive srcDir lists rather than one directory per version:

dir	contents	used by
`java`	the 35 version-neutral files	all
`spark3x`	4 Paimon classes + authz parser (3.x form)	3.3, 3.4, 3.5
`spark33`	`SparkHiveTable`, `HiveGravitinoOperationOperator` (singular exception)	3.3
`spark34plus`	the same 2 with the plural exception, + the 4 `*34` converters	3.4, 3.5, 4.0
`spark40`	authz parser (4.0 form)	4.0

I checked the pieces that make spark34plus viable, and compiled that combination to be sure: shared sources plus the spark4 SparkHiveTable and HiveGravitinoOperationOperator plus the four *34 converters build cleanly against both 3.4.3 and 3.5.3. The PartitionsAlreadyExistException(String, InternalRow, StructType) constructor they use exists in both, and the four converters are byte-identical to their v3.4 originals. Moving them there removes both the copy this PR makes and the v3.5 to v3.4 edge, which is the concrete payoff of your principle 2. Today v3.5/spark depends on spark-connector:spark-3.4: four catalogs extend *Spark34, and GravitinoJdbcCatalogSpark35 uses the *34 converters.

The shared tests are the larger half. All four version modules consume spark-common's test classes as a testArtifacts jar, ~50 files compiled once against 3.3. Under "tests follow the same structure" that tree needs the same flavor split and per-version compilation, and testArtifacts stops being shareable as a binary. That is also what would make the 4.0 ITs exercise 4.0-compiled shared code instead of 3.3 bytecode, which they do not today. The Paimon tests are part of this: three files today, excluded by a Scala-version-keyed rule rather than a Spark-version one, so on the test side that rule does need to change even though the directory split handles the main sources.

For scale, the shared sources end up compiled once per supported (Spark, Scala) pair after the refactor: 3.3/3.4/3.5 under 2.12 and 2.13, plus 4.0 under 2.13, with the 2.13 ones excluding the Paimon subset.

One limit worth stating: the Iceberg catalog subclasses cannot fully follow principle 2. implements ProcedureCatalog resolves to a class Iceberg ships on 3.x and one Spark ships on 4.0, and implements is declaration-level, so those stay per-version. The refactor can reach "no cross-version module dependency" but not "no per-version code".

Questions:

Does the table above look right, and is additive srcDir composition acceptable? I would rather build the layout you want than guess at naming.
Should spark-common stay a Gradle module, or should the version modules compile its sources directly? Two things keep it from being purely sources-only: it owns extra["glueHiveJarsDir"] and the downloadGlueHiveJars task that all three v3.x modules reach through evaluationDependsOn, and it is the only producer of the testArtifacts jar the spark-connector modules share. Both are relocatable, but that is a decision rather than a file move. Note also that its compileJava is pinned to --release 8 by a path-prefix rule in the root build while the 4.0 compilation needs 17, which is why spark4-common had to be listed in jdk17OnlyProjectPaths.
If it stays a module, does it keep publishing as gravitino-spark-common (currently no Scala or Spark suffix, despite being built against 3.3)? Once the version modules consume sources instead, that jar has no internal consumer.

LuciferYang · 2026-08-12T02:38:27Z

friendly ping @diqiu50

diqiu50 · 2026-08-12T03:06:37Z

We could consider supporting Spark 4.1 instead of 4.0. Spark 4.1 is more stable.

LuciferYang · 2026-08-12T04:49:31Z

No problem, I can adjust it to add support for Spark 4.1. Do you have any feedback on the three questions I raised earlier? Thanks

LuciferYang · 2026-08-12T05:49:11Z

However, my company’s business are still running on Spark 4.0.x. Is it feasible to support both Spark 4.0 and 4.1 simultaneously? I can submit multiple patches to achieve compatibility with both versions. @diqiu50

also cc @jerryshao

diqiu50 · 2026-08-12T06:50:02Z

I think we can simplify the layout further.

First, we don't need to keep Spark 3.3 support, so there is no need to introduce a spark33 source set.

Second, I don't think spark-common needs to remain a Gradle module. I would prefer to make it a shared source set that is compiled directly by each supported Spark version. This avoids giving spark-common an implicit Spark/JDK/Scala version and keeps each version fully independent.

Conceptually:

3.4 -> common + spark34
3.5 -> common + spark35
4.0 -> common + spark40

The tests should follow the same model: shared test sources should be compiled separately for each Spark version rather than distributed as a precompiled testArtifacts JAR.

The Glue download task and similar build logic can be moved out separately. I don't think build-time utilities are a strong enough reason to keep spark-common as a standalone module.

I'm also fine with a reasonable amount of code duplication between version-specific implementations. Keeping clear version boundaries, avoiding cross-version dependencies, and making each supported Spark version independently removable are more important than eliminating every duplicated line of code.

LuciferYang · 2026-08-12T06:52:08Z

Thanks for your feedback. @diqiu50

diqiu50 · 2026-08-12T06:57:12Z

Thanks @LuciferYang
We can tolerate some code duplication between spark34, spark35, and spark40, but we should first evaluate how much duplication there actually is. If it is significant, we should consider extracting more truly version-neutral code into common while still avoiding cross-version dependencies.

LuciferYang · 2026-08-12T07:00:50Z

Thanks @LuciferYang We can tolerate some code duplication between spark34, spark35, and spark40, but we should first evaluate how much duplication there actually is. If it is significant, we should consider extracting more truly version-neutral code into common while still avoiding cross-version dependencies.

Alright, let me evaluate it against this rule first.

jerryshao · 2026-08-12T10:06:19Z

Hi @LuciferYang , we also discussed offline. I think we can support Spark 4.0 along with 4.1. But I don't want to maintain too many Spark versions. So I want to drop support for Spark 3.4, WDYT? Also cc @diqiu50

LuciferYang · 2026-08-12T11:04:21Z

Agreed. Spark's versioning policy keeps 3.5.x on extended LTS through November 2027, while 3.3 and 3.4 are past their maintenance windows: the last releases were 3.3.4 and 3.4.4 (October 2024).

This also answers the duplication question @diqiu50 raised. Under the strict one-directory-per-version layout, dropping 3.4 takes duplicated lines from 873 to 285, and the v3.5 to v3.4 module dependency disappears by construction, which is the cross-version edge principle 2 targets. The four *34 converters merge back into their base classes, since TimestampNTZType and UpdateColumnDefaultValue exist on every remaining version.

Final matrix: 3.5, 4.0, 4.1. One note: this stops publishing gravitino-spark-connector-runtime-3.4, same as 3.3.

diqiu50 · 2026-08-13T07:24:21Z

I suggest we start by supporting two versions. That will give us a clearer picture of the feasibility of this approach and the maintenance cost involved.

LuciferYang · 2026-08-13T07:32:23Z

I suggest we start by supporting two versions. That will give us a clearer picture of the feasibility of this approach and the maintenance cost involved.

Got it, I'll combine my local PR1 and PR2 into a single submission here, and ping you once the code is ready.

…he 3.x line Spark 3.3 and 3.4 are past their maintenance windows, so 3.5 is the only 3.x line that will see another release. Dropping them lets the version-specific converter and catalog subclasses collapse back into spark-common. Stops publishing gravitino-spark-connector-runtime-3.3 and -3.4.

…ared source set spark-common stops being a Gradle module. Each version module composes its own source set from spark-common/src/main/java, spark-common/src/main/spark<NN> and its own src/main/java, so the shared code is compiled against every supported Spark version rather than once. The test tree works the same way, replacing the testArtifacts jar. Stops publishing gravitino-spark-common.

Adds v4.0/spark and v4.0/spark-runtime, publishing gravitino-spark-4.0_2.13 and gravitino-spark-connector-runtime-4.0_2.13. Spark 4 is Scala 2.13 only and needs JDK 17. Moves SparkIcebergTable and SparkJdbcTable back to the shared tree: both compile unchanged on 4.0, so the spark35 copies were duplication. Only the authorization parser needs a per-version flavor.

…xclusion lists The spark-connector ITs are meant to run only in the dedicated Spark IT workflow; every other workflow excludes them. v4.0 was added without being added to those lists, so its ITs also ran inside the backend IT job, where -PjdbcBackend=mysql makes two suites drop and recreate the same MYSQL_JDBC_BACKEND database out from under each other. Excluded alongside 3.5 in backend IT and in the two Trino workflows. Also raises the build job cap from 90 to 120 minutes. It has been finishing in 75-84 minutes and :core:test alone has varied from 27 to 45 minutes across runs on identical code, so the margin was already thinner than the runner's own variance.

LuciferYang · 2026-08-14T06:27:59Z

CI passed, Please review it when you have time, and I will revise it if there are any issues. Thanks @diqiu50

diqiu50 · 2026-08-14T08:57:54Z

The Spark version is already fixed by the jar on the classpath, so CatalogNameAdaptor reading SPARK_VERSION and looking the catalog classes up again is a second dispatch on the same information — and it costs us 11 hardcoded class names that nothing checks, plus a misleading provider is not supported yet warning when the jar does not match the cluster. Could each version module declare its own catalog classes and pass them to the driver plugin, instead of the shared code looking them up by version? Then the names are checked at compile time, CatalogNameAdaptor and the version package go away, and adding a Spark version stops touching shared code.

…are its own catalogs CatalogNameAdaptor read SPARK_VERSION at runtime and looked catalog classes up by string, a second dispatch on information the jar on the classpath already fixes. It cost 11 hardcoded class names that nothing checked and a misleading "provider is not supported yet" warning when the jar did not match the cluster. Each version module now declares a SparkCatalogs class at the same FQN, holding compile-time class references, so the driver plugin reads the table from whichever jar is present. Provider to catalog-kind mapping is the same on every version, so that stays shared as SparkCatalogKind. Adding a Spark version no longer touches shared code.

LuciferYang · 2026-08-17T02:33:52Z

Done in e321b66. Agreed it was a second dispatch, and the version package is gone.

Each version module now declares its own SparkCatalogs at the same FQN with compile-time class references, and the driver plugin reads the table from whichever jar is on the classpath. What stays shared is provider to catalog kind, as a SparkCatalogKind enum: that mapping is the same on every version, and it is where the jdbc-* prefix rule lives. Adding a Spark version no longer touches shared code, and the misleading warning goes with the version lookup, since a jar can now only name catalogs it contains.

One entry cannot be a class reference: Paimon publishes no paimon-spark-3.5_2.13, so the Scala 2.13 build compiles that package out and a .class reference breaks it. It stays a string in v3.5's own SparkCatalogs, added only when the class is present. v4.0 has no Paimon entry.

TestCatalogNameAdaptor is replaced by TestSparkCatalogKind for the provider mapping, plus TestSparkCatalogs, which compiles into both modules and asserts what must hold on every version rather than a fixed list of names: each declared class resolves and is a BaseCatalog, and the five kinds every build ships are present. The names are the compiler's job now.

Verified: unit tests at 123 cases on 3.5 Scala 2.12, 120 on 2.13, 120 on 4.0; SparkHiveCatalogIT40 51 cases and SparkPaimonCatalogFilesystemBackendIT35 36 as end-to-end registration checks; both runtime jars rebuilt with no connector/version/ entries. No failures.

LuciferYang · 2026-08-18T05:05:59Z

friendly ping @diqiu50

Copilot

Pull request overview

Adds Spark 4.0 connector support while narrowing the supported matrix to Spark 3.5 and 4.0.

Changes:

Compiles shared connector sources and tests independently for each Spark version.
Adds Spark 4.0 catalogs, runtime packaging, compatibility adaptations, and CI coverage.
Removes Spark 3.3/3.4 modules and consolidates their remaining behavior into shared code.

Reviewed changes

Copilot reviewed 102 out of 109 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`spark-connector/v4.0/spark/src/test/resources/log4j2.properties`	Adds Spark 4 test logging.
`spark-connector/v4.0/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/jdbc/SparkJdbcPostgreSqlCatalogIT40.java`	Adds Spark 4 PostgreSQL IT.
`spark-connector/v4.0/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/jdbc/SparkJdbcMysqlCatalogIT40.java`	Adds Spark 4 MySQL IT.
`spark-connector/v4.0/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/iceberg/SparkIcebergCatalogHiveBackendIT40.java`	Adds Spark 4 Iceberg IT adaptations.
`spark-connector/v4.0/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/hive/SparkHiveCatalogIT40.java`	Adds Spark 4 Hive IT.
`spark-connector/v4.0/spark/src/main/java/org/apache/gravitino/spark/connector/jdbc/postgresql/GravitinoPostgreSqlCatalogSpark40.java`	Adds Spark 4 PostgreSQL catalog.
`spark-connector/v4.0/spark/src/main/java/org/apache/gravitino/spark/connector/jdbc/GravitinoJdbcCatalogSpark40.java`	Adds Spark 4 JDBC catalog.
`spark-connector/v4.0/spark/src/main/java/org/apache/gravitino/spark/connector/iceberg/GravitinoIcebergCatalogSpark40.java`	Implements Spark 4 Iceberg procedures.
`spark-connector/v4.0/spark/src/main/java/org/apache/gravitino/spark/connector/hive/GravitinoHiveCatalogSpark40.java`	Adds Spark 4 Hive catalog.
`spark-connector/v4.0/spark/src/main/java/org/apache/gravitino/spark/connector/glue/GravitinoGlueCatalogSpark40.java`	Adds Spark 4 Glue catalog.
`spark-connector/v4.0/spark/src/main/java/org/apache/gravitino/spark/connector/catalog/SparkCatalogs.java`	Declares Spark 4 catalog implementations.
`spark-connector/v4.0/spark/build.gradle.kts`	Configures Spark 4 sources, dependencies, and tests.
`spark-connector/v4.0/spark-runtime/build.gradle.kts`	Packages the Spark 4 runtime artifact.
`spark-connector/v3.5/spark/src/test/java/org/apache/gravitino/spark/connector/version/TestCatalogNameAdaptor.java`	Removes obsolete version-name tests.
`spark-connector/v3.5/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/authorization/SparkAuthorizationIT35.java`	Simplifies Spark 3.5 authorization IT.
`spark-connector/v3.5/spark/src/main/java/org/apache/gravitino/spark/connector/paimon/GravitinoPaimonCatalogSpark35.java`	Uses the consolidated Paimon base.
`spark-connector/v3.5/spark/src/main/java/org/apache/gravitino/spark/connector/jdbc/GravitinoJdbcCatalogSpark35.java`	Uses the consolidated JDBC base.
`spark-connector/v3.5/spark/src/main/java/org/apache/gravitino/spark/connector/iceberg/GravitinoIcebergCatalogSpark35.java`	Moves Spark 3.5 procedure integration here.
`spark-connector/v3.5/spark/src/main/java/org/apache/gravitino/spark/connector/hive/GravitinoHiveCatalogSpark35.java`	Uses the consolidated Hive base.
`spark-connector/v3.5/spark/src/main/java/org/apache/gravitino/spark/connector/glue/GravitinoGlueCatalogSpark35.java`	Uses the consolidated Glue base.
`spark-connector/v3.5/spark/src/main/java/org/apache/gravitino/spark/connector/catalog/SparkCatalogs.java`	Declares Spark 3.5 catalog implementations.
`spark-connector/v3.5/spark/build.gradle.kts`	Compiles shared sources directly for Spark 3.5.
`spark-connector/v3.4/spark/src/test/resources/log4j2.properties`	Removes Spark 3.4 test logging.
`spark-connector/v3.4/spark/src/test/java/org/apache/gravitino/spark/connector/version/TestCatalogNameAdaptor.java`	Removes Spark 3.4 catalog-name tests.
`spark-connector/v3.4/spark/src/test/java/org/apache/gravitino/spark/connector/TestSparkTypeConverter34.java`	Removes superseded converter tests.
`spark-connector/v3.4/spark/src/test/java/org/apache/gravitino/spark/connector/TestSparkTableChangeConverter34.java`	Removes superseded table-change tests.
`spark-connector/v3.4/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/sql/SparkSQLRegressionTest34.java`	Removes Spark 3.4 SQL tests.
`spark-connector/v3.4/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/paimon/SparkPaimonCatalogFilesystemBackendIT34.java`	Removes Spark 3.4 Paimon IT.
`spark-connector/v3.4/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/iceberg/SparkIcebergCatalogRestBackendIT34.java`	Removes Spark 3.4 Iceberg REST IT.
`spark-connector/v3.4/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/iceberg/SparkIcebergCatalogHiveBackendIT34.java`	Removes Spark 3.4 Iceberg Hive IT.
`spark-connector/v3.4/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/glue/SparkAwsGlueCatalogIT34.java`	Removes Spark 3.4 Glue IT.
`spark-connector/v3.4/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/authorization/SparkJwksAuthorizationIT34.java`	Removes Spark 3.4 JWKS IT.
`spark-connector/v3.4/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/authorization/SparkAuthorizationIT34.java`	Removes Spark 3.4 authorization IT.
`spark-connector/v3.4/spark/src/main/java/org/apache/gravitino/spark/connector/SparkTypeConverter34.java`	Removes version-specific type conversion.
`spark-connector/v3.4/spark/src/main/java/org/apache/gravitino/spark/connector/SparkTableChangeConverter34.java`	Removes version-specific table-change conversion.
`spark-connector/v3.4/spark/src/main/java/org/apache/gravitino/spark/connector/paimon/GravitinoPaimonCatalogSpark34.java`	Removes Spark 3.4 Paimon catalog.
`spark-connector/v3.4/spark/src/main/java/org/apache/gravitino/spark/connector/jdbc/SparkJdbcTypeConverter34.java`	Removes Spark 3.4 JDBC converter.
`spark-connector/v3.4/spark/src/main/java/org/apache/gravitino/spark/connector/jdbc/GravitinoJdbcCatalogSpark34.java`	Removes Spark 3.4 JDBC catalog.
`spark-connector/v3.4/spark/src/main/java/org/apache/gravitino/spark/connector/iceberg/GravitinoIcebergCatalogSpark34.java`	Removes Spark 3.4 Iceberg catalog.
`spark-connector/v3.4/spark/src/main/java/org/apache/gravitino/spark/connector/hive/SparkHiveTypeConverter34.java`	Removes Spark 3.4 Hive converter.
`spark-connector/v3.4/spark/src/main/java/org/apache/gravitino/spark/connector/hive/GravitinoHiveCatalogSpark34.java`	Removes Spark 3.4 Hive catalog.
`spark-connector/v3.4/spark/src/main/java/org/apache/gravitino/spark/connector/glue/GravitinoGlueCatalogSpark34.java`	Removes Spark 3.4 Glue catalog.
`spark-connector/v3.4/build.gradle.kts`	Removes Spark 3.4 grouping configuration.
`spark-connector/v3.3/spark/src/test/java/org/apache/gravitino/spark/connector/version/TestCatalogNameAdaptor.java`	Removes Spark 3.3 catalog-name tests.
`spark-connector/v3.3/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/sql/SparkSQLRegressionTest33.java`	Removes Spark 3.3 SQL tests.
`spark-connector/v3.3/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/paimon/SparkPaimonCatalogFilesystemBackendIT33.java`	Removes Spark 3.3 Paimon IT.
`spark-connector/v3.3/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/jdbc/SparkJdbcPostgreSqlCatalogIT33.java`	Removes Spark 3.3 PostgreSQL IT.
`spark-connector/v3.3/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/jdbc/SparkJdbcMysqlCatalogIT33.java`	Removes Spark 3.3 MySQL IT.
`spark-connector/v3.3/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/iceberg/SparkIcebergCatalogRestBackendIT33.java`	Removes Spark 3.3 Iceberg REST IT.
`spark-connector/v3.3/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/hive/SparkHiveCatalogIT33.java`	Removes Spark 3.3 Hive IT.
`spark-connector/v3.3/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/glue/SparkAwsGlueCatalogIT33.java`	Removes Spark 3.3 Glue IT.
`spark-connector/v3.3/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/authorization/SparkJwksAuthorizationIT33.java`	Removes Spark 3.3 JWKS IT.
`spark-connector/v3.3/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/authorization/SparkAuthorizationIT33.java`	Removes Spark 3.3 authorization IT.
`spark-connector/v3.3/spark/src/main/java/org/apache/gravitino/spark/connector/paimon/GravitinoPaimonCatalogSpark33.java`	Removes Spark 3.3 Paimon catalog.
`spark-connector/v3.3/spark/src/main/java/org/apache/gravitino/spark/connector/jdbc/postgresql/GravitinoPostgreSqlCatalogSpark33.java`	Removes Spark 3.3 PostgreSQL catalog.
`spark-connector/v3.3/spark/src/main/java/org/apache/gravitino/spark/connector/iceberg/GravitinoIcebergCatalogSpark33.java`	Removes Spark 3.3 Iceberg catalog.
`spark-connector/v3.3/spark/build.gradle.kts`	Removes Spark 3.3 connector build.
`spark-connector/v3.3/spark-runtime/build.gradle.kts`	Removes Spark 3.3 runtime build.
`spark-connector/v3.3/build.gradle.kts`	Removes Spark 3.3 grouping configuration.
`spark-connector/spark-common/src/test/java/org/apache/gravitino/spark/connector/TestSparkTypeConverter.java`	Tests shared timestamp-NTZ conversion.
`spark-connector/spark-common/src/test/java/org/apache/gravitino/spark/connector/plugin/TestGravitinoDriverPlugin.java`	Tests conditional Paimon extension registration.
`spark-connector/spark-common/src/test/java/org/apache/gravitino/spark/connector/integration/test/util/SparkUtilIT.java`	Updates cross-version test guidance.
`spark-connector/spark-common/src/test/java/org/apache/gravitino/spark/connector/integration/test/util/SparkTableInfo.java`	Suppresses cross-version schema deprecation.
`spark-connector/spark-common/src/test/java/org/apache/gravitino/spark/connector/integration/test/sql/SparkSQLRegressionTest.java`	Updates regression-test command.
`spark-connector/spark-common/src/test/java/org/apache/gravitino/spark/connector/integration/test/jdbc/SparkJdbcPostgreSqlCatalogIT.java`	Makes the shared PostgreSQL IT abstract.
`spark-connector/spark-common/src/test/java/org/apache/gravitino/spark/connector/integration/test/iceberg/SparkIcebergCatalogIT.java`	Uses Spark-compatible literal construction.
`spark-connector/spark-common/src/test/java/org/apache/gravitino/spark/connector/integration/test/hive/SparkHiveCatalogIT.java`	Tests Spark's plural partition exception.
`spark-connector/spark-common/src/test/java/org/apache/gravitino/spark/connector/integration/test/authorization/SparkAuthorizationIT.java`	Consolidates write-privilege assertions.
`spark-connector/spark-common/src/test/java/org/apache/gravitino/spark/connector/hive/TestSparkHiveTypeConverter.java`	Expands shared Hive conversion tests.
`spark-connector/spark-common/src/test/java/org/apache/gravitino/spark/connector/catalog/TestTransformTableChange.java`	Tests shared default-value conversion.
`spark-connector/spark-common/src/test/java/org/apache/gravitino/spark/connector/catalog/TestSparkCatalogs.java`	Validates per-version catalog declarations.
`spark-connector/spark-common/src/test/java/org/apache/gravitino/spark/connector/catalog/TestSparkCatalogKind.java`	Tests provider-to-catalog mapping.
`spark-connector/spark-common/src/test/java/org/apache/gravitino/spark/connector/authorization/TestRequiredPrivilegesSparkResolution.java`	Handles cross-version API deprecation.
`spark-connector/spark-common/src/main/spark40/org/apache/gravitino/spark/connector/authorization/GravitinoAuthorizationSparkSessionExtensions.java`	Adds the Spark 4 authorization parser flavor.
`spark-connector/spark-common/src/main/spark35/org/apache/gravitino/spark/connector/paimon/SparkPaimonTable.java`	Restricts Paimon table code to Spark 3.5.
`spark-connector/spark-common/src/main/spark35/org/apache/gravitino/spark/connector/paimon/PaimonPropertiesConverter.java`	Restricts Paimon property conversion to Spark 3.5.
`spark-connector/spark-common/src/main/spark35/org/apache/gravitino/spark/connector/paimon/PaimonPropertiesConstants.java`	Restricts Paimon constants to Spark 3.5.
`spark-connector/spark-common/src/main/spark35/org/apache/gravitino/spark/connector/paimon/GravitinoPaimonCatalog.java`	Restricts the Paimon catalog to Spark 3.5.
`spark-connector/spark-common/src/main/spark35/org/apache/gravitino/spark/connector/authorization/GravitinoAuthorizationSparkSessionExtensions.java`	Adds the Spark 3.5 authorization parser flavor.
`spark-connector/spark-common/src/main/java/org/apache/gravitino/spark/connector/version/CatalogNameAdaptor.java`	Removes runtime Spark-version catalog lookup.
`spark-connector/spark-common/src/main/java/org/apache/gravitino/spark/connector/utils/SparkPartitionUtils.java`	Preserves char/varchar types on Spark 4.
`spark-connector/spark-common/src/main/java/org/apache/gravitino/spark/connector/utils/HiveGravitinoOperationOperator.java`	Adopts the plural Spark partition exception.
`spark-connector/spark-common/src/main/java/org/apache/gravitino/spark/connector/SparkTypeConverter.java`	Consolidates NTZ and char/varchar conversion.
`spark-connector/spark-common/src/main/java/org/apache/gravitino/spark/connector/SparkTableChangeConverter.java`	Consolidates column-default conversion.
`spark-connector/spark-common/src/main/java/org/apache/gravitino/spark/connector/plugin/GravitinoDriverPlugin.java`	Uses per-module catalog declarations.
`spark-connector/spark-common/src/main/java/org/apache/gravitino/spark/connector/jdbc/SparkJdbcTypeConverter.java`	Consolidates JDBC timestamp handling.
`spark-connector/spark-common/src/main/java/org/apache/gravitino/spark/connector/iceberg/GravitinoIcebergCatalog.java`	Moves procedure APIs to version modules.
`spark-connector/spark-common/src/main/java/org/apache/gravitino/spark/connector/hive/SparkHiveTypeConverter.java`	Consolidates Hive timestamp conversion.
`spark-connector/spark-common/src/main/java/org/apache/gravitino/spark/connector/hive/SparkHiveTable.java`	Updates partition exception signature.
`spark-connector/spark-common/src/main/java/org/apache/gravitino/spark/connector/catalog/SparkCatalogKind.java`	Introduces shared provider classification.
`spark-connector/spark-common/src/main/java/org/apache/gravitino/spark/connector/catalog/BaseCatalog.java`	Centralizes write-privilege table loading.
`spark-connector/spark-common/src/main/java/org/apache/gravitino/spark/connector/authorization/AuthorizationTable.java`	Handles schema API deprecation.
`spark-connector/spark-common/build.gradle.kts`	Removes the shared Gradle module.
`spark-connector/build.gradle.kts`	Owns shared formatting and Glue test assets.
`settings.gradle.kts`	Registers only Spark 3.5 and 4.0 modules.
`gradle/libs.versions.toml`	Adds Spark 4 and Iceberg 4.0 versions.
`docs/spark-connector/spark-integration-test.md`	Updates supported test commands.
`docs/spark-connector/spark-connector.md`	Documents the new support matrix.
`docs/spark-connector/spark-catalog-paimon.md`	Documents Spark 4's Paimon limitation.
`docs/spark-connector/spark-catalog-iceberg.md`	Adds Spark 4 Iceberg artifacts.
`docs/spark-connector/spark-catalog-glue.md`	Narrows verified Glue support.
`docs/how-to-build.md`	Updates connector build instructions.
`build.gradle.kts`	Assigns Spark 4 modules a JDK 17 target.
`.github/workflows/trino-multi-version-test.yml`	Updates excluded Spark modules.
`.github/workflows/trino-integration-test.yml`	Updates excluded Spark modules.
`.github/workflows/spark-integration-test-action.yml`	Runs Spark 3.5 and 4.0 integration tests.
`.github/workflows/cron-integration-test.yml`	Uploads Spark 4 integration logs.
`.github/workflows/build.yml`	Updates Scala builds and timeout.
`.github/workflows/backend-integration-test-action.yml`	Updates excluded Spark test modules.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+    } catch (ClassNotFoundException e) {
+      return false;
+    }


+      throw new RuntimeException("Failed to load Iceberg Procedure " + identifier, e);
+    }
+
+    throw new RuntimeException("Procedure does not exist: " + identifier);


 ## Usage

-1. [Build](../how-to-build.md) or download the package ([gravitino-spark-connector-runtime-3.3](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-spark-connector-runtime-3.3), [gravitino-spark-connector-runtime-3.4](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-spark-connector-runtime-3.4), [gravitino-spark-connector-runtime-3.5](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-spark-connector-runtime-3.5)), and place it to the classpath of Spark.
+1. [Build](../how-to-build.md) or download the package matching your Spark minor version ([gravitino-spark-connector-runtime-3.5](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-spark-connector-runtime-3.5), [gravitino-spark-connector-runtime-4.0](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-spark-connector-runtime-4.0)), and place it to the classpath of Spark.


+  private static final Logger LOG = LoggerFactory.getLogger(HiveGravitinoOperationOperator.class);
+
  private org.apache.gravitino.rel.Table gravitinoTable;
  private static final String PARTITION_NAME_DELIMITER = "/";
  private static final String PARTITION_VALUE_DELIMITER = "=";


diqiu50 · 2026-08-19T06:51:01Z

An alternative that keeps the SparkCatalogKind split: move GravitinoSparkPlugin into the version modules and have it build a bindings object to pass to GravitinoDriverPlugin. The FQN stays the same, so spark.plugins is unchanged for users; the bindings are keyed by SparkCatalogKind and built from class literals, with Paimon as a string for the reason it already is. The dependency then runs version module → shared code — ordinary Java, with no import that resolves only after source-set composition.

Three things it buys:

The authorization extension becomes a binding too. It is dispatched by flavor file today only because the shared plugin names the class directly; binding it leaves the spark35/spark40 same-name mechanism with Paimon as its only remaining user.
A missing binding surfaces earlier. Preconditions in the builder make "a version module forgot the JDBC catalog" fail when the plugin is instantiated, rather than the first time that provider is used.
The dispatch becomes testable. A test can construct its own bindings and assert what lands in spark.sql.catalog.*, independently of what the current module ships.

Cost is a ~15-line GravitinoSparkPlugin per version module and one constructor argument on GravitinoDriverPlugin.

diqiu50 · 2026-08-19T07:39:44Z

- * under the License.
- */
-import com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar
-


Two suggestions on the build layout

Keep spark-common as a Gradle module. No need to publish it or depend on it as a jar — the srcDirs composition can stay as is — but as a module the shared tree still compiles on its own, the way trino-connector:trino-connector does. It also gives Spotless and Javadoc a natural home instead of the special-casing in spark-connector/build.gradle.kts.

Factor the common build config into a reusable plugin. srcDirs carries sources, not dependencies, so the module alone does not remove the duplication: the dependencies {} blocks in v3.5/spark and v4.0/spark are ~140 lines each and differ by about a dozen. The common part could move into a convention plugin both apply.

diqiu50 · 2026-08-19T07:41:51Z

+ * that Spark 4 added the abstract {@code parseRoutineParam} method to {@code ParserInterface}.
+ */
+public class GravitinoAuthorizationSparkSessionExtensions
+    implements Function1<SparkSessionExtensions, Void> {


I suggest extracting the reusable parts into a base class and placing it in the shared module.

diqiu50 · 2026-08-19T07:47:59Z

+        if (!dest.exists()) {
+          logger.lifecycle("Downloading $jarName ...")
+          URI(downloadUrl).toURL().openStream().use { input ->
+            dest.outputStream().use { output -> input.copyTo(output) }


Move these to spark common is better

diqiu50 · 2026-08-19T07:49:19Z

+    ":spark-connector:spark-4.0",
+    ":spark-connector:spark-runtime-4.0"
+  )
+


I suggest configuring it in the specific module rather than at the top level. Please refer to the Trino implementation.

diqiu50 · 2026-08-19T08:00:14Z

+    if (!SparkCatalogs.classNames().containsKey(SparkCatalogKind.LAKEHOUSE_PAIMON)) {
+      LOG.warn(
+          "Skip registering Paimon session extensions because {} is not supported yet.",
+          PAIMON_PROVIDER);


Why don't we throw an exception here

…ark 4.0 support - SparkCatalogs on 3.5 caught only ClassNotFoundException when probing the Paimon catalog. Class.forName links the class, so a deployment without the Paimon runtime, which is the documented default, got NoClassDefFoundError while initializing the table, breaking Hive and JDBC sessions too. Reproduced on a Paimon-free classpath; now catches LinkageError as well. - Documented the Spark 4 exception contract in GravitinoIcebergCatalogSpark40: Spark 4 ships no NoSuchProcedureException and wraps non-SparkThrowable in FAILED_TO_LOAD_ROUTINE, which is why Iceberg's own BaseCatalog throws a plain RuntimeException there. - Fixed the runtime artifact links in spark-connector.md, which omitted the Scala suffix the artifacts are published with. - Moved the two partition delimiter constants above the instance field in HiveGravitinoOperationOperator, per the repository's member order.

…itinoSparkPlugin Each version module now owns GravitinoSparkPlugin, which builds a SparkBindings and passes it to the shared GravitinoDriverPlugin. The FQN is unchanged, so spark.plugins is the same for users. The dependency runs version module to shared code, ordinary Java with no import that resolves only after source-set composition. Three things follow. The authorization extension becomes a binding, so it moves out of the flavor directories into each version module and spark40 goes away entirely, leaving Paimon as spark35's only remaining user. A version module that forgets a catalog now fails a Preconditions check when the plugin is instantiated, rather than the first time that provider is used. And the dispatch is testable: TestGravitinoDriverPlugin supplies its own bindings and asserts what lands in spark.sql.catalog.*, independently of what a module ships. Replaces the per-version SparkCatalogs and TestSparkCatalogs from e321b66.

…he dispatch with tests A cr-fix pass over the review round that introduced SparkBindings found the builder accepted values that break at use time, and the catalog dispatch it feeds had no unit coverage at all. SparkBindings.Builder now rejects a blank class name and a kind bound twice, and the same for the authorization extension. A blank one there was the worst case: Spark's own parser filters blank entries out of spark.sql.extensions, so the authorization extension would be dropped and every session would run unauthorized, with no error and no log line. Both Class overloads narrowed to Class<? extends TableCatalog> and Class<? extends Function1<SparkSessionExtensions, Void>>, so "the compiler checks it" is now true. registerGravitinoCatalogs resolves the kind once instead of comparing raw provider literals, keeping the two opt-in gates in step with SparkCatalogKind rather than duplicating the mapping. The conf reads move into registerOptInExtensions so the keys are inside the tested seam. Tests: the provider-to-class dispatch, what registerCatalog writes under spark.sql.catalog.<name>, the four opt-in flag combinations over both catalogs and session extensions, merging into a user's own spark.sql.extensions, and the binding rejections. Each version module gains a test that builds its own GravitinoSparkPlugin, so a forgotten catalog fails a unit test rather than SparkSession startup; v3.5 also pins the Paimon catalog name, which is a string because the Scala 2.13 build compiles that package out. 3.5 at 136 cases under Scala 2.12 and green under 2.13, 4.0 at 132. No failures.

LuciferYang · 2026-08-20T05:38:34Z

To @diqiu50

Done in b2a7cd1, with follow-up validation and tests in 65e24fa. All three things you named came out as you described.

GravitinoSparkPlugin now lives in each version module and builds a SparkBindings for the shared GravitinoDriverPlugin. FQN unchanged, so spark.plugins is the same for users, and the dependency runs version module to shared code with no import that resolves only after source-set composition.

The authorization extension became a binding, which let both flavor copies move into their own version module. spark40 is now empty and gone; spark35 is down to the Paimon package, its only remaining user.

A missing binding fails in the builder, and so does a duplicate or a blank one. SparkBindings.build() requires every kind except Paimon, which some builds legitimately omit. The blank check matters most for the authorization extension: Spark's own parser filters blank entries out of spark.sql.extensions, so a blank binding there would drop the extension and every session would run unauthorized, with no error and no log line. Both Class overloads are bounded now, Class<? extends TableCatalog> and Class<? extends Function1<SparkSessionExtensions, Void>>, so the compile-time check is real rather than just "the class exists".

The dispatch is testable, and tested. TestGravitinoDriverPlugin builds bindings out of org.example.* names, so it asserts what the plugin does with what it is given rather than restating what a module ships: which class each provider resolves to, what registerCatalog writes under spark.sql.catalog.<name>, that a kind with no binding registers nothing, and that the Iceberg and Paimon opt-in flags gate both catalogs and session extensions. Each version module also has a test that builds its own plugin, so a forgotten catalog fails there rather than at SparkSession startup.

Paimon stays a string in v3.5's plugin, resolved by presence, for the reason it already was — and v3.5 has a test that pins that string against the class it ships, since the compiler cannot.

Resolve the import block of TestGravitinoDriverPlugin: keep both sides' imports, drop GravitinoAuthorizationSparkSessionExtensions since this branch's tests bind org.example.* names instead.

LuciferYang · 2026-08-24T06:16:21Z

Merge upstream into this branch and resolved the conflicts.

diqiu50

Test coverage on shared logic is solid, but the new Spark 4.0-specific code (authorization, Iceberg REST backend, procedure loading) is largely untested.

diqiu50 · 2026-09-02T07:49:00Z

+          (TableChange.UpdateColumnDefaultValue) change;
+      return org.apache.gravitino.rel.TableChange.updateColumnDefaultValue(
+          updateColumnDefaultValue.fieldNames(),
+          Literals.stringLiteral(updateColumnDefaultValue.newDefaultValue()));


Is there any issue with using String for all of these fields?

diqiu50 · 2026-09-02T07:51:48Z

-              if ("lakehouse-paimon".equals(provider.toLowerCase(Locale.ROOT))
-                  && !enablePaimonSupport) {
+              if (SparkCatalogKind.LAKEHOUSE_PAIMON.equals(kind) && !enablePaimonSupport) {
                return;


Duplicated check SparkCatalogKind.

diqiu50 · 2026-09-02T07:57:33Z

+      return true;
+    } catch (ClassNotFoundException | LinkageError e) {
+      return false;
+    }


We shouldn't swallow the exception.

diqiu50 · 2026-09-02T08:10:32Z

+/**
+ * Registers Gravitino authorization checks with a Spark session.
+ *
+ * <p>This is the Spark 4 flavor. It matches the Spark 3.5 one under {@code src/main/spark35} except


Resolve build.yml: upstream raised the build job cap to 120 minutes on its own in apache#12583, so drop the comment this branch added to justify the same bump. The branch no longer touches timeout-minutes.

… the 4.0 ITs Resolve the provider to kind mapping once on the catalog register path, log why the Paimon presence probe failed, fix two stale javadoc claims, and add the Spark 4.0 authorization and Iceberg REST backend IT subclasses.

LuciferYang · 2026-09-02T15:06:58Z

@diqiu50 Done in 270a2b0, with main merged in.

Three of the four inline comments were right and are fixed. registerGravitinoCatalogs now maps the provider to a kind once, and every decision below it is made from the kind, so registerCatalog no longer carries a provider that could disagree with it. isPresent no longer swallows: ClassNotFoundException and LinkageError are caught separately and each logs its reason with the throwable. Both log at debug, since both are the normal state of a 3.5 deployment that does not add the Paimon runtime, and the warning a user needs already fires later, when a config asks for Paimon and the build has none. If you would rather have warn on the LinkageError branch, I will change it. The Spark 4 authorization extension's javadoc pointed at src/main/spark35, which stopped being where the 3.5 copy lives once the plugin moved into the version modules.

SparkTableChangeConverter I have left alone, and would like your call on it. UpdateColumnDefaultValue.newDefaultValue() returns a String on 3.5 and 4.0 alike, so a string literal is all the connector has at that point, and the code is not new here: it is SparkTableChangeConverter34 from main, folded into the shared converter unchanged by the first commit. There is a real cost downstream. A string literal is not a NumericType, so the JDBC layer quotes it and an int column's default goes out as DEFAULT '42'; MySQL and PostgreSQL accept that, but the type is lost in Gravitino's metadata. Fixing it needs the column type at a seam that only receives field names and the default text, so it would have to load the table, which changes behaviour on 3.5 as much as on 4.0. I would rather do that as a separate issue than widen this PR.

On coverage: the Iceberg REST backend was a real gap and now has a 4.0 subclass, along with the two authorization ITs. All three are picked up without a workflow change, since the Spark IT job selects by package. The other two you named are already covered on 4.0, which the diff does not make obvious. TestRequiredPrivilegesSparkResolution sits in the shared test tree, so it compiles into the 4.0 module and starts a real Spark 4 session with the extension configured; deleting injectPostHocResolutionRule from the 4.0 copy turns that module red while 3.5 stays green. Procedure loading runs through testIcebergCallOperations in the shared Iceberg suite, which SparkIcebergCatalogHiveBackendIT40 inherits, and building the procedure against the internal Spark catalog instead of this fails it.

Resolve three things from apache#12709, which added Iceberg REST routing with a per-version IT: drop the 3.3 and 3.4 subclasses, whose modules this branch removed and which git tried to relocate into v4.0; add the 4.0 subclass with the metadata-column override its two siblings use; and take upstream's 120-minute cap on the Spark IT job.

leosanqing · 2026-09-04T02:23:03Z

THX, I also need Spark4.x version. I just cherry-pick this PR for compatible test. HOPE for merging soon.

diqiu50 · 2026-09-07T12:49:18Z

@diqiu50 Done in 270a2b0, with main merged in.

Three of the four inline comments were right and are fixed. registerGravitinoCatalogs now maps the provider to a kind once, and every decision below it is made from the kind, so registerCatalog no longer carries a provider that could disagree with it. isPresent no longer swallows: ClassNotFoundException and LinkageError are caught separately and each logs its reason with the throwable. Both log at debug, since both are the normal state of a 3.5 deployment that does not add the Paimon runtime, and the warning a user needs already fires later, when a config asks for Paimon and the build has none. If you would rather have warn on the LinkageError branch, I will change it. The Spark 4 authorization extension's javadoc pointed at src/main/spark35, which stopped being where the 3.5 copy lives once the plugin moved into the version modules.

SparkTableChangeConverter I have left alone, and would like your call on it. UpdateColumnDefaultValue.newDefaultValue() returns a String on 3.5 and 4.0 alike, so a string literal is all the connector has at that point, and the code is not new here: it is SparkTableChangeConverter34 from main, folded into the shared converter unchanged by the first commit. There is a real cost downstream. A string literal is not a NumericType, so the JDBC layer quotes it and an int column's default goes out as DEFAULT '42'; MySQL and PostgreSQL accept that, but the type is lost in Gravitino's metadata. Fixing it needs the column type at a seam that only receives field names and the default text, so it would have to load the table, which changes behaviour on 3.5 as much as on 4.0. I would rather do that as a separate issue than widen this PR.

On coverage: the Iceberg REST backend was a real gap and now has a 4.0 subclass, along with the two authorization ITs. All three are picked up without a workflow change, since the Spark IT job selects by package. The other two you named are already covered on 4.0, which the diff does not make obvious. TestRequiredPrivilegesSparkResolution sits in the shared test tree, so it compiles into the 4.0 module and starts a real Spark 4 session with the extension configured; deleting injectPostHocResolutionRule from the 4.0 copy turns that module red while 3.5 stays green. Procedure loading runs through testIcebergCallOperations in the shared Iceberg suite, which SparkIcebergCatalogHiveBackendIT40 inherits, and building the procedure against the internal Spark catalog instead of this fails it.

OK，Let's ignore the default value issue in this PR

diqiu50

LGTM

LuciferYang · 2026-09-08T02:53:08Z

Thank you @diqiu50 and @jerryshao
I’ll open the PR for 4.1 support shortly.

LuciferYang force-pushed the spark4-support branch from b258afc to 5923cfc Compare August 11, 2026 04:12

roryqi requested a review from diqiu50 August 11, 2026 05:47

LuciferYang force-pushed the spark4-support branch from db1654b to 1a55481 Compare August 13, 2026 03:38

LuciferYang changed the title ~~[#8771] feat(spark-connector): Support Apache Spark 4.0~~ [#8771] refactor(spark-connector): Support Spark 3.5 only and compile spark-common per version Aug 13, 2026

diqiu50 assigned LuciferYang and diqiu50 Aug 13, 2026

LuciferYang marked this pull request as draft August 13, 2026 07:12

LuciferYang added 2 commits August 13, 2026 17:56

LuciferYang force-pushed the spark4-support branch from 1a55481 to b5f71dd Compare August 13, 2026 13:52

LuciferYang changed the title ~~[#8771] refactor(spark-connector): Support Spark 3.5 only and compile spark-common per version~~ [#8771] feat(spark-connector): Add Spark 4.0 support, drop 3.3 and 3.4 Aug 14, 2026

LuciferYang marked this pull request as ready for review August 14, 2026 06:26

diqiu50 requested a balanced review from Copilot August 19, 2026 04:09

Copilot started reviewing on behalf of diqiu50 August 19, 2026 04:09 View session

Copilot AI reviewed Aug 19, 2026

View reviewed changes

diqiu50 reviewed Aug 19, 2026

View reviewed changes

LuciferYang added 3 commits August 19, 2026 16:27

Merge upstream/main into spark4-support

90ac253

Resolve the import block of TestGravitinoDriverPlugin: keep both sides' imports, drop GravitinoAuthorizationSparkSessionExtensions since this branch's tests bind org.example.* names instead.

diqiu50 reviewed Sep 2, 2026

View reviewed changes

Merge upstream/main into spark4-support

1ec084c

Resolve build.yml: upstream raised the build job cap to 120 minutes on its own in apache#12583, so drop the comment this branch added to justify the same bump. The branch no longer touches timeout-minutes.

jiangxt2 mentioned this pull request Sep 2, 2026

[FEATURE] Add governed Apache Doris batch read and write support to the Gravitino Spark connector #12846

Open

diqiu50 approved these changes Sep 7, 2026

View reviewed changes

diqiu50 merged commit bcf36c7 into apache:main Sep 8, 2026
39 checks passed

LuciferYang mentioned this pull request Sep 9, 2026

[Improvement] Support Apache Spark 4.1 in the Spark connector #13018

Open

Uh oh!

Conversation

LuciferYang commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

github-actions Bot commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage Report

Uh oh!

LuciferYang commented Aug 11, 2026

Uh oh!

diqiu50 commented Aug 11, 2026

Uh oh!

LuciferYang commented Aug 11, 2026

Uh oh!

LuciferYang commented Aug 11, 2026

Uh oh!

LuciferYang commented Aug 12, 2026

Uh oh!

diqiu50 commented Aug 12, 2026

Uh oh!

LuciferYang commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LuciferYang commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

diqiu50 commented Aug 12, 2026

Uh oh!

LuciferYang commented Aug 12, 2026

Uh oh!

diqiu50 commented Aug 12, 2026

Uh oh!

LuciferYang commented Aug 12, 2026

Uh oh!

jerryshao commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LuciferYang commented Aug 12, 2026

Uh oh!

diqiu50 commented Aug 13, 2026

Uh oh!

LuciferYang commented Aug 13, 2026

Uh oh!

LuciferYang commented Aug 14, 2026

Uh oh!

diqiu50 commented Aug 14, 2026

Uh oh!

LuciferYang commented Aug 17, 2026

Uh oh!

LuciferYang commented Aug 18, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

diqiu50 commented Aug 19, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LuciferYang commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LuciferYang commented Aug 24, 2026

Uh oh!

diqiu50 left a comment

Choose a reason for hiding this comment

LuciferYang commented Aug 11, 2026 •

edited

Loading

github-actions Bot commented Aug 11, 2026 •

edited

Loading

LuciferYang commented Aug 12, 2026 •

edited

Loading

LuciferYang commented Aug 12, 2026 •

edited

Loading

jerryshao commented Aug 12, 2026 •

edited

Loading

LuciferYang commented Aug 20, 2026 •

edited

Loading