[SPARK-53958][BUILD] Simplify Jackson deps management by using BOM by pan3793 · Pull Request #52668 · apache/spark

pan3793 · 2025-10-20T14:28:20Z

What changes were proposed in this pull request?

Currently, it's relatively complicated with dependency version management due to the different resolution mechanisms between Maven and SBT.

Take Jackson deps as an example, to ensure all Jackson deps have the same version on the classpath, for both Maven and SBT, we must exclude all transitive Jackson deps and re-declare them manually, otherwise SBT may pick the higher version from transitive deps then breaks the building.

This PR proposes to use BOM to simplify the above process, see technical details at https://github.com/FasterXML/jackson-bom

Maven has built-in support for BOM, SBT also has a plugin https://github.com/heremaps/here-sbt-bom

Why are the changes needed?

Simplify dependency management.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass GHA.

Manually checked the output of build/sbt dependencyTree, all Jackson deps are correctly resolved to the expected version - 2.19.2.

Was this patch authored or co-authored using generative AI tooling?

No.

pan3793 · 2025-10-20T14:38:46Z

 <scalafmt.validateOnly>true</scalafmt.validateOnly>
 <scalafmt.changedOnly>true</scalafmt.changedOnly>
 <fasterxml.jackson.version>2.19.2</fasterxml.jackson.version>
- <fasterxml.jackson.databind.version>2.19.2</fasterxml.jackson.databind.version>


in case that jackson-databind has a standalone patched version, the jackson-bom also has a new release, for example
https://mvnrepository.com/artifact/com.fasterxml.jackson/jackson-bom/2.13.4.20221013

pan3793 · 2025-10-20T14:42:12Z

I hit some Jackson deps issues (#52664 (comment)) in upgrading Avro to 1.12.1, and haven't figured out the root cause, so I am looking into this new simple approach to address the issues.

cc @LuciferYang @sarutak @dongjoon-hyun WDYT?

dongjoon-hyun

Could you revert all elimination of <exclusion> tags from this PR?

I believe this PR should work without touching the existing exclusion rules in order to verify that BOM is actually working correctly, @pan3793 .

pan3793 · 2025-10-21T02:33:19Z

@dongjoon-hyun, either w/ or w/o the elimination of <exclusion> should work, I tend to eliminate them to simplify the pom.xml, but also fine if you think we should keep them.

dongjoon-hyun

+1, LGTM. Thank you for revising this PR, @pan3793 .

Merged to master for Apache Spark 4.1.0-preview3.

sarutak

Late LGTM.

### What changes were proposed in this pull request? Currently, it's relatively complicated with dependency version management due to the different resolution mechanisms between Maven and SBT. Take Jackson deps as an example, to ensure all Jackson deps have the same version on the classpath, for both Maven and SBT, we must exclude all transitive Jackson deps and re-declare them manually, otherwise SBT may pick the higher version from transitive deps then breaks the building. This PR proposes to use BOM to simplify the above process, see technical details at https://github.com/FasterXML/jackson-bom Maven has built-in support for BOM, SBT also has a plugin https://github.com/heremaps/here-sbt-bom ### Why are the changes needed? Simplify dependency management. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GHA. Manually checked the output of `build/sbt dependencyTree`, all Jackson deps are correctly resolved to the expected version - 2.19.2. ### Was this patch authored or co-authored using generative AI tooling? No. Closesapache#52668 from pan3793/SPARK-53958. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

### What changes were proposed in this pull request? This PR aims to upgrade `spark.jackson.version` to 2.21.1 in bench module. ### Why are the changes needed? From 2.21, we need to use BOM. https://cowtowncoder.medium.com/jackson-2-21-released-ce0b6582ae67 To use the latest version like Apache Spark. - apache/spark#52668 (Apache Spark 4.1.0) - apache/spark#53886 (Apache Spark 4.2.0-preview2) - apache/spark#54626 ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: `Gemini 3.1 Pro (High)` on `Antigravity`. Closes#2570 from dongjoon-hyun/ORC-2122. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

### What changes were proposed in this pull request? This PR aims to upgrade `spark.jackson.version` to 2.21.1 in bench module. ### Why are the changes needed? From 2.21, we need to use BOM. https://cowtowncoder.medium.com/jackson-2-21-released-ce0b6582ae67 To use the latest version like Apache Spark. - apache/spark#52668 (Apache Spark 4.1.0) - apache/spark#53886 (Apache Spark 4.2.0-preview2) - apache/spark#54626 ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: `Gemini 3.1 Pro (High)` on `Antigravity`. Closes#2570 from dongjoon-hyun/ORC-2122. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 8167cbc) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

### What changes were proposed in this pull request? This PR uses `io.grpc:grpc-bom` to manage `io.grpc` artifact versions, replacing the per-artifact `<version>${io.grpc.version}</version>` declarations across `sql/connect/common`, `sql/connect/server`, and `udf/worker/grpc`. Mirrors the existing `jackson-bom` (#52668) and `jjwt-bom` (#56155) imports. ### Why are the changes needed? Simplify dependency management. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. No-op on dependency resolution — `io.grpc` resolves identically (`1.76.0`, direct and transitive) before/after, verified with `build/mvn help:effective-pom` and `build/mvn dependency:tree`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) Closes#56741 from gillespeiffer-db/grpc-bom. Authored-by: gillespeiffer-db <263405958+gillespeiffer-db@users.noreply.github.com> Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com>

…using BOM ### What changes were proposed in this pull request? This PR uses `io.fabric8:kubernetes-client-bom` to manage `io.fabric8` artifact versions, replacing the per-artifact `<version>${kubernetes-client.version}</version>` declarations across `resource-managers/kubernetes/core` and `resource-managers/kubernetes/integration-tests` (`kubernetes-client`, `volcano-client`, `volcano-model`). The BOM is mirrored in `project/SparkBuild.scala` (sbt-pom-reader does not process import-scope BOMs). Mirrors the existing `jackson-bom` (#52668), `jjwt-bom` (#56155), and `grpc-bom` (#56741) imports. ### Why are the changes needed? Simplify dependency management. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. No-op on dependency resolution — `io.fabric8` resolves identically (`7.7.0`, direct and transitive) before/after, verified with `build/mvn help:effective-pom` and `build/mvn dependency:tree`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) Closes#56774 from gillespeiffer-db/kubernetes-client-bom. Authored-by: gillespeiffer-db <263405958+gillespeiffer-db@users.noreply.github.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…naged by jackson-bom ### What changes were proposed in this pull request? This PR drops the redundant per-module `<version>${fasterxml.jackson.version}</version>` declarations on the `com.fasterxml.jackson.dataformat` artifacts in `connector/kinesis-asl`, `hadoop-cloud`, and `resource-managers/kubernetes/core`. These artifacts are already managed by the `com.fasterxml.jackson:jackson-bom` import (#52668), so the per-module versions are redundant. Completes the `jackson-bom` adoption, consistent with the `jjwt-bom` (#56155) and `grpc-bom` (#56741) imports. ### Why are the changes needed? Simplify dependency management. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. No-op on dependency resolution — the `com.fasterxml.jackson.dataformat` artifacts resolve identically (`2.22.0`, direct and transitive) before/after, verified with `build/mvn dependency:tree`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) Closes#56773 from gillespeiffer-db/jackson-bom-cleanup. Authored-by: gillespeiffer-db <263405958+gillespeiffer-db@users.noreply.github.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

github-actionsBot added STRUCTURED STREAMING KUBERNETES BUILD CORE DSTREAM labels Oct 20, 2025

pan3793 mentioned this pull request Oct 20, 2025
[SPARK-53954][BUILD] Bump Avro 1.12.1 #52664
Closed

[SPARK-53958][BUILD] Simplify Jackson deps management by using BOM
46c5d4b

pan3793 force-pushed the SPARK-53958 branch from f47c0fe to 46c5d4bCompare October 20, 2025 14:37

pan3793 commented Oct 20, 2025
View reviewed changes

dongjoon-hyun reviewed Oct 20, 2025
View reviewed changes

revert exclusion
d15ff1c

github-actionsBot removed STRUCTURED STREAMING KUBERNETES CORE DSTREAM labels Oct 21, 2025

revert
c6fd772

pan3793 requested a review from dongjoon-hyun October 21, 2025 05:32

dongjoon-hyun approved these changes Oct 21, 2025
View reviewed changes

dongjoon-hyun closed this in 85bf6ecOct 21, 2025

sarutak reviewed Oct 21, 2025
View reviewed changes

dongjoon-hyun mentioned this pull request Mar 4, 2026
ORC-2122: Upgrade spark.jackson.version to 2.21.1 in bench module apache/orc#2570
Closed

gillespeiffer-db mentioned this pull request Jun 24, 2026
[SPARK-57663][BUILD] Simplify gRPC deps management by using BOM #56741
Closed

gillespeiffer-db mentioned this pull request Jun 25, 2026
[SPARK-58259][BUILD] Remove redundant Jackson version declarations managed by jackson-bom #56773
Closed

gillespeiffer-db mentioned this pull request Jun 25, 2026
[SPARK-57693][BUILD] Simplify kubernetes-client deps management by using BOM #56774
Closed

Navigation Menu

Uh oh!

[SPARK-53958][BUILD] Simplify Jackson deps management by using BOM - #52668

[SPARK-53958][BUILD] Simplify Jackson deps management by using BOM#52668
pan3793 wants to merge 3 commits into
apache:masterfrom
pan3793:SPARK-53958

pan3793 commented Oct 20, 2025

Uh oh!

pan3793Oct 20, 2025

Uh oh!

pan3793 commented Oct 20, 2025

Uh oh!

dongjoon-hyun left a comment •
edited
Loading

Uh oh!

pan3793 commented Oct 21, 2025

Uh oh!

dongjoon-hyun left a comment

Uh oh!

sarutak left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

pan3793 commented Oct 20, 2025

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

pan3793Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

pan3793 commented Oct 20, 2025

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pan3793 commented Oct 21, 2025

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

sarutak left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dongjoon-hyun left a comment •
edited
Loading