[SPARK-47706][BUILD] Bump json4s 4.0.7 by pan3793 · Pull Request #45838 · apache/spark

pan3793 · 2024-04-03T07:20:05Z

What changes were proposed in this pull request?

Bump json4s from 3.7.0-M11 to 4.0.7

Why are the changes needed?

4.0.7 is the latest stable version of json4s.

https://mvnrepository.com/artifact/org.json4s/json4s-jackson

Does this PR introduce any user-facing change?

No, all Mima complaints are private API.

How was this patch tested?

Pass GHA.

Was this patch authored or co-authored using generative AI tooling?

No

LuciferYang · 2024-04-03T09:30:38Z

Previously, there were two unsuccessful upgrade attempts:

cc @dongjoon-hyun @sarutak @JoshRosen FYI

LuciferYang · 2024-04-03T09:44:05Z

I get errors when I manually run the mima check. Could you please update the MimaExcludes? @pan3793

[error] spark-sql-api: Failed binary compatibility check against org.apache.spark:spark-sql-api_2.13:3.5.0! Found 1 potential problems (filtered 170)
[error] * method unapplySeq(org.json4s.JsonAST#JValue)scala.Option in object org.apache.spark.sql.types.DataType#JSortedObject's type is different in current version, where it is (org.json4s.JValue)scala.Option instead of (org.json4s.JsonAST#JValue)scala.Option
[error] filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.types.DataType#JSortedObject.unapplySeq")
[error] spark-sql: Failed binary compatibility check against org.apache.spark:spark-sql_2.13:3.5.0! Found 1 potential problems (filtered 861)
[error] * method jsonValue()org.json4s.JsonAST#JValue in class org.apache.spark.sql.expressions.MutableAggregationBuffer has a different result type in current version, where it is org.json4s.JValue rather than org.json4s.JsonAST#JValue
[error] filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.expressions.MutableAggregationBuffer.jsonValue")
[error] spark-mllib: Failed binary compatibility check against org.apache.spark:spark-mllib_2.13:3.5.0! Found 11 potential problems (filtered 477)
[error] * method params()org.json4s.JsonAST#JValue in class org.apache.spark.ml.util.DefaultParamsReader#Metadata has a different result type in current version, where it is org.json4s.JValue rather than org.json4s.JsonAST#JValue
[error] filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.params")
[error] * method defaultParams()org.json4s.JsonAST#JValue in class org.apache.spark.ml.util.DefaultParamsReader#Metadata has a different result type in current version, where it is org.json4s.JValue rather than org.json4s.JsonAST#JValue
[error] filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.defaultParams")
[error] * method metadata()org.json4s.JsonAST#JValue in class org.apache.spark.ml.util.DefaultParamsReader#Metadata has a different result type in current version, where it is org.json4s.JValue rather than org.json4s.JsonAST#JValue
[error] filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.metadata")
[error] * method getParamValue(java.lang.String)org.json4s.JsonAST#JValue in class org.apache.spark.ml.util.DefaultParamsReader#Metadata has a different result type in current version, where it is org.json4s.JValue rather than org.json4s.JsonAST#JValue
[error] filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.getParamValue")
[error] * method copy(java.lang.String,java.lang.String,Long,java.lang.String,org.json4s.JsonAST#JValue,org.json4s.JsonAST#JValue,org.json4s.JsonAST#JValue,java.lang.String)org.apache.spark.ml.util.DefaultParamsReader#Metadata in class org.apache.spark.ml.util.DefaultParamsReader#Metadata's type is different in current version, where it is (java.lang.String,java.lang.String,Long,java.lang.String,org.json4s.JValue,org.json4s.JValue,org.json4s.JValue,java.lang.String)org.apache.spark.ml.util.DefaultParamsReader#Metadata instead of (java.lang.String,java.lang.String,Long,java.lang.String,org.json4s.JsonAST#JValue,org.json4s.JsonAST#JValue,org.json4s.JsonAST#JValue,java.lang.String)org.apache.spark.ml.util.DefaultParamsReader#Metadata
[error] filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy")
[error] * synthetic method copy$default$5()org.json4s.JsonAST#JValue in class org.apache.spark.ml.util.DefaultParamsReader#Metadata has a different result type in current version, where it is org.json4s.JValue rather than org.json4s.JsonAST#JValue
[error] filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$5")
[error] * synthetic method copy$default$6()org.json4s.JsonAST#JValue in class org.apache.spark.ml.util.DefaultParamsReader#Metadata has a different result type in current version, where it is org.json4s.JValue rather than org.json4s.JsonAST#JValue
[error] filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$6")
[error] * synthetic method copy$default$7()org.json4s.JsonAST#JValue in class org.apache.spark.ml.util.DefaultParamsReader#Metadata has a different result type in current version, where it is org.json4s.JValue rather than org.json4s.JsonAST#JValue
[error] filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$7")
[error] * method this(java.lang.String,java.lang.String,Long,java.lang.String,org.json4s.JsonAST#JValue,org.json4s.JsonAST#JValue,org.json4s.JsonAST#JValue,java.lang.String)Unit in class org.apache.spark.ml.util.DefaultParamsReader#Metadata's type is different in current version, where it is (java.lang.String,java.lang.String,Long,java.lang.String,org.json4s.JValue,org.json4s.JValue,org.json4s.JValue,java.lang.String)Unit instead of (java.lang.String,java.lang.String,Long,java.lang.String,org.json4s.JsonAST#JValue,org.json4s.JsonAST#JValue,org.json4s.JsonAST#JValue,java.lang.String)Unit
[error] filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.this")
[error] * method apply(java.lang.String,java.lang.String,Long,java.lang.String,org.json4s.JsonAST#JValue,org.json4s.JsonAST#JValue,org.json4s.JsonAST#JValue,java.lang.String)org.apache.spark.ml.util.DefaultParamsReader#Metadata in object org.apache.spark.ml.util.DefaultParamsReader#Metadata in current version does not have a correspondent with same parameter signature among (java.lang.String,java.lang.String,Long,java.lang.String,org.json4s.JValue,org.json4s.JValue,org.json4s.JValue,java.lang.String)org.apache.spark.ml.util.DefaultParamsReader#Metadata, (java.lang.Object,java.lang.Object,java.lang.Object,java.lang.Object,java.lang.Object,java.lang.Object,java.lang.Object,java.lang.Object)java.lang.Object
[error] filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.apply")
[error] * method readMetadata(org.json4s.JsonAST#JValue)org.apache.spark.mllib.tree.model.TreeEnsembleModel#SaveLoadV1_0#Metadata in object org.apache.spark.mllib.tree.model.TreeEnsembleModel#SaveLoadV1_0's type is different in current version, where it is (org.json4s.JValue)org.apache.spark.mllib.tree.model.TreeEnsembleModel#SaveLoadV1_0#Metadata instead of (org.json4s.JsonAST#JValue)org.apache.spark.mllib.tree.model.TreeEnsembleModel#SaveLoadV1_0#Metadata
[error] filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.mllib.tree.model.TreeEnsembleModel#SaveLoadV1_0.readMetadata")

pan3793 · 2024-04-03T09:51:46Z

@LuciferYang~~json4s classes leaked to public(maybe? at least in the Mima audit list) API, and json4s 4.x changed those classes, this requires the Spark community to evaluate if we should do such an upgrading~~

update: json4s did not leak to public API, they are false-positive results reported by Mima

pan3793 · 2024-04-03T14:57:06Z

@LuciferYangMimaExcludes is updated.

dongjoon-hyun · 2024-04-03T15:08:20Z

Wow.. Everything is inevitable?

This is because the JValue package is changed, it does not mean can not read the model saved by the old Spark
def readMetadata(metadata: JValue): Metadata

this should not be an issue
private[tree] object TreeEnsembleModel extends Logging { object SaveLoadV1_0 { ... } ... }

dongjoon-hyun · 2024-04-03T15:09:52Z

Is this removed or deprecated?

No, it's there. But method JsonMethods#parse signature changed. As this is the test code, I chose the simplest way to change it.

dongjoon-hyun

It looks like inevitable change if we want to use the latest one.

+1 for Apache Spark 4.0.0.

WDYT, cc @HyukjinKwon , @cloud-fan , @gengliangwang ?

gengliangwang · 2024-04-06T23:54:54Z

Since there are many connectors/libraries that depend on Spark, I am +0 on this one.

cloud-fan · 2024-04-08T04:23:14Z

does json4s release notes mention anything about backward compatibility? We have persistent event logs in JSON format that may need to be consumed by new version of Spark history server.

pan3793 · 2024-04-08T05:02:20Z

@cloud-fan seems json4s does not have release note. the API binary incompatible does not affect JSON serialize/deserialize, the case of parsing event log generated by the previous Spark version is already covered by Spark UT.

pan3793 · 2024-04-08T05:11:13Z

I noticed that DefaultParamsReader's scope is private[ml], but Mima still complains, is it expected? cc @LuciferYang
private[ml] class DefaultParamsReader

Could you file a JIRA issue for this, @pan3793 ? It seems that we had better fix that first.

I filed SPARK-47770.

dongjoon-hyun · 2024-04-09T00:23:45Z

To @pan3793 , for MIMA issue, I made the following PR to fix it.

[SPARK-47770][INFRA] Fix GenerateMIMAIgnore.isPackagePrivateModule to return false instead of failing #45938

pan3793 · 2024-04-09T03:49:25Z

it's weird
trait Row { ... private[sql] def jsonValue: JValue ... } abstract class MutableAggregationBuffer extends Row { ... }

pan3793 · 2024-04-09T03:50:37Z

this should not be a real issue
object DataType { private object JSortedObject { ... } ... }

pan3793 · 2024-04-09T03:53:38Z

@dongjoon-hyun @cloud-fan @gengliangwang @LuciferYang I updated MimaExcludes, now it looks good.

dongjoon-hyun · 2024-04-09T04:58:39Z

Thank you for rebasing.

pan3793 · 2024-04-10T02:46:43Z

rebased on master

LuciferYang · 2024-04-10T03:18:25Z

If there are no compatibility issues, this is great. It seems that json4s 4.x supports Scala 3, which further increases the possibility of us upgrading to Scala 3 in the future. If this upgrade is feasible, we can close https://issues.apache.org/jira/browse/SPARK-44172 at the same time.

LuciferYang

+1, LGTM
Thanks @pan3793 and @dongjoon-hyun

dongjoon-hyun · 2024-04-10T04:52:53Z

Could you re-trigger the failed test pipeline, @pan3793 ?

pan3793 · 2024-04-10T05:20:45Z

@LuciferYang @dongjoon-hyun The failed test is known flaky case, I re-triggered CI, thanks for your approval and help with Mima

dongjoon-hyun · 2024-04-10T06:04:12Z

Thank you, @pan3793 and all.
Merged to master for Apache Spark 4.0.0.

github-actionsBot added SQL BUILD labels Apr 3, 2024

pan3793 commented Apr 3, 2024
View reviewed changes

Comment threadsql/api/pom.xml Outdated

pan3793 marked this pull request as draft April 3, 2024 07:36

github-actionsBot added the PROTOBUF label Apr 3, 2024

pan3793 marked this pull request as ready for review April 3, 2024 14:56

dongjoon-hyun reviewed Apr 3, 2024
View reviewed changes

dongjoon-hyun approved these changes Apr 6, 2024
View reviewed changes

pan3793 commented Apr 8, 2024
View reviewed changes

dongjoon-hyun mentioned this pull request Apr 9, 2024
[SPARK-47770][INFRA] Fix GenerateMIMAIgnore.isPackagePrivateModule to return false instead of failing #45938
Closed

pan3793 force-pushed the SPARK-47706 branch from 5cca1ce to 01c01c6Compare April 9, 2024 03:42

pan3793 commented Apr 9, 2024
View reviewed changes

pan3793 added 5 commits April 10, 2024 10:45

[SPARK-47706][BUILD] Bump json4s 4.0.7
0ad6c6a

fix
2b02da9

fix
0765c1a

Update MimaExcludes
a05c8e9

mima
e77054f

pan3793 force-pushed the SPARK-47706 branch from 01c01c6 to e77054fCompare April 10, 2024 02:45

LuciferYang approved these changes Apr 10, 2024
View reviewed changes

dongjoon-hyun approved these changes Apr 10, 2024
View reviewed changes

dongjoon-hyun closed this in 7996b03Apr 10, 2024

Uh oh!

Conversation

pan3793 commented Apr 3, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

Uh oh!

LuciferYang commented Apr 3, 2024

Uh oh!

LuciferYang commented Apr 3, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pan3793 commented Apr 3, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pan3793 commented Apr 3, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

gengliangwang commented Apr 6, 2024

Uh oh!

cloud-fan commented Apr 8, 2024

Uh oh!

pan3793 commented Apr 8, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyunApr 8, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Apr 9, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pan3793 commented Apr 9, 2024

Uh oh!

dongjoon-hyun commented Apr 9, 2024

Uh oh!

pan3793 commented Apr 10, 2024

Uh oh!

LuciferYang commented Apr 10, 2024

Uh oh!

LuciferYang left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Apr 10, 2024

Uh oh!

pan3793 commented Apr 10, 2024

Uh oh!

dongjoon-hyun commented Apr 10, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

pan3793 commented Apr 3, 2024•
edited
Loading

LuciferYang commented Apr 3, 2024•
edited
Loading

pan3793 commented Apr 3, 2024•
edited
Loading

dongjoon-hyunApr 8, 2024•
edited
Loading