Build with Spark 3.2 failed

[XorSum]

Describe the bug

comet fails to compile with Spark version 3.2

[ERROR] Failed to execute goal com.diffplug.spotless:spotless-maven-plugin:2.43.0:check (default) on project comet-parent-spark3.2_2.12: Execution default of goal com.diffplug.spotless:spotless-maven-plugin:2.43.0:check failed: Unable to load the mojo 'check' in the plugin 'com.diffplug.spotless:spotless-maven-plugin:2.43.0' due to an API incompatibility: org.codehaus.plexus.component.repository.exception.ComponentLookupException: com/diffplug/spotless/maven/SpotlessCheckMojo has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0

[ERROR] spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala:1069: value offset is not a member of org.apache.spark.sql.DataFrame

Steps to reproduce

PROFILES="-Pspark-3.2" make

Expected behavior

Compilation passes

Additional context

No response

[XorSum]

The first error is easy to fix. Just add property <spotless.version>2.29.0</spotless.version> to profile spark-3.2.

The second error is caused by api DataSet.offset, which is introduced since Spark 3.4. Should we split the code in two parts for both pre and post 3.4 versions?

#85
https://issues.apache.org/jira/browse/SPARK-39159

[viirya]

We skip certain tests which are only for Spark 3.4 internally. But seems we don't do it for Comet now.

[advancedxy]

Maybe we should also run CI with different Spark version profiles too.
However the CI run time is already quite long, we should enable that after reducing the CI run time.

[sunchao]

I think the Spark 3.2 & 3.3 profiles still have quite a few issues right now. I agree that we should add CI pipelines for them at some point, and perhaps make them trigger at post-commit time.

[advancedxy]

I agree that we should add CI pipelines for them at some point, and perhaps make them trigger at post-commit time.

After a second thought, maybe we can just compile(skipTests) against Spark 3.2 and 3.3 for now on CI? The test pipelines could be added later.

[sunchao]

maybe we can just compile(skipTests) against Spark 3.2 and 3.3

Yes I think we can do that. We can also try to enable them and see if they fail.