[KYUUBI #3441] Change default Spark version to 3.3.1

[yikf]

Why are the changes needed?

Fix #3441;

This pr aims to two points as follow:

• Bump Spark 3.3.0 to Spark 3.3.1 in spark-3.3 profile
• Change default Spark version to 3.3.1

How was this patch tested?

• Add some test cases that check the changes thoroughly including negative and positive cases if possible

• Add screenshots for manual tests if appropriate

• Run test locally before make a pull request

[yikf]

@pan3793 , Tracking from here

[cfmcgrady]

FYI: Spark-3.3.1-rc2 is out.
https://www.mail-archive.com/dev@spark.apache.org/msg29460.html

[yikf]

@cfmcgrady Thanks, will test

[pan3793]

The test failed consistently, could you please take a look?

[yikf]

The test failed consistently, could you please take a look?

will check

[yikf]

It looks like that we currently support spark 3.2 only, We should specify the profile here to prevent some incompatibilities (on k8s w/ client mode)

[pan3793]

it's tricky, what's the specific issue here?

[yikf]

In spark on k8s w/ client mode, task deserialization may fail due to inconsistent Jar packages on the Driver and Executor ends

[pan3793]

oh sorry, i missed we hardcoded the spark image in code, how about changing it to get from ENV then we can override the in test?

[pan3793]

We can get the compile Spark version by

SPARK_VERSION=$("$MVN" help:evaluate -Dexpression=spark.version $@ 2>/dev/null\
    | grep -v "INFO"\
    | grep -v "WARNING"\
    | tail -n 1)

[yikf]

Yes, I had the idea that we could pass the spark version of the current profile through the maven plugin so that the image version would be consistent with the spark version of the current profile, but I wanted to do this later for two reasons:

• This is to address the current todo and seems to be unrelated to the current pr
• The official spark image does not have a sub-version like Spark 3.3.1 https://hub.docker.com/r/apache/spark/tags, the Spark community is currently working on making the spark image the official docker image, maybe we can solve this todo after that?

[pan3793]

make sense, thanks

[pan3793]

LGTM, it's in good shape now, let's wait for the next RC or GA version, thanks @yikf

[yikf]

Test for Spark 3.3.1 (RC4)

[pan3793]

Spark 3.3.1 (RC4) vote passed.
https://www.mail-archive.com/dev@spark.apache.org/msg29565.html

[pan3793]

Thanks @yikf, merging to master! Also thanks @wangyum for releasing Spark 3.3.1

[yikf]

Thanks all.