[SPARK-42888][BUILD] Upgrade gcs-connector to 2.2.11

[cnauroth]

What changes were proposed in this pull request?

Upgrade the GCS Connector bundled in the Spark distro from version 2.2.7 to 2.2.11.

Why are the changes needed?

The new release contains multiple bug fixes and enhancements discussed in the Release Notes. Notable changes include:

• Improved socket timeout handling.
• Trace logging capabilities.
• Fix bug that prevented usage of GCS as a Hadoop Credential Provider.
• Dependency upgrades.
• Support OAuth2 based client authentication.

Does this PR introduce any user-facing change?

Distributions built with -Phadoop-cloud now include GCS connector 2.2.11 instead of 2.2.7.

cnauroth@cnauroth-2-1-m:~/spark-3.5.0-SNAPSHOT-bin-custom-spark$ ls -lrt jars/gcs*
-rw-r--r-- 1 cnauroth cnauroth 36497606 Mar 21 00:42 jars/gcs-connector-hadoop3-2.2.11-shaded.jar

How was this patch tested?

Build

I built a custom distro with -Phadoop-cloud:

./dev/make-distribution.sh --name custom-spark --pip --tgz -Phadoop-3 -Phadoop-cloud -Pscala-2.12

Run

I ran a PySpark job that successfully reads and writes using GCS:

from pyspark.sql import SparkSession
def main() -> None:
# Create SparkSession.
spark = (SparkSession.builder
.appName('copy-shakespeare')
.getOrCreate())
# Read.
df = spark.read.text('gs://dataproc-datasets-us-central1/shakespeare')
# Write.
df.write.text('gs://cnauroth-hive-metastore-proxy-dist/output/copy-shakespeare')
spark.stop()
if __name__ == '__main__':
main()

Authored-by: Chris Nauroth cnauroth@apache.org

[cnauroth]

@dongjoon-hyun , may I ask for your review, since you did the original import of the GCS connector in SPARK-33605/#37745? Thank you!

[dongjoon-hyun]

Sure. Thank you for making a PR, @cnauroth !
+1, LGTM (Pending CIs).

[dongjoon-hyun]

I also verified this PR manually with my personal account.

$ KEYFILE=...json
$ EMAIL=$(jq -r '.client_email' < $KEYFILE)
$ PRIVATE_KEY_ID=$(jq -r '.private_key_id' < $KEYFILE)
$ PRIVATE_KEY="$(jq -r '.private_key' < $KEYFILE)"
$ bin/spark-shell \
-c spark.hadoop.fs.gs.auth.service.account.email=$EMAIL \
-c spark.hadoop.fs.gs.auth.service.account.private.key.id=$PRIVATE_KEY_ID \
-c spark.hadoop.fs.gs.auth.service.account.private.key="$PRIVATE_KEY"
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/03/21 17:57:45 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id = local-1679446666474).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.5.0-SNAPSHOT
/_/
Using Scala version 2.12.17 (OpenJDK 64-Bit Server VM, Java 1.8.0_312)
Type in expressions to have them evaluated.
Type :help for more information.
scala> spark.read.text("gs://apache-spark-bucket/README.md").count()
res0: Long = 124
scala> spark.read.text("gs://apache-spark-bucket/README.md").write.text("gs://apache-spark-bucket/README.out")
scala> spark.read.text("gs://apache-spark-bucket/README.out").count()
res2: Long = 124

[dongjoon-hyun]

Merged to master/3.4 for Apache Spark 3.4.0. This will be a part of next Apache Spark 3.4.0 RC.
I added you to the Apache Spark contributor group and assigned SPARK-42888 to you.
Welcome to the Apache Spark community, @cnauroth .

Also, cc @sunchao

[sunchao]

LGTM too, thanks @cnauroth@dongjoon-hyun !

[cnauroth]

@dongjoon-hyun and @sunchao , thank you for the commit and the warm welcome!