[SPARK-23854] Update Guava to 16.0.1

[Fokko]

Currently Spark is still on Guava 14.0.1, and therefore I would like to bump the version to 16.0.1.

Babysteps are important here, because we don't want to become incompatible with other technology stacks, but 14.0.1 is getting old. HashCodes has been deprecated in favor of HashCode.

https://issues.apache.org/jira/projects/SPARK/issues/SPARK-23854

What changes were proposed in this pull request?

To stay compatible with the software that uses a later version of Guava.

How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

Running the testsuite

Please review http://spark.apache.org/contributing.html before opening a pull request.

[hvanhovell]

@Fokko upgrading guava has been notoriously hard to do, it is used in a lot Spark's dependencies and the guava developers aggressively remove deprecated APIs; updating can easily break stuff (missing methods that sort of thing).

I am not saying that we shouldn't do this, I am just saying that we should be super careful and that a minor release (2.4) of Spark might not be the best time. cc @JoshRosen@srowen who might be able to shed a bit more light on this.

[srowen]

Agree, this is the problem. If there is no actual improvement or fix in the upgrade here, no value in doing this.

[AmplabJenkins]

Can one of the admins verify this patch?

[Fokko]

I think we should start updating Guava in small steps. In some of the application that I'm using, we're up to date at 24.1-jre. The 14.0.1 that is still from March 2013. I took the liberty to check the dependency tree of Apache Spark:

Hadoop 2.6.5 is still on 14.0.1:

[INFO] +- org.apache.hadoop:hadoop-client:jar:2.6.5:test
[INFO] | +- org.apache.hadoop:hadoop-common:jar:2.6.5:test
[INFO] | | +- com.google.guava:guava:jar:14.0.1:provided

Jersey is running on a repackaged Guava:

[INFO] +- org.glassfish.jersey.core:jersey-common:jar:2.22.2:compile
[INFO] | +- org.glassfish.jersey.bundles.repackaged:jersey-guava:jar:2.22.2:compile

The rest is also still on 14.0.1:

[INFO] +- org.apache.orc:orc-mapreduce:jar:nohive:1.4.1:compile
[INFO] | +- com.google.guava:guava:jar:14.0.1:provided

[INFO] +- org.apache.curator:curator-recipes:jar:2.6.0:provided
[INFO] | +- org.apache.curator:curator-framework:jar:2.6.0:provided
[INFO] | \- com.google.guava:guava:jar:14.0.1:provided

[hvanhovell]

@Fokko you are kind of making my point here. Those dependencies rely on unshaded guava, if we upgrade our dependency then we might have to update those dependencies as well, which is probably not feasible.

Why - besides it being old - should we update?