[SPARK-34512][BUILD][SQL] Upgrade built-in Hive to 2.3.9

[wangyum]

What changes were proposed in this pull request?

This pr upgrades built-in Hive to 2.3.9. Hive 2.3.9 changes:

• [HIVE-17155] - findConfFile() in HiveConf.java has some issues with the conf path
• [HIVE-24797] - Disable validate default values when parsing Avro schemas
• [HIVE-24608] - Switch back to get_table in HMS client for Hive 2.3.x
• [HIVE-21200] - Vectorization: date column throwing java.lang.UnsupportedOperationException for parquet
• [HIVE-21563] - Improve Table#getEmptyTable performance by disabling registerAllFunctionsOnce
• [HIVE-19228] - Remove commons-httpclient 3.x usage

Why are the changes needed?

Fix regression caused by AVRO-2035.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Unit test.

[wangyum]

cc @sunchao

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43750/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43750/

[SparkQA]

Test build #139227 has finished for PR 32750 at commit 570dc84.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Thank you, @wangyum . It seems that the downloading still have some issue.

sbt.ForkMain$ForkError: java.lang.RuntimeException: [unresolved dependency: org.apache.hive.hcatalog#hive-hcatalog-core;2.3.9: not found]
at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1446)

[SparkQA]

Test build #139246 has finished for PR 32750 at commit d48df90.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43769/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43770/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43770/

[SparkQA]

Test build #139247 has finished for PR 32750 at commit f81813f.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43782/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43782/

[SparkQA]

Test build #139258 has finished for PR 32750 at commit c7194cf.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Hi, @wangyum .

@sunchao uploaded Apache Hive 2.3.9 to the Maven Central. Could you revise this PR?

• https://repo1.maven.org/maven2/org/apache/hive/hive-exec/2.3.9/

[dongjoon-hyun]

+1, LGTM (Pending CIs)
Thank you for updating.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44191/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44191/

[SparkQA]

Test build #139663 has finished for PR 32750 at commit ff0170e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Merged to master for Apache Spark 3.2.0.
Thank you so much, @wangyum , @srowen and @sunchao !