[SPARK-27074][SQL][test-hadoop3.2][test-maven] Hive 3.1 metastore support HiveClientImpl.runHive

[wangyum]

What changes were proposed in this pull request?

Hive 3.1.1's CommandProcessor have 2 changes:

1. HIVE-17626(Hive 3.0.0) add ReExecDriver. So the current code path is:

   spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala

   Lines 736 to 742 in 02bbe97

   	case _ =>
   	if (state.out !=null) {
   	// scalastyle:off println
   	state.out.println(tokens(0) +""+ cmd_1)
   	// scalastyle:on println
   	}
   	Seq(proc.run(cmd_1).getResponseCode.toString)

   
   We can disable hive.query.reexecution.enabled to workaround this change.
2. HIVE-18238(Hive 3.0.0) changed the Driver.close() function return type. We can workaround it by driver.getClass.getMethod("close").invoke(driver)

So Hive 3.1 metastore could support HiveClientImpl.runHive after this pr.

How was this patch tested?

unit tests

[SparkQA]

Test build #103099 has finished for PR 23992 at commit 513098c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #105341 has finished for PR 23992 at commit c266a8e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[wangyum]

cc @dongjoon-hyun

[SparkQA]

Test build #105359 has finished for PR 23992 at commit 53ed19d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Could you add a comment about the reason why we use DISABLE REWRITE in Hive 3.x?

[wangyum]

OK.
https://github.com/apache/hive/blob/8fd90ea13e24c2de5e4abbd9e6ea427ede559a6c/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L13456-L13460

[SparkQA]

Test build #105385 has finished for PR 23992 at commit 38cd028.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[wangyum]

how can we build against the Hive 3.1 metastore to test? Or do we just have to manually provide hive 3.1 jars for now.

bin/spark-shell --conf spark.sql.hive.metastore.version=3.1.1 --conf spark.sql.hive.metastore.jars=maven

[SparkQA]

Test build #105454 has finished for PR 23992 at commit 7cf2875.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Retest this please.

[SparkQA]

Test build #105550 has finished for PR 23992 at commit 7cf2875.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[wangyum]

retest this please

[SparkQA]

Test build #105559 has finished for PR 23992 at commit 7cf2875.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Retest this please.

[dongjoon-hyun]

Retest this please.

[SparkQA]

Test build #105791 has finished for PR 23992 at commit 7cf2875.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #105792 has finished for PR 23992 at commit 7cf2875.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[wangyum]

Retest this please.

[SparkQA]

Test build #105795 has finished for PR 23992 at commit 7cf2875.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

+1, LGTM. Thank you, @wangyum .

Merged to master.