[SPARK-27402][SQL][test-hadoop3.2][test-maven] Fix hadoop-3.2 test issue(except the hive-thriftserver module) by wangyum · Pull Request #24391 · apache/spark

wangyum · 2019-04-17T06:47:45Z

What changes were proposed in this pull request?

This pr fix hadoop-3.2 test issues(except the hive-thriftserver module):

Add hive.metastore.schema.verification and datanucleus.schema.autoCreateAll to HiveConf.
hadoop-3.2 support access the Hive metastore from 0.12 to 2.2

After SPARK-27176 and this PR, we upgraded the built-in Hive to 2.3 when enabling the Hadoop 3.2+ profile. This upgrade fixes the following issues:

HIVE-6727: Table level stats for external tables are set incorrectly.
HIVE-15653: Some ALTER TABLE commands drop table stats.
SPARK-12014: Spark SQL query containing semicolon is broken in Beeline.
SPARK-25193: insert overwrite doesn't throw exception when drop old data fails.
SPARK-25919: Date value corrupts when tables are "ParquetHiveSerDe" formatted and target table is Partitioned.
SPARK-26332: Spark sql write orc table on viewFS throws exception.
SPARK-26437: Decimal data becomes bigint to query, unable to query.

How was this patch tested?

This pr test Spark’s Hadoop 3.2 profile on jenkins and #24591 test Spark’s Hadoop 2.7 profile on jenkins

This PR close#24591

wangyum · 2019-04-17T06:55:43Z

 for dep in dependencies:
 dep.dependent_modules.add(self)
- all_modules.append(self)
+ if name == "hive-thriftserver" and hadoop_version == "hadoop3.2":


Used to skip hive-thriftserver module for hadoop-3.2. Will revert this change once we can merge.

Maybe leave a TODO here just to try to make sure that doesn't get lost

SparkQA · 2019-04-17T07:05:01Z

Test build #104646 has finished for PR 24391 at commit 9c0046a.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-04-17T07:12:12Z

retest this please

wangyum · 2019-04-17T10:26:06Z

 protected def isBarrierClass(name: String): Boolean =
 name.startsWith(classOf[HiveClientImpl].getName) ||
 name.startsWith(classOf[Shim].getName) ||
+ name.startsWith(classOf[ShimLoader].getName) ||


Add org.apache.hadoop.hive.shims.ShimLoader to BarrierClass, otherwise hadoop-3.2 can't access the Hive metastore from 0.12 to 2.2:
build/sbt"hive/testOnly *.VersionsSuite" -Phadoop-3.2 -Phive ... [info] - 0.12: createclient *** FAILED *** (36seconds, 207milliseconds) [info] java.lang.reflect.InvocationTargetException: [info] atsun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeMethod) [info] atsun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [info] atsun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [info] atjava.lang.reflect.Constructor.newInstance(Constructor.java:423) [info] atorg.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:295) [info] atorg.apache.spark.sql.hive.client.HiveClientBuilder$.buildClient(HiveClientBuilder.scala:58) [info] atorg.apache.spark.sql.hive.client.VersionsSuite.$anonfun$new$7(VersionsSuite.scala:130) [info] atorg.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85) [info] atorg.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83) [info] atorg.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) [info] atorg.scalatest.Transformer.apply(Transformer.scala:22) [info] atorg.scalatest.Transformer.apply(Transformer.scala:20) [info] atorg.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186) [info] atorg.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:105) [info] atorg.scalatest.FunSuiteLike.invokeWithFixture$1(FunSuiteLike.scala:184) [info] atorg.scalatest.FunSuiteLike.$anonfun$runTest$1(FunSuiteLike.scala:196) [info] atorg.scalatest.SuperEngine.runTestImpl(Engine.scala:289) [info] atorg.scalatest.FunSuiteLike.runTest(FunSuiteLike.scala:196) [info] atorg.scalatest.FunSuiteLike.runTest$(FunSuiteLike.scala:178) [info] atorg.scalatest.FunSuite.runTest(FunSuite.scala:1560) [info] atorg.scalatest.FunSuiteLike.$anonfun$runTests$1(FunSuiteLike.scala:229) [info] atorg.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:396) [info] atscala.collection.immutable.List.foreach(List.scala:392) [info] atorg.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384) [info] atorg.scalatest.SuperEngine.runTestsInBranch(Engine.scala:379) [info] atorg.scalatest.SuperEngine.runTestsImpl(Engine.scala:461) [info] atorg.scalatest.FunSuiteLike.runTests(FunSuiteLike.scala:229) [info] atorg.scalatest.FunSuiteLike.runTests$(FunSuiteLike.scala:228) [info] atorg.scalatest.FunSuite.runTests(FunSuite.scala:1560) [info] atorg.scalatest.Suite.run(Suite.scala:1147) [info] atorg.scalatest.Suite.run$(Suite.scala:1129) [info] atorg.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560) [info] atorg.scalatest.FunSuiteLike.$anonfun$run$1(FunSuiteLike.scala:233) [info] atorg.scalatest.SuperEngine.runImpl(Engine.scala:521) [info] atorg.scalatest.FunSuiteLike.run(FunSuiteLike.scala:233) [info] atorg.scalatest.FunSuiteLike.run$(FunSuiteLike.scala:232) [info] atorg.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:54) [info] atorg.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213) [info] atorg.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210) [info] atorg.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208) [info] atorg.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:54) [info] atorg.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314) [info] atorg.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:507) [info] atsbt.ForkMain$Run$2.call(ForkMain.java:296) [info] atsbt.ForkMain$Run$2.call(ForkMain.java:286) [info] atjava.util.concurrent.FutureTask.run(FutureTask.java:266) [info] atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [info] atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [info] atjava.lang.Thread.run(Thread.java:748) [info] Cause: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: java.lang.IllegalArgumentException: UnrecognizedHadoopmajorversionnumber: 3.2.0 [info] atorg.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:286) [info] atorg.apache.spark.sql.hive.client.HiveClientImpl.newState(HiveClientImpl.scala:187) [info] atorg.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:119)

SparkQA · 2019-04-17T11:30:28Z

Test build #104649 has finished for PR 24391 at commit 9c0046a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

…commons-httpclient.jar (1ms)

SparkQA · 2019-04-17T12:18:57Z

Test build #104647 has finished for PR 24391 at commit 9c0046a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-17T16:58:19Z

Test build #104657 has finished for PR 24391 at commit 490a751.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

# Conflicts: # pom.xml

SparkQA · 2019-04-19T18:24:35Z

Test build #104750 has finished for PR 24391 at commit 70b4b0b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-04-20T02:02:53Z

retest this please

SparkQA · 2019-04-20T03:30:15Z

Test build #104768 has finished for PR 24391 at commit 70b4b0b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-04-22T15:54:47Z

retest this please

SparkQA · 2019-04-22T16:54:07Z

Test build #104805 has finished for PR 24391 at commit 0893172.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-22T18:12:52Z

Test build #104809 has finished for PR 24391 at commit 0893172.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-04-23T15:15:06Z

retest this please

SparkQA · 2019-04-23T17:58:48Z

Test build #104838 has finished for PR 24391 at commit 0893172.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-27T14:44:43Z

Test build #104963 has finished for PR 24391 at commit 262be84.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-04-27T14:56:57Z

retest this please

SparkQA · 2019-04-27T16:34:11Z

Test build #104964 has finished for PR 24391 at commit 262be84.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-04-27T22:25:18Z

retest this please

gatorsmile · 2019-05-03T06:02:13Z

- assert(properties("totalSize").toLong <= 0, "external table totalSize must be <= 0")
- assert(properties("rawDataSize").toLong <= 0, "external table rawDataSize must be <= 0")
+ if (HiveUtils.isHive23) {
+ assert(properties("totalSize").toLong > 0, "external table totalSize must be > 0")


what is the number? How does Hive collect this info?

gatorsmile · 2019-05-03T06:57:54Z

My review is not finished. I will review it this weekend. If I do not leave the comments, please ping me using IMs

SparkQA · 2019-05-03T21:03:19Z

Test build #105106 has finished for PR 24391 at commit e321f61.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2019-05-06T17:01:20Z

+ hadoop_version = os.environ.get("AMPLAB_JENKINS_BUILD_PROFILE", "hadoop2.7")
+else:
+ hadoop_version = os.environ.get("HADOOP_PROFILE", "hadoop2.7")
+


We should have a log message to show which profile we are using; otherwise, it is hard for us to know which profile is activated.

Yes. we already have these log message:

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105106/consoleFull

What I mean, we should have a log message to show which profile is being actually used.
Just relying on the command parameter lists does not sound very reliable

OK. updated to:

gatorsmile · 2019-05-06T20:50:31Z


 ## Upgrading From Spark SQL 2.4 to 3.0

+ - Since Spark 3.0, we upgraded the built-in Hive to 2.3 for hadoop-3.x. This upgrade fixes some Hive issues:


for hadoop-3.x
when enabling the Hadoop 3.2+ profile.

SparkQA · 2019-05-07T04:51:58Z

Test build #105177 has finished for PR 24391 at commit aaf2952.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2019-05-10T04:43:55Z


 ## Upgrading From Spark SQL 2.4 to 3.0

+ - Since Spark 3.0, we upgraded the built-in Hive to 2.3 when enabling the Hadoop 3.2+ profile. This upgrade fixes some Hive issues:


some Hive issues -> the following issues:

SparkQA · 2019-05-10T06:33:21Z

Test build #105302 has finished for PR 24391 at commit 994b5a7.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2019-05-10T07:04:57Z

 // assert(storageFormat.properties.get("path") === expected)
 // }
- assert(storageFormat.locationUri === Some(expected))
+ assert(Some(storageFormat.locationUri.get.getPath) === Some(expected.getPath))


Do we need to add Some?
What is the reason we need to change this test case?

Add Some to support both Hive 1.2 and 2.3.

storageFormat.locationUri.map(_.getPath) then

gatorsmile · 2019-05-10T07:07:58Z

+ // We hard-code these configurations here to allow bin/spark-shell, bin/spark-sql
+ // and sbin/start-thriftserver.sh automatically creates Derby metastore.
+ hiveConf.setBoolean("hive.metastore.schema.verification", false)
+ hiveConf.setBoolean("datanucleus.schema.autoCreateAll", true)


Could you add a test case to ensure that we still respect the values set from users?
For example, if users set a different value, we do not reset them to our values.

I update it to:
defconf:HiveConf= { valhiveConf= state.getConf // Hive changed the default of datanucleus.schema.autoCreateAll from true to false// and hive.metastore.schema.verification from false to true since Hive 2.0.// For details, see the JIRA HIVE-6113, HIVE-12463 and HIVE-1841.// For the production environment. Either isDefaultMSUri or isDerbyMS should not be true.// We hard-code hive.metastore.schema.verification and datanucleus.schema.autoCreateAll to allow// bin/spark-shell, bin/spark-sql and sbin/start-thriftserver.sh to automatically create the// Derby Metastore when running Spark in the non-production environment.valisDefaultMSUri= hiveConf.getVar(METASTOREURIS).equals(METASTOREURIS.defaultStrVal) valisDerbyMS= hiveConf.getVar(METASTORECONNECTURLKEY).startsWith("jdbc:derby") if (isDefaultMSUri && isDerbyMS) { hiveConf.setBoolean("hive.metastore.schema.verification", false) hiveConf.setBoolean("datanucleus.schema.autoCreateAll", true) } hiveConf }
Because we don't know it's default value or set by user. so just set it for non-production environment.

gatorsmile · 2019-05-10T07:20:36Z

My review is finished. Overall, it looks good to me. After we ensure that maven can pass all the tests, we also need to rerun SBT as well.

gatorsmile · 2019-05-10T07:20:49Z

@wangyum Thank you for your hard work!!!

srowen · 2019-05-10T09:12:10Z

 for dep in dependencies:
 dep.dependent_modules.add(self)
- all_modules.append(self)
+ if name == "hive-thriftserver" and hadoop_version == "hadoop3.2":


Maybe leave a TODO here just to try to make sure that doesn't get lost

srowen · 2019-05-10T09:12:51Z


 ## Upgrading From Spark SQL 2.4 to 3.0

+ - Since Spark 3.0, we upgraded the built-in Hive to 2.3 when enabling the Hadoop 3.2+ profile. This upgrade fixes the following issues:


Does this need to go in the migration guide? it's helpful info but does it help a user know what they need to do differently?

Add to release note?

The JIRA lists can be moved to the PR description, but we should document the actual behavior difference in migration guide.

OK. Create SPARK-27686 to update migration guide

srowen · 2019-05-10T09:13:41Z

 // assert(storageFormat.properties.get("path") === expected)
 // }
- assert(storageFormat.locationUri === Some(expected))
+ assert(Some(storageFormat.locationUri.get.getPath) === Some(expected.getPath))


storageFormat.locationUri.map(_.getPath) then

…ronment

SparkQA · 2019-05-10T21:04:09Z

Test build #105319 has finished for PR 24391 at commit 4372759.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-05-11T06:00:32Z

Test build #105326 has finished for PR 24391 at commit 8520124.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2019-05-13T04:06:27Z


 /** Returns the configuration for the current session. */
- def conf: HiveConf = state.getConf
+ def conf: HiveConf = {


To avoid a regression, could you change it to ?
if (!isHive23) { state.getConf } else { ... }

SparkQA · 2019-05-13T07:05:01Z

Test build #105344 has finished for PR 24391 at commit 27e5b6e.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-05-13T07:13:02Z

retest this please

SparkQA · 2019-05-13T12:26:55Z

Test build #105346 has finished for PR 24391 at commit 27e5b6e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2019-05-13T17:34:45Z

LGTM

Thanks! Merged to master.

HyukjinKwon · 2019-05-20T02:02:46Z

 import re
+import os
+
+if os.environ.get("AMPLAB_JENKINS"):


@wangyum, this will shows the info every time this modules is imported. why did we do this here?

okay. it's a temp fix so I'm fine. I will make a followup to handle https://github.com/apache/spark/pull/24639/files

Skip the hive-thriftserver module when running the hadoop-3.2 test. Will remove it in another PR: https://github.com/apache/spark/pull/24628/files

Thank you @HyukjinKwon

Fix hadoop-3.2 test issue
9c0046a

wangyum commented Apr 17, 2019
View reviewed changes

Workaround: [NOT FOUND ] commons-httpclient#commons-httpclient;3.0.1!…
490a751
…commons-httpclient.jar (1ms)

wangyum commented Apr 17, 2019
View reviewed changes

Comment threadsql/hive/src/main/scala/org/apache/spark/sql/hive/client/package.scala Outdated

HyukjinKwon reviewed Apr 18, 2019
View reviewed changes

Comment threadsql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala Outdated

wangyum added 2 commits April 20, 2019 00:12

Merge remote-tracking branch 'upstream/master' intoSPARK-27402
24598ee
# Conflicts: # pom.xml

Try to revert exclusions
70b4b0b

wangyum added 2 commits April 22, 2019 22:09

Merge remote-tracking branch 'upstream/master' intoSPARK-27402
d2f83f6

Remove derby from SharedClass
0893172

wangyum changed the title ~~[SPARK-27402][SQL][test-hadoop3.2] Fix hadoop-3.2 test issue(except the hive-thriftserver module)~~[SPARK-27402][SQL][test-hadoop2.7] Fix hadoop-3.2 test issue(except the hive-thriftserver module)Apr 22, 2019

wangyum changed the title ~~[SPARK-27402][SQL][test-hadoop2.7] Fix hadoop-3.2 test issue(except the hive-thriftserver module)~~[SPARK-27402][SQL][test-hadoop3.2] Fix hadoop-3.2 test issue(except the hive-thriftserver module)Apr 23, 2019

Revert exclusion of commons-httpclient because it's jenkins issue
262be84

gatorsmile reviewed May 3, 2019
View reviewed changes

Upgrade migration guide
e321f61

gatorsmile reviewed May 6, 2019
View reviewed changes

for hadoop-3.x -> when enabling the Hadoop 3.2+ profile
aaf2952

gatorsmile reviewed May 10, 2019
View reviewed changes

print Hadoop profile.
994b5a7

gatorsmile reviewed May 10, 2019
View reviewed changes

srowen reviewed May 10, 2019
View reviewed changes

Only set datanucleus.schema.autoCreateAll=true in non-production envi…
4372759
…ronment

Fix Hive 0.12 test error
8520124

gatorsmile reviewed May 13, 2019
View reviewed changes

avoid regression
27e5b6e

wangyum mentioned this pull request May 13, 2019
[WIP][SPARK-27402][SQL] Test SPARK-27402 using Spark’s Hadoop 2.7 profile #24591
Closed

gatorsmile closed this in f3ddd6fMay 13, 2019

HyukjinKwon reviewed May 20, 2019
View reviewed changes


		## Upgrading From Spark SQL 2.4 to 3.0

		- Since Spark 3.0, we upgraded the built-in Hive to 2.3 for hadoop-3.x. This upgrade fixes some Hive issues:

Uh oh!

Conversation

wangyum commented Apr 17, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 17, 2019

Uh oh!

wangyum commented Apr 17, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 17, 2019

Uh oh!

Uh oh!

SparkQA commented Apr 17, 2019

Uh oh!

SparkQA commented Apr 17, 2019

Uh oh!

Uh oh!

SparkQA commented Apr 19, 2019

Uh oh!

wangyum commented Apr 20, 2019

Uh oh!

SparkQA commented Apr 20, 2019

Uh oh!

wangyum commented Apr 22, 2019

Uh oh!

SparkQA commented Apr 22, 2019

Uh oh!

SparkQA commented Apr 22, 2019

Uh oh!

wangyum commented Apr 23, 2019

Uh oh!

SparkQA commented Apr 23, 2019

Uh oh!

SparkQA commented Apr 27, 2019

Uh oh!

wangyum commented Apr 27, 2019

Uh oh!

SparkQA commented Apr 27, 2019

Uh oh!

wangyum commented Apr 27, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gatorsmile commented May 3, 2019

Uh oh!

SparkQA commented May 3, 2019

Uh oh!

gatorsmileMay 6, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gatorsmileMay 6, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 7, 2019

Uh oh!

gatorsmileMay 10, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 10, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

wangyum commented Apr 17, 2019•
edited
Loading

gatorsmileMay 6, 2019•
edited
Loading

gatorsmileMay 6, 2019•
edited
Loading

gatorsmileMay 10, 2019•
edited
Loading