[SPARK-12653][SQL] Re-enable test "SPARK-8489: MissingRequirementError during reflection" by dongjoon-hyun · Pull Request #11744 · apache/spark

dongjoon-hyun · 2016-03-16T00:06:23Z

What changes were proposed in this pull request?

The purpose of SPARK-12653 is re-enabling a regression test.
Historically, the target regression test is added by SPARK-8498, but is temporarily disabled by SPARK-12615 due to binary compatibility error.

The following is the current error message at the submitting spark job with the pre-built test.jar file in the target regression test.

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.SparkContext$.$lessinit$greater$default$6()Lscala/collection/Map;

Simple rebuilding test.jar can not recover the purpose of testcase since we need to support both Scala 2.10 and 2.11 for a while. For example, we will face the following Scala 2.11 error if we use test.jar built by Scala 2.10.

Exception in thread "main" java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaMirrors$JavaMirror;

This PR replace the existing test.jar with test-2.10.jar and test-2.11.jar and improve the regression test to use the suitable jar file.

How was this patch tested?

Pass the existing Jenkins test.

…ing reflection" Current Error: ``` Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.SparkContext$.$lessinit$greater$default$6()Lscala/collection/Map; ``` Scala 2.11 Error with test.jar built by Scala 2.10.5: ``` Exception in thread "main" java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaMirrors$JavaMirror; ```

SparkQA · 2016-03-16T01:23:57Z

Test build #53246 has finished for PR 11744 at commit 95f740f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2016-03-16T06:26:37Z

 // For more detail, see sql/hive/src/test/resources/regression-test-SPARK-8489/*scala.
- val testJar = "sql/hive/src/test/resources/regression-test-SPARK-8489/test.jar"
+ import Properties.versionString
+ val version = versionString.substring(versionString.indexOf(" ") + 1,


it might be better if you explicitly match 2.10 and 2.11, and throw exceptions for other things. I think the error message then would be more obvious when we introduce support for 2.12.

Thank you. I see. I will change soon.

rxin · 2016-03-16T06:26:43Z

LGTM other than that one comment.

dongjoon-hyun · 2016-03-16T07:44:16Z

I updated the code to use exact matching and tested on both 2.10 and 2.11 again.
Now, it becomes much robust as you said.
Thank you again, @rxin.

rxin · 2016-03-16T07:58:18Z

LGTM pending Jenkins

SparkQA · 2016-03-16T08:55:26Z

Test build #53300 has finished for PR 11744 at commit 38b9752.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2016-03-16T09:05:48Z

Merged to master. We'll watch the 2.10 builds too.

dongjoon-hyun · 2016-03-16T15:20:17Z

Thank you, @srowen .

…r during reflection" ## What changes were proposed in this pull request? The purpose of [SPARK-12653](https://issues.apache.org/jira/browse/SPARK-12653) is re-enabling a regression test. Historically, the target regression test is added by [SPARK-8498](apache@093c348), but is temporarily disabled by [SPARK-12615](apache@8ce645d) due to binary compatibility error. The following is the current error message at the submitting spark job with the pre-built `test.jar` file in the target regression test. ``` Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.SparkContext$.$lessinit$greater$default$6()Lscala/collection/Map; ``` Simple rebuilding `test.jar` can not recover the purpose of testcase since we need to support both Scala 2.10 and 2.11 for a while. For example, we will face the following Scala 2.11 error if we use `test.jar` built by Scala 2.10. ``` Exception in thread "main" java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaMirrors$JavaMirror; ``` This PR replace the existing `test.jar` with `test-2.10.jar` and `test-2.11.jar` and improve the regression test to use the suitable jar file. ## How was this patch tested? Pass the existing Jenkins test. Author: Dongjoon Hyun <dongjoon@apache.org> Closesapache#11744 from dongjoon-hyun/SPARK-12653.

## What changes were proposed in this pull request? Introduced by #21320 and #11744 ``` $ sbt > ++2.12.6 > project sql > compile ... [error] [warn] spark/sql/core/src/main/scala/org/apache/spark/sql/execution/ProjectionOverSchema.scala:41: match may not be exhaustive. [error] It would fail on the following inputs: (_, ArrayType(_, _)), (_, _) [error] [warn] getProjection(a.child).map(p => (p, p.dataType)).map { [error] [warn] [error] [warn] spark/sql/core/src/main/scala/org/apache/spark/sql/execution/ProjectionOverSchema.scala:52: match may not be exhaustive. [error] It would fail on the following input: (_, _) [error] [warn] getProjection(child).map(p => (p, p.dataType)).map { [error] [warn] ... ``` And ``` $ sbt > ++2.12.6 > project hive > testOnly *ParquetMetastoreSuite ... [error] /Users/rendong/wdi/spark/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSparkSubmitSuite.scala:22: object tools is not a member of package scala [error] import scala.tools.nsc.Properties [error] ^ [error] /Users/rendong/wdi/spark/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSparkSubmitSuite.scala:146: not found: value Properties [error] val version = Properties.versionNumberString match { [error] ^ [error] two errors found ... ``` ## How was this patch tested? Existing tests. Closes#22260 from sadhen/fix_exhaustive_match. Authored-by: 忍冬 <rendong@wacai.com> Signed-off-by: hyukjinkwon <gurwls223@apache.org>

## What changes were proposed in this pull request? Introduced by apache#21320 and apache#11744 ``` $ sbt > ++2.12.6 > project sql > compile ... [error] [warn] spark/sql/core/src/main/scala/org/apache/spark/sql/execution/ProjectionOverSchema.scala:41: match may not be exhaustive. [error] It would fail on the following inputs: (_, ArrayType(_, _)), (_, _) [error] [warn] getProjection(a.child).map(p => (p, p.dataType)).map { [error] [warn] [error] [warn] spark/sql/core/src/main/scala/org/apache/spark/sql/execution/ProjectionOverSchema.scala:52: match may not be exhaustive. [error] It would fail on the following input: (_, _) [error] [warn] getProjection(child).map(p => (p, p.dataType)).map { [error] [warn] ... ``` And ``` $ sbt > ++2.12.6 > project hive > testOnly *ParquetMetastoreSuite ... [error] /Users/rendong/wdi/spark/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSparkSubmitSuite.scala:22: object tools is not a member of package scala [error] import scala.tools.nsc.Properties [error] ^ [error] /Users/rendong/wdi/spark/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSparkSubmitSuite.scala:146: not found: value Properties [error] val version = Properties.versionNumberString match { [error] ^ [error] two errors found ... ``` ## How was this patch tested? Existing tests. Closesapache#22260 from sadhen/fix_exhaustive_match. Authored-by: 忍冬 <rendong@wacai.com> Signed-off-by: hyukjinkwon <gurwls223@apache.org>

dongjoon-hyun changed the title ~~[SPARK-12653] Re-enable test "SPARK-8489: MissingRequirementError during reflection"~~[SPARK-12653][SQL] Re-enable test "SPARK-8489: MissingRequirementError during reflection"Mar 16, 2016

rxin reviewed Mar 16, 2016
View reviewed changes

Use exact pattern match to prevent future problem like Scala 2.12.
38b9752

asfgit closed this in 431a3d0Mar 16, 2016

dongjoon-hyun deleted the SPARK-12653 branch March 29, 2016 05:28

da-liii mentioned this pull request Aug 29, 2018
[SQL][MINOR] Fix compiling for scala 2.12 #22260
Closed

da-liii mentioned this pull request Sep 2, 2018
[SPARK-25304][SQL][TEST] Fix HiveSparkSubmitSuite SPARK-8489 test for Scala 2.12 #22308
Closed

Navigation Menu

Uh oh!

[SPARK-12653][SQL] Re-enable test "SPARK-8489: MissingRequirementError during reflection" - #11744

[SPARK-12653][SQL] Re-enable test "SPARK-8489: MissingRequirementError during reflection"#11744
dongjoon-hyun wants to merge 2 commits into
apache:masterfrom
dongjoon-hyun:SPARK-12653

dongjoon-hyun commented Mar 16, 2016

Uh oh!

SparkQA commented Mar 16, 2016

Uh oh!

rxinMar 16, 2016

Uh oh!

dongjoon-hyunMar 16, 2016

Uh oh!

rxin commented Mar 16, 2016

Uh oh!

dongjoon-hyun commented Mar 16, 2016

Uh oh!

rxin commented Mar 16, 2016

Uh oh!

SparkQA commented Mar 16, 2016

Uh oh!

srowen commented Mar 16, 2016

Uh oh!

dongjoon-hyun commented Mar 16, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

dongjoon-hyun commented Mar 16, 2016

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Mar 16, 2016

Uh oh!

rxinMar 16, 2016

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyunMar 16, 2016

Choose a reason for hiding this comment

Uh oh!

rxin commented Mar 16, 2016

Uh oh!

dongjoon-hyun commented Mar 16, 2016

Uh oh!

rxin commented Mar 16, 2016

Uh oh!

SparkQA commented Mar 16, 2016

Uh oh!

srowen commented Mar 16, 2016

Uh oh!

dongjoon-hyun commented Mar 16, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants