[SPARK-31101][BUILD] Upgrade Janino to 3.0.16 by HeartSaVioR · Pull Request #27932 · apache/spark

HeartSaVioR · 2020-03-17T04:14:10Z

What changes were proposed in this pull request?

This PR(SPARK-31101) proposes to upgrade Janino to 3.0.16 which is released recently.

Merged pull request Issue #113: Grow the code for relocatables, and do fixup, and relocate janino-compiler/janino#114 "Grow the code for relocatables, and do fixup, and relocate".

Please see the commit log.

https://github.com/janino-compiler/janino/commits/3.0.16

You can see the changelog from the link: http://janino-compiler.github.io/janino/changelog.html / though release note for Janino 3.0.16 is actually incorrect.

Why are the changes needed?

We got some report on failure on user's query which Janino throws error on compiling generated code. The issue is here: janino-compiler/janino#113 It contains the information of generated code, symptom (error), and analysis of the bug, so please refer the link for more details.
Janino 3.0.16 contains the PR janino-compiler/janino#114 which would enable Janino to succeed to compile user's query properly.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing UTs.

…lish via Jitpack...

SparkQA · 2020-03-17T04:26:05Z

Test build #119906 has finished for PR 27932 at commit 1bf9e7d.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-03-17T06:50:08Z

Test build #119908 has finished for PR 27932 at commit 78f8021.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR · 2020-03-17T06:54:14Z

The failure is not from compilation from generated code, but no big deal on rerunning tests.

HeartSaVioR · 2020-03-17T06:54:21Z

retest this, please

SparkQA · 2020-03-17T09:56:49Z

Test build #119918 has finished for PR 27932 at commit 78f8021.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR · 2020-03-17T23:33:34Z

retest this, please

SparkQA · 2020-03-18T02:19:58Z

Test build #119956 has finished for PR 27932 at commit 78f8021.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR · 2020-03-18T07:19:35Z

retest this, please

SparkQA · 2020-03-18T09:33:29Z

Test build #119974 has finished for PR 27932 at commit 78f8021.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR · 2020-03-18T10:25:28Z

retest this, please

SparkQA · 2020-03-18T13:08:27Z

Test build #119984 has finished for PR 27932 at commit 78f8021.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-03-19T03:14:57Z

Test build #120016 has finished for PR 27932 at commit ff25ac5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR · 2020-03-19T03:43:30Z

org.apache.spark.scheduler.BarrierTaskContextSuite.successively sync with allGather and barrier
Error Details
org.apache.spark.SparkException: Job aborted due to stage failure: Could not recover from a failed barrier ResultStage. Most recent failure reason: Stage failed because barrier task ResultTask(0, 3) finished unsuccessfully. com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map due to end-of-input at [Source: (String)""; line: 1, column: 0] at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59) at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1442) at com.fasterxml.jackson.databind.ObjectReader._initForReading(ObjectReader.java:360) at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:1704) at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1261) at org.json4s.jackson.JsonMethods.parse(JsonMethods.scala:25) at org.json4s.jackson.JsonMethods.parse$(JsonMethods.scala:19) at org.json4s.jackson.JsonMethods$.parse(JsonMethods.scala:55) at org.apache.spark.BarrierTaskContext.allGather(BarrierTaskContext.scala:222) at org.apache.spark.scheduler.BarrierTaskContextSuite.$anonfun$new$12(BarrierTaskContextSuite.scala:118) at org.apache.spark.rdd.RDDBarrier.$anonfun$mapPartitions$2(RDDBarrier.scala:51) at org.apache.spark.rdd.RDDBarrier.$anonfun$mapPartitions$2$adapted(RDDBarrier.scala:51) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349) at org.apache.spark.rdd.RDD.iterator(RDD.scala:313) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:127) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:460) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:463) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)

Doesn't seem to be related.

HeartSaVioR · 2020-03-19T03:43:55Z

retest this, please

SparkQA · 2020-03-19T06:37:31Z

Test build #120019 has finished for PR 27932 at commit ff25ac5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR · 2020-03-19T06:43:53Z

org.apache.spark.ExecutorAllocationManagerSuite.add executors capped by num pending tasks Error Details
org.scalatest.exceptions.TestFailedException: 6 did not equal 5
org.apache.spark.scheduler.WorkerDecommissionSuite.verify a task with all workers decommissioned succeeds Error Details
java.util.concurrent.TimeoutException: Futures timed out after [2 seconds]
org.apache.spark.storage.BlockManagerReplicationSuite.block replication - mixed between 1x to 5x Error Details
org.scalatest.exceptions.TestFailedException: 3 did not equal 4 master did not have 4 locations for test_block-with-disk-memory-deserialized-4x-replicated

Another flaky ones.

SparkQA · 2020-03-19T10:10:38Z

Test build #120030 has finished for PR 27932 at commit a9f1f5d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun

+1, LGTM. Merged to master.
Thank you for your long investigating and patience, @HeartSaVioR .

I added the following into the PR description.

https://github.com/janino-compiler/janino/commits/3.0.16

HeartSaVioR · 2020-03-24T02:19:00Z

Thanks for sorting it out. Could we port back to 3.0 & 2.4 as well, given the origin bug was from Spark 2.3? I'll submit PRs for these versions. Thanks again!

### What changes were proposed in this pull request? This PR(SPARK-31101) proposes to upgrade Janino to 3.0.16 which is released recently. * Merged pull request janino-compiler/janino#114 "Grow the code for relocatables, and do fixup, and relocate". Please see the commit log. - https://github.com/janino-compiler/janino/commits/3.0.16 You can see the changelog from the link: http://janino-compiler.github.io/janino/changelog.html / though release note for Janino 3.0.16 is actually incorrect. ### Why are the changes needed? We got some report on failure on user's query which Janino throws error on compiling generated code. The issue is here: janino-compiler/janino#113 It contains the information of generated code, symptom (error), and analysis of the bug, so please refer the link for more details. Janino 3.0.16 contains the PR janino-compiler/janino#114 which would enable Janino to succeed to compile user's query properly. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Existing UTs. Closesapache#27932 from HeartSaVioR/SPARK-31101-janino-3.0.16. Authored-by: Jungtaek Lim (HeartSaVioR) <kabhwan.opensource@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

This PR(SPARK-31101) proposes to upgrade Janino to 3.0.16 which is released recently. * Merged pull request janino-compiler/janino#114 "Grow the code for relocatables, and do fixup, and relocate". Please see the commit log. - https://github.com/janino-compiler/janino/commits/3.0.16 You can see the changelog from the link: http://janino-compiler.github.io/janino/changelog.html / though release note for Janino 3.0.16 is actually incorrect. We got some report on failure on user's query which Janino throws error on compiling generated code. The issue is here: janino-compiler/janino#113 It contains the information of generated code, symptom (error), and analysis of the bug, so please refer the link for more details. Janino 3.0.16 contains the PR janino-compiler/janino#114 which would enable Janino to succeed to compile user's query properly. No. Existing UTs. Closesapache#27932 from HeartSaVioR/SPARK-31101-janino-3.0.16. Authored-by: Jungtaek Lim (HeartSaVioR) <kabhwan.opensource@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

HeartSaVioR · 2020-03-24T05:54:12Z

#27996 for Spark 3.0.0 / #27997 for Spark 2.4.x

### What changes were proposed in this pull request? This PR(SPARK-31101) proposes to upgrade Janino to 3.0.16 which is released recently. * Merged pull request janino-compiler/janino#114 "Grow the code for relocatables, and do fixup, and relocate". Please see the commit log. - https://github.com/janino-compiler/janino/commits/3.0.16 You can see the changelog from the link: http://janino-compiler.github.io/janino/changelog.html / though release note for Janino 3.0.16 is actually incorrect. ### Why are the changes needed? We got some report on failure on user's query which Janino throws error on compiling generated code. The issue is here: janino-compiler/janino#113 It contains the information of generated code, symptom (error), and analysis of the bug, so please refer the link for more details. Janino 3.0.16 contains the PR janino-compiler/janino#114 which would enable Janino to succeed to compile user's query properly. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Existing UTs. Closesapache#27932 from HeartSaVioR/SPARK-31101-janino-3.0.16. Authored-by: Jungtaek Lim (HeartSaVioR) <kabhwan.opensource@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

HeartSaVioR added 2 commits March 17, 2020 13:06

[SPARK-31101][SQL][BUILD] Upgrade Janino to 3.1.1
7f266d7

WIP! Bump up Janino to 3.0.15 +apache#113+ some modification to pub…
1bf9e7d
…lish via Jitpack...

Roll back the change only applicable in Janino 3.1.x
78f8021

HeartSaVioR mentioned this pull request Mar 17, 2020
[SPARK-31115][SQL] Detect known Janino bug janino-compiler/janino#113 and apply workaround automatically as a fail-back via avoid using switch statement in generated code #27872
Closed

HeartSaVioR changed the title ~~[WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16~~[WIP][SPARK-31101][SQL][BUILD][test-java11] Upgrade Janino to 3.0.16Mar 17, 2020

HeartSaVioR changed the title ~~[WIP][SPARK-31101][SQL][BUILD][test-java11] Upgrade Janino to 3.0.16~~[WIP][SPARK-31101][SQL][BUILD][test-hadoop3.2][test-java11] Upgrade Janino to 3.0.16Mar 18, 2020

HeartSaVioR changed the title ~~[WIP][SPARK-31101][SQL][BUILD][test-hadoop3.2][test-java11] Upgrade Janino to 3.0.16~~[SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16Mar 19, 2020

Apply Janino 3.0.16, and remove all hack around using Jitpack
a9f1f5d

HeartSaVioR force-pushed the SPARK-31101-janino-3.0.16 branch from ff25ac5 to a9f1f5dCompare March 19, 2020 07:07

dongjoon-hyun added the BUILD label Mar 22, 2020

dongjoon-hyun changed the title ~~[SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16~~[SPARK-31101][BUILD] Upgrade Janino to 3.0.16Mar 22, 2020

dongjoon-hyun approved these changes Mar 22, 2020
View reviewed changes

dongjoon-hyun closed this in f55f6b5Mar 22, 2020

HeartSaVioR deleted the SPARK-31101-janino-3.0.16 branch March 24, 2020 05:54

Uh oh!

Conversation

HeartSaVioR commented Mar 17, 2020• edited by dongjoon-hyun Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Mar 17, 2020

Uh oh!

SparkQA commented Mar 17, 2020

Uh oh!

HeartSaVioR commented Mar 17, 2020

Uh oh!

HeartSaVioR commented Mar 17, 2020

Uh oh!

SparkQA commented Mar 17, 2020

Uh oh!

HeartSaVioR commented Mar 17, 2020

Uh oh!

SparkQA commented Mar 18, 2020

Uh oh!

HeartSaVioR commented Mar 18, 2020

Uh oh!

SparkQA commented Mar 18, 2020

Uh oh!

HeartSaVioR commented Mar 18, 2020

Uh oh!

SparkQA commented Mar 18, 2020

Uh oh!

SparkQA commented Mar 19, 2020

Uh oh!

HeartSaVioR commented Mar 19, 2020

Uh oh!

HeartSaVioR commented Mar 19, 2020

Uh oh!

SparkQA commented Mar 19, 2020

Uh oh!

HeartSaVioR commented Mar 19, 2020

Uh oh!

SparkQA commented Mar 19, 2020

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HeartSaVioR commented Mar 24, 2020

Uh oh!

HeartSaVioR commented Mar 24, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HeartSaVioR commented Mar 17, 2020•
edited by dongjoon-hyun
Loading

dongjoon-hyun left a comment •
edited
Loading