[SPARK-31655][BUILD][3.0] Upgrade snappy-java to 1.1.7.5 by AngersZhuuuu · Pull Request #28509 · apache/spark

AngersZhuuuu · 2020-05-12T04:13:54Z

What changes were proposed in this pull request?

snappy-java have release v1.1.7.5, upgrade to latest version.

Fixed in v1.1.7.4

Caching internal buffers for SnappyFramed streams Fix org.scala-lang: * inconsistent versions #234
Fixed the native lib for ppc64le to work with glibc 2.17 (Previously it depended on 2.22)

Fixed in v1.1.7.5

Fixes java.lang.NoClassDefFoundError: org/xerial/snappy/pool/DefaultPoolFactory in 1.1.7.4

xerial/snappy-java@1.1.7.3...1.1.7.5

v 1.1.7.5 release note:
xerial/snappy-java@edc4ec2

Why are the changes needed?

Fix bug

Does this PR introduce any user-facing change?

No

How was this patch tested?

No need

dongjoon-hyun

Do you think we can have a valid unit test case for this? (non-ppc64le part)

SparkQA · 2020-05-12T04:44:30Z

Test build #122532 has finished for PR 28509 at commit 692db6a.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-05-12T04:55:46Z

BTW, currently, Jenkins jobs seems to be broken due to some outside issues.

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/

AngersZhuuuu · 2020-05-12T05:00:00Z

BTW, currently, Jenkins jobs seems to be broken due to some outside issues.
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/

No wonder this strange mistake

AngersZhuuuu · 2020-05-12T10:30:51Z

retest this please

SparkQA · 2020-05-12T10:58:55Z

Test build #122542 has finished for PR 28509 at commit 692db6a.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2020-05-12T13:29:12Z

This needs to go into master first, right?

AngersZhuuuu · 2020-05-12T13:43:44Z

This needs to go into master first, right?

Has been merged into master
#28472

Backport to 2.4 & 3.0 now.

Seems jenkins have some problem

srowen · 2020-05-12T13:48:14Z

Does test-dependencies.sh pass locally? I don't see any clear output here.

AngersZhuuuu · 2020-05-12T13:48:47Z

Does test-dependencies.sh pass locally? I don't see any clear output here.

Yeah, in local can pass it.

dongjoon-hyun · 2020-05-12T16:22:01Z

Retest this please.

SparkQA · 2020-05-12T16:55:01Z

Test build #122556 has finished for PR 28509 at commit 692db6a.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2020-05-13T11:18:23Z

Retest this please.

Do you know what happened? seems in master mvn is different from branch-3.0 and branch-2.4

In master's succes build

exec: curl --silent --show-error -L https://downloads.lightbend.com/zinc/0.3.15/zinc-0.3.15.tgz
exec: curl --silent --show-error -L https://downloads.lightbend.com/scala/2.12.10/scala-2.12.10.tgz
exec: curl --silent --show-error -L https://www.apache.org/dyn/closer.lua?action=download&filename=/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz
Using `mvn` from path: /home/jenkins/workspace/Spark............
..............

In branch 3.0 failed build

========================================================================
Running build tests
========================================================================
exec: curl -s -L https://downloads.lightbend.com/zinc/0.3.15/zinc-0.3.15.tgz
exec: curl -s -L https://downloads.lightbend.com/scala/2.12.10/scala-2.12.10.tgz
exec: curl -s -L https://archive.apache.org/dist/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn
Performing Maven install for hadoop-2.7-hive-1.2
Falling back to archive.apache.org to download Maven

in Branch-2.4 failed build

RAT checks passed.
========================================================================
Running build tests
========================================================================
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder@2/build/apache-maven-3.5.4/bin/mvn
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder@2/build/apache-maven-3.5.4/bin/mvn
Performing Maven install for hadoop-2.6
Falling back to archive.apache.org to download Maven

AngersZhuuuu · 2020-05-13T11:22:16Z

In auto check, test-deps.sh success.

dongjoon-hyun · 2020-05-13T20:27:52Z

There is a network outage in the Jenkins farm.
Please see the current status.

https://issues.apache.org/jira/browse/SPARK-31693

AngersZhuuuu · 2020-05-14T02:21:04Z

retest this please

SparkQA · 2020-05-14T02:34:17Z

Test build #122602 has finished for PR 28509 at commit 692db6a.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-05-14T22:03:55Z

FYI, the dependency failure will be fixed by the following.

[SPARK-31713][INFRA] Make test-dependencies.sh detect version string correctly #28532

dongjoon-hyun · 2020-05-15T05:28:24Z

Retest this please.

SparkQA · 2020-05-15T07:05:02Z

Test build #122652 has finished for PR 28509 at commit 692db6a.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-05-15T07:37:30Z

retest this please

SparkQA · 2020-05-15T12:18:21Z

Test build #122655 has finished for PR 28509 at commit 692db6a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2020-05-16T08:50:10Z

retest this please

SparkQA · 2020-05-16T09:03:36Z

Test build #122725 has finished for PR 28509 at commit 692db6a.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2020-05-16T11:39:29Z

@dongjoon-hyun Seems something still wrong, missing something on branch-3.0?

dongjoon-hyun · 2020-05-16T14:36:11Z

Yes. amp-jenkins-worker-05 server is still broken. If you click the Jenkins log, you can see that it failed on amp-jenkins-worker-05 in 10 minutes.

dongjoon-hyun · 2020-05-16T14:37:43Z

BTW, the previous failed UT, support multiple barrier() call within a single task, is disabled by @cloud-fan now. I merged it. Let's test again.

dongjoon-hyun · 2020-05-16T14:37:50Z

Retest this please

srowen · 2020-05-16T14:44:32Z

Hm, I wonder if mvn -U dependency:purge-local-repository would help, like opening a PR that just tries changing this in the test script, to see if it can be run once on the offending machine (and then not merge it)

dongjoon-hyun · 2020-05-16T18:47:42Z

Thank you for the tip. Yes. It's possible, but in a shared cluster, is it okay to purge it by one Maven job while the other Maven jobs are working? Maybe, we can enforce it at the specific single machine by recognizing the machine name because the machine seems to fail already always in any way.

srowen · 2020-05-16T19:30:30Z

Hm, not sure. I think it's thread-safe. At least, if it doesn't work at all now, can't be much worse. This would not be merged, just a one-time thing.

dongjoon-hyun · 2020-05-16T19:54:44Z

Got it. I'll try to make a PR to purge it.

SparkQA · 2020-05-16T19:57:07Z

Test build #122728 has finished for PR 28509 at commit 692db6a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun

+1, LGTM. Merged to branch-3.0.
cc @rxin and @yhuai since this is a dependency change in 3.0.

### What changes were proposed in this pull request? snappy-java have release v1.1.7.5, upgrade to latest version. Fixed in v1.1.7.4 - Caching internal buffers for SnappyFramed streams #234 - Fixed the native lib for ppc64le to work with glibc 2.17 (Previously it depended on 2.22) Fixed in v1.1.7.5 - Fixes java.lang.NoClassDefFoundError: org/xerial/snappy/pool/DefaultPoolFactory in 1.1.7.4 xerial/snappy-java@1.1.7.3...1.1.7.5 v 1.1.7.5 release note: xerial/snappy-java@edc4ec2 ### Why are the changes needed? Fix bug ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? No need Closes#28509 from AngersZhuuuu/SPARK-31655-3.0. Authored-by: angerszhu <angers.zhu@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

[SPARK-31655][BUILD][3.0] Upgrade snappy-java to 1.1.7.5
692db6a

probot-autolabelerBot added the BUILD label May 12, 2020

dongjoon-hyun reviewed May 12, 2020
View reviewed changes

AngersZhuuuu mentioned this pull request May 12, 2020
[SPARK-31655][BUILD][2.4] Upgrade snappy-java to 1.1.7.5 #28506
Closed

dongjoon-hyun mentioned this pull request May 15, 2020
[SPARK-31732][TESTS] Disable some flaky tests temporarily #28547
Closed

dongjoon-hyun approved these changes May 16, 2020
View reviewed changes

dongjoon-hyun closed this May 16, 2020

Uh oh!

Conversation

AngersZhuuuu commented May 12, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 12, 2020

Uh oh!

dongjoon-hyun commented May 12, 2020

Uh oh!

AngersZhuuuu commented May 12, 2020

Uh oh!

AngersZhuuuu commented May 12, 2020

Uh oh!

SparkQA commented May 12, 2020

Uh oh!

srowen commented May 12, 2020

Uh oh!

AngersZhuuuu commented May 12, 2020

Uh oh!

srowen commented May 12, 2020

Uh oh!

AngersZhuuuu commented May 12, 2020

Uh oh!

dongjoon-hyun commented May 12, 2020

Uh oh!

SparkQA commented May 12, 2020

Uh oh!

AngersZhuuuu commented May 13, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AngersZhuuuu commented May 13, 2020

Uh oh!

dongjoon-hyun commented May 13, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AngersZhuuuu commented May 14, 2020

Uh oh!

SparkQA commented May 14, 2020

Uh oh!

dongjoon-hyun commented May 14, 2020

Uh oh!

dongjoon-hyun commented May 15, 2020

Uh oh!

SparkQA commented May 15, 2020

Uh oh!

maropu commented May 15, 2020

Uh oh!

SparkQA commented May 15, 2020

Uh oh!

AngersZhuuuu commented May 16, 2020

Uh oh!

SparkQA commented May 16, 2020

Uh oh!

AngersZhuuuu commented May 16, 2020

Uh oh!

dongjoon-hyun commented May 16, 2020

Uh oh!

dongjoon-hyun commented May 16, 2020

Uh oh!

dongjoon-hyun commented May 16, 2020

Uh oh!

srowen commented May 16, 2020

Uh oh!

dongjoon-hyun commented May 16, 2020

Uh oh!

srowen commented May 16, 2020

Uh oh!

dongjoon-hyun commented May 16, 2020

Uh oh!

SparkQA commented May 16, 2020

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

dongjoon-hyun left a comment •
edited
Loading

AngersZhuuuu commented May 13, 2020•
edited
Loading

dongjoon-hyun commented May 13, 2020•
edited
Loading