[BEAM-8676] sdks/java: gax and grpc upgrades by suztomo · Pull Request #10554 · apache/beam

suztomo · 2020-01-10T19:56:39Z

https://issues.apache.org/jira/browse/BEAM-8676. This PR is a revival of #10438.

Gax and gRPC work together. Upgrading their dependencies to higher versions.

The latest gax-grpc:1.52.0 depends on gRPC 1.25.0. (So not setting grpc 1.26.0)
Although grpc 1.25 declares Guava 28.1 dependency, grpc 1.25 does not have linkage errors with Guava 25.1-jre (which current master has).

Which modules use gax and grpc?

find . -name 'build.gradle' |xargs grep -ir library.java.gax (and grpc) gave

./sdks/java/io/google-cloud-platform/build.gradle
./sdks/java/extensions/sql/datacatalog/build.gradle
./sdks/java/extensions/sql/zetasql/build.gradle

Linkage Check

beam-sdks-java-io-google-cloud-platform

before: https://gist.github.com/suztomo/d0446fb63aea142db1a8086da702a04d (master ba4dc72)
after: https://gist.github.com/suztomo/5c0cd8699f019d03ae8bc0a58e3c7116 (this PR b4bd0bc)

This PR removes the following linkage errors:

suztomo@suxtomo24:~/beam6$ diff /tmp/linkagecheck-beam-sdks-java-io-google-cloud-platform-ba4dc72c /tmp/linkagecheck-beam-sdks-java-io-google-cloud-platform-b4bd0bc5 22,36d21
< Class io.opencensus.trace.Tracestate is not found;
< referenced by 1 class file
< io.opencensus.contrib.http.util.CloudTraceFormat (opencensus-contrib-http-util-0.18.0.jar)
< Class io.opencensus.trace.Tracestate$Builder is not found;
< referenced by 1 class file
< io.opencensus.contrib.http.util.CloudTraceFormat (opencensus-contrib-http-util-0.18.0.jar)
< Class com.google.common.truth.Truth is not found;
< referenced by 1 class file
< io.grpc.internal.testing.AbstractTransportTest (grpc-testing-1.17.1.jar)
< Class com.google.common.truth.StringSubject is not found;
< referenced by 1 class file
< io.grpc.internal.testing.AbstractTransportTest (grpc-testing-1.17.1.jar)
< Class com.google.common.truth.LongSubject is not found;
< referenced by 1 class file
< io.grpc.internal.testing.AbstractTransportTest (grpc-testing-1.17.1.jar)
44,55d28
< Class org.apache.tools.ant.Task is not found;
< referenced by 1 class file
< org.mockito.cglib.transform.AbstractProcessTask (mockito-core-1.9.5.jar)
< Class org.apache.tools.ant.types.FileSet is not found;
< referenced by 1 class file
< org.mockito.cglib.transform.AbstractProcessTask (mockito-core-1.9.5.jar)
< Class org.apache.tools.ant.DirectoryScanner is not found;
< referenced by 1 class file
< org.mockito.cglib.transform.AbstractProcessTask (mockito-core-1.9.5.jar)
< Class org.apache.tools.ant.BuildException is not found;
< referenced by 1 class file
< org.mockito.cglib.transform.AbstractProcessTask (mockito-core-1.9.5.jar)
870,872d842
< (opencensus-api-0.15.0.jar) io.opencensus.trace.SpanContext's method create(io.opencensus.trace.TraceId arg1, io.opencensus.trace.SpanId arg2, io.opencensus.trace.TraceOptions arg3, io.opencensus.trace.Tracestate arg4) is not found;
< referenced by 1 class file
< io.opencensus.contrib.http.util.CloudTraceFormat (opencensus-contrib-http-util-0.18.0.jar)
883c853
< BUILD SUCCESSFUL in 31s
---
> BUILD SUCCESSFUL in 30s

beam-sdks-java-extensions-sql-zetasql

Linkage Checker does not work for beam-sdks-java-extensions-sql-zetasql. (Bug: GoogleCloudPlatform/cloud-opensource-java#1087)

beam-sdks-java-extensions-sql-datacatalog

Linkage Checker does not work for beam-sdks-java-extensions-sql-datacatalog https://gist.github.com/suztomo/9c7723a2d126459bebfa2f52b5573757 (the same reason as above)

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	SDK	Apex	Dataflow	Gearpump	Samza	Spark
Go		---	---	---	---
Java
Python		---		---	---
XLang	---	---	---	---	---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website
Non-portable
Portable	---		---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

suztomo · 2020-01-10T20:28:43Z

 compile library.java.grpc_all
 compile library.java.grpc_auth
 compile library.java.grpc_core
+ compile library.java.grpc_context


Unless this declares grpc-context dependency, grpc-context’s old version sneaks in the class path. Here is an error from linkage checker before adding this grpc-context dependency:
(grpc-context-1.16.1.jar) io.grpc.Deadline's method getSystemTicker() is not found; referenced by 1 class file io.grpc.internal.AbstractServerImplBuilder (grpc-core-1.25.0.jar)
https://gist.github.com/suztomo/9af333b78b157d6a829d59ac5353e01d#file-gistfile1-txt-L856

suztomo · 2020-01-10T20:29:03Z

 def google_cloud_core_version = "1.61.0"
 def google_cloud_spanner_version = "1.6.0"
- def grpc_version = "1.17.1"
+ def grpc_version = "1.25.0"


The latest version of gRPC is 1.26.0, but setting 1.25.0 as gax-grpc:1.52.0 depends on 1.25.0.

suztomo · 2020-01-10T20:29:21Z

 compile project(":sdks:java:extensions:protobuf")
 compile library.java.avro
 compile library.java.bigdataoss_util
+ compile library.java.gax


As BigQueryServicesImpl uses gax's com.google.api.gax.core.FixedCredentialsProvider (undeclared-but-used dependency), declaring it explicitly.

suztomo · 2020-01-10T20:56:15Z

R: @lukecwik, @kennknowles

apilloud · 2020-01-10T21:03:40Z

Run Java PostCommit

apilloud · 2020-01-10T21:04:43Z

Run Java HadoopFormatIO Performance Test

apilloud · 2020-01-10T21:04:52Z

Run BigQueryIO Streaming Performance Test Java

apilloud · 2020-01-10T21:04:59Z

Run Dataflow ValidatesRunner

apilloud · 2020-01-10T21:05:05Z

Run Spark ValidatesRunner

apilloud · 2020-01-10T21:05:11Z

Run SQL Postcommit

suztomo · 2020-01-11T01:46:41Z

It seems Jenkins went down while running the 4 checks.

java.io.IOException: Backing channel 'JNLP4-connect connection from 255.248.197.104.bc.googleusercontent.com/104.197.248.255:33122' is disconnected.
https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_PR/187/console

yifanzou · 2020-01-13T07:58:18Z

Run Java PostCommit

yifanzou · 2020-01-13T07:58:29Z

Run Java HadoopFormatIO Performance Test

yifanzou · 2020-01-13T07:58:42Z

Run BigQueryIO Streaming Performance Test Java

yifanzou · 2020-01-13T07:58:51Z

Run Dataflow ValidatesRunner

yifanzou · 2020-01-13T07:59:03Z

Run Spark ValidatesRunner

yifanzou · 2020-01-13T07:59:10Z

Run SQL Postcommit

suztomo · 2020-01-13T13:31:25Z

Apache Spark Runner ValidatesRunner Tests failed:
java.lang.IllegalStateException: No TransformEvaluator registered for UNBOUNDED transform TestStream

Google Cloud Dataflow Runner ValidatesRunner Tests failed:
java.lang.RuntimeException: Workflow failed. Causes: Unknown streaming source: test_stream

I will check whether they’re caused by this dependency upgrades or not.

suztomo · 2020-01-13T13:50:40Z

@yifanzou thanks

suztomo · 2020-01-13T15:58:06Z

The two postcommit checks Apache Spark Runner ValidatesRunner Tests and Google Cloud Dataflow Runner ValidatesRunner Tests fail because of BEAM-9083PR9677 breaks ValidatesRunnerTest of open source runners since 3 days ago.

iemejia · 2020-01-13T16:10:37Z

Run Java PreCommit

suztomo · 2020-01-13T17:04:12Z

R: @lukecwik, @kennknowles

Now all checks, except the ones in BEAM-9083 (no ETA), passed.

suztomo · 2020-01-13T17:04:25Z

@iemejia Thank you.

markflyhigh · 2020-01-13T19:33:50Z

Run Java PreCommit

lukecwik · 2020-01-13T21:08:31Z

Run Spark ValidatesRunner

lukecwik · 2020-01-13T21:08:45Z

#10564 may have addressed the spark issues

lukecwik · 2020-01-13T21:10:46Z

The dataflow VR test failure of Unknown streaming source: test_stream is being investigated in https://lists.apache.org/thread.html/r65d981d36d91bb2fd85ddedac3db4256c97f08747c38384166fe5481%40%3Cdev.beam.apache.org%3E

lukecwik · 2020-01-13T23:33:13Z

Run Java PreCommit

lukecwik · 2020-01-13T23:33:24Z

Run Dataflow ValidatesRunner

suztomo · 2020-01-14T13:55:48Z

The last two builds seem to have triggered but did not start.

tvalentyn · 2020-01-14T17:03:32Z

Run Dataflow ValidatesRunner

tvalentyn · 2020-01-14T17:03:37Z

Run Java PreCommit

suztomo · 2020-01-14T17:13:36Z

@tvalentyn Now it has "Details" button. Thank you.

lukecwik · 2020-01-14T18:17:06Z

Run Java PreCommit

suztomo · 2020-01-14T18:58:20Z

"Google Cloud Dataflow Runner ValidatesRunner Tests" got stuck at 13:07. With test failures:

13:07:40 org.apache.beam.sdk.testing.PAssertTest > testWindowedIsEqualTo FAILED
13:07:40 java.lang.RuntimeException at PAssertTest.java:347
13:08:04 13:08:04 org.apache.beam.sdk.testing.PAssertTest > testContainsInAnyOrderFalse FAILED
13:08:04 java.lang.AssertionError at PAssertTest.java:468

https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_PR/189/console

Both tests succeed in my local environment. I see it's transient error:

java.lang.RuntimeException: Workflow failed. Causes: Project apache-beam-testing has insufficient quota(s) to execute this workflow with 1 instances in region us-central1. Quota summary (required/available): 1/19670 instances, 1/965 CPUs, 250/352715 disk GB, 0/3191 SSD disk GB, 1/0 instance groups, 1/0 managed instance groups, 1/237 instance templates, 1/876 in-use IP addresses.

https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_PR/189/testReport/junit/org.apache.beam.sdk.testing/PAssertTest/testWindowedIsEqualTo/

This seems a transient error.

suztomo · 2020-01-14T19:33:26Z

Java PreCommit failed:

org.apache.samza.SamzaException: Error opening RocksDB store beamStore at location /tmp/store1

https://builds.apache.org/job/beam_PreCommit_Java_Phrase/lastCompletedBuild/

This seems a transient error.

suztomo · 2020-01-14T21:31:42Z

Run Java PreCommit

lukecwik · 2020-01-14T22:07:01Z

Run Java PreCommit

lukecwik · 2020-01-14T22:15:11Z

Run Dataflow ValidatesRunner

suztomo · 2020-01-15T00:35:00Z

Run Dataflow ValidatesRunner

suztomo · 2020-01-15T02:42:46Z

Dataflow quota error for jobs-per-project quota. Project apache-beam-testing is running 303 jobs.
https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_PR/190/testReport/junit/org.apache.beam.sdk/PipelineTest/testTupleProjectionTransform/

aaltay · 2020-01-15T02:46:31Z

Run Dataflow ValidatesRunner

suztomo · 2020-01-15T02:51:37Z

Thank you, Ahmet.

suztomo · 2020-01-15T05:13:40Z

21 successful checks!

kennknowles · 2020-01-16T17:54:25Z

We've seen this change break some of our own internal use of Beam. It may be that it is not a breaking change except if a user has things pinned in particular ways. But it is common for minor version bumps in libs like these to actually be breaking changes. Be on the lookout for Beam users for whom this is a breaking change; we have not yet determined the fix.

lukecwik · 2020-01-16T18:10:52Z

Specifically, the internal usage was pinning tcnative which was causing:

java.lang.NoClassDefFoundError: io/netty/internal/tcnative/SSLPrivateKeyMethod at io.netty.handler.ssl.SslContext.newClientContextInternal(SslContext.java:824)

updating internal usage to io.netty:netty-tcnative-boringssl-static:2.0.26.Final resolved the issue

suztomo · 2020-01-16T18:30:29Z

@kennknowles @lukecwik Thank you for updates. Glad to hear you resolved the problem.

suztomo added 2 commits January 10, 2020 15:14

Gax and grpc version upgrades
2c482c2

io/google-cloud-platform to have grpc-context
b4bd0bc

suztomo force-pushed the gax_grpc branch from a47935e to b4bd0bcCompare January 10, 2020 20:15

suztomo marked this pull request as ready for review January 10, 2020 20:26

suztomo commented Jan 10, 2020
View reviewed changes

suztomo requested a review from lukecwik January 15, 2020 18:35

lukecwik merged commit a37e5c5 into apache:masterJan 15, 2020

suztomo deleted the gax_grpc branch January 17, 2020 00:57

Uh oh!

Conversation

suztomo commented Jan 10, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which modules use gax and grpc?

Linkage Check

beam-sdks-java-io-google-cloud-platform

beam-sdks-java-extensions-sql-zetasql

beam-sdks-java-extensions-sql-datacatalog

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

Uh oh!

suztomoJan 10, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

suztomoJan 10, 2020

Choose a reason for hiding this comment

Uh oh!

suztomoJan 10, 2020

Choose a reason for hiding this comment

Uh oh!

suztomo commented Jan 10, 2020

Uh oh!

apilloud commented Jan 10, 2020

Uh oh!

apilloud commented Jan 10, 2020

Uh oh!

apilloud commented Jan 10, 2020

Uh oh!

apilloud commented Jan 10, 2020

Uh oh!

apilloud commented Jan 10, 2020

Uh oh!

apilloud commented Jan 10, 2020

Uh oh!

suztomo commented Jan 11, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yifanzou commented Jan 13, 2020

Uh oh!

yifanzou commented Jan 13, 2020

Uh oh!

yifanzou commented Jan 13, 2020

Uh oh!

yifanzou commented Jan 13, 2020

Uh oh!

yifanzou commented Jan 13, 2020

Uh oh!

yifanzou commented Jan 13, 2020

Uh oh!

suztomo commented Jan 13, 2020

Uh oh!

suztomo commented Jan 13, 2020

Uh oh!

suztomo commented Jan 13, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iemejia commented Jan 13, 2020

Uh oh!

suztomo commented Jan 13, 2020

Uh oh!

suztomo commented Jan 13, 2020

Uh oh!

markflyhigh commented Jan 13, 2020

Uh oh!

lukecwik commented Jan 13, 2020

Uh oh!

lukecwik commented Jan 13, 2020

Uh oh!

lukecwik commented Jan 13, 2020

Uh oh!

lukecwik commented Jan 13, 2020

Uh oh!

lukecwik commented Jan 13, 2020

Uh oh!

suztomo commented Jan 14, 2020

Uh oh!

tvalentyn commented Jan 14, 2020

Uh oh!

suztomo commented Jan 10, 2020•
edited
Loading

suztomoJan 10, 2020•
edited
Loading

suztomo commented Jan 11, 2020•
edited
Loading

suztomo commented Jan 13, 2020•
edited
Loading

suztomo commented Jan 14, 2020•
edited
Loading

suztomo commented Jan 14, 2020•
edited
Loading

lukecwik commented Jan 16, 2020•
edited
Loading