[BEAM-11648] In preparation for new BigQuery sink, update GCP BOM by reuvenlax · Pull Request #13765 · apache/beam

reuvenlax · 2021-01-16T00:26:51Z

No description provided.

reuvenlax · 2021-01-16T00:27:11Z

run dataflow validatesrunner

chamikaramj · 2021-01-16T00:34:22Z

Probably run the linkage checker ? https://cwiki.apache.org/confluence/display/BEAM/Dependency+Upgrades

codecov · 2021-01-16T00:54:48Z

Codecov Report

Merging #13765 (5136c4e) into master (88190dc) will decrease coverage by 0.01%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #13765 +/- ##
==========================================
- Coverage 82.87% 82.86% -0.02% 
==========================================
Files 466 466 Lines 57613 57613 ==========================================
- Hits 47749 47741 -8 - Misses 9864 9872 +8

Impacted Files	Coverage Δ
...ks/python/apache_beam/runners/worker/data_plane.py	`89.52% <0.00%> (-1.80%)`	⬇️
sdks/python/apache_beam/internal/metrics/metric.py	`86.45% <0.00%> (-1.05%)`	⬇️
sdks/python/apache_beam/io/localfilesystem.py	`91.66% <0.00%> (-0.76%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6e03cfd...5136c4e. Read the comment docs.

reuvenlax · 2021-01-19T17:26:27Z

@chamikaramj I'm not sure how to run this tool. If I run the suggested syntax, e.g.

sdks/java/build-tools/beam-linkage-check.sh "beam-runners-google-cloud-dataflow-java"

I get

Usage: sdks/java/build-tools/beam-linkage-check.sh [artifact lists]

chamikaramj · 2021-01-19T17:28:46Z

@suztomo or @kennknowles might be able to help. Does linkage checker work for BOM updates ?

reuvenlax · 2021-01-19T17:29:32Z

Run Java PreCommit

suztomo · 2021-01-19T17:56:11Z

(Yes, beam-linkage-check.sh is a good tool when upgrading the BOM) Would you try this?

sdks/java/build-tools/beam-linkage-check.sh origin/master reuvenlax/update_bom

(This may take few ten minutes) This command checks the dependency compatibility of beam-sdks-java-core, beam-sdks-java-io-google-cloud-platform, beam-runners-google-cloud-dataflow-java, beam-sdks-java-io-hadoop-format. If you want to exclude the rest, you can run:

sdks/java/build-tools/beam-linkage-check.sh origin/master reuvenlax/update_bom beam-runners-google-cloud-dataflow-java

I'll update the script usage.

suztomo · 2021-01-19T18:00:54Z

Updated this PR for fixing the usage in the comment: https://github.com/apache/beam/pull/13757/files#diff-24374dd1ad59dfdae51815cec95a1f488e7a2f527a7bdb8f064d79244ec11430R23

suztomo · 2021-01-19T21:22:06Z

@kileys The work of making beam-linkage-check.sh as a Jenkins Job would be beneficial here. Looking forward.

kennknowles · 2021-01-19T23:06:17Z

Also adding @kileys

reuvenlax · 2021-01-27T00:16:15Z

@suztomo If I try to run that command, it tries to execute sdks:java:core:javadoc. This consistently fails with a NullPointerException. Is this a known issue?

javadoc: error - An internal exception has occurred.
(java.lang.NullPointerException)
Please file a bug against the javadoc tool via the Java bug reporting page
(http://bugreport.java.com) after checking the Bug Database (http://bugs.java.com)
for duplicates. Include error messages and the following diagnostic in your report. Thank you.
java.lang.NullPointerException
at jdk.compiler/com.sun.tools.javac.code.Types.erasure(Types.java:2340)
at jdk.compiler/com.sun.tools.javac.code.Types$14.visitTypeVar(Types.java:2398)
at jdk.compiler/com.sun.tools.javac.code.Types$14.visitTypeVar(Types.java:2348)
at jdk.compiler/com.sun.tools.javac.code.Type$TypeVar.accept(Type.java:1654)
at jdk.compiler/com.sun.tools.javac.code.Types$DefaultTypeVisitor.visit(Types.java:4857)
at jdk.compiler/com.sun.tools.javac.code.Type$StructuralTypeMapping.visitMethodType(Type.java:282)
at jdk.compiler/com.sun.tools.javac.code.Type$StructuralTypeMapping.visitMethodType(Type.java:230)
at jdk.compiler/com.sun.tools.javac.code.Type$MethodType.accept(Type.java:1448)
at jdk.compiler/com.sun.tools.javac.code.Types$DefaultTypeVisitor.visit(Types.java:4857)
at jdk.compiler/com.sun.tools.javac.code.Types.erasure(Types.java:2343)
at jdk.compiler/com.sun.tools.javac.code.Types.erasure(Types.java:2329)
at jdk.compiler/com.sun.tools.javac.code.Types.isSubSignature(Types.java:2748)
at jdk.compiler/com.sun.tools.javac.code.Types.isSubSignature(Types.java:2744)
at jdk.compiler/com.sun.tools.javac.code.Symbol$MethodSymbol.overrides(Symbol.java:1820)
at jdk.compiler/com.sun.tools.javac.code.Symbol$MethodSymbol.overrides(Symbol.java:1792)
at jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.WorkArounds.overrides(WorkArounds.java:325)
at jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.util.Utils.overriddenMethod(Utils.java:810)
at jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.util.DocFinder.search(DocFinder.java:267)
at jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.builders.MemberSummaryBuilder.buildSummary(MemberSummaryBuilder.java:322)
at jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.builders.MemberSummaryBuilder.addSummary(MemberSummaryBuilder.java:495)
at jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.builders.MemberSummaryBuilder.buildMethodsSummary(MemberSummaryBuilder.java:288)
at jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.builders.MemberSummaryBuilder$1.build(MemberSummaryBuilder.java:112)
at jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.builders.ClassBuilder.buildMemberSummary(ClassBuilder.java:324)
at jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.builders.ClassBuilder.buildClassDoc(ClassBuilder.java:144)
at jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.builders.ClassBuilder.build(ClassBuilder.java:120)
at jdk.javadoc/jdk.javadoc.internal.doclets.formats.html.HtmlDoclet.generateClassFiles(HtmlDoclet.java:264)
at jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.AbstractDoclet.generateClassFiles(AbstractDoclet.java:286)
at jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.AbstractDoclet.generateClassFiles(AbstractDoclet.java:268)
at jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.AbstractDoclet.startGeneration(AbstractDoclet.java:207)
at jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.AbstractDoclet.run(AbstractDoclet.java:114)
at jdk.javadoc/jdk.javadoc.doclet.StandardDoclet.run(StandardDoclet.java:72)
at jdk.javadoc/jdk.javadoc.internal.tool.Start.parseAndExecute(Start.java:582)
at jdk.javadoc/jdk.javadoc.internal.tool.Start.begin(Start.java:431)
at jdk.javadoc/jdk.javadoc.internal.tool.Start.begin(Start.java:344)
at jdk.javadoc/jdk.javadoc.internal.tool.Main.execute(Main.java:63)
at jdk.javadoc/jdk.javadoc.internal.tool.Main.main(Main.java:52)

suztomo · 2021-01-27T00:19:10Z

No, that's new to me. Linkage Checker shouldn't have any effect to javadoc generation. I'll try to reproduce that problem.

Do you use Java 8?

reuvenlax · 2021-01-27T00:27:49Z

Ah good point. My default install is java 11. Let me fix my JAVA_HOME and see if it fixes things.

…

On Tue, Jan 26, 2021 at 4:19 PM Tomo Suzuki ***@***.***> wrote: No, that's new to me. Linkage Checker shouldn't have any effect to javadoc generation. I'll try to reproduce that problem. Do you use Java 8? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#13765 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFAYJVLIFD325MFOOHEDKC3S35LYZANCNFSM4WEZQAAA> .

reuvenlax · 2021-01-27T01:58:31Z

@suztomo Now I get the following

Execution failed for task ':sdks:java:io:hcatalog:compileJava'.

Could not resolve all files for configuration ':sdks:java:io:hcatalog:provided'.
Could not resolve org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde.
Required by:
project :sdks:java:io:hcatalog > org.apache.hive:hive-exec:2.1.0 > org.apache.calcite:calcite-core:1.6.0
> Could not resolve org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde.
> Could not get resource 'https://repo.spring.io/plugins-release/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom'.
> Could not GET 'https://repo.spring.io/plugins-release/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom'. Received status code 401 from server: Unauthorized

chamikaramj · 2021-01-27T02:02:18Z

That seems like https://issues.apache.org/jira/browse/BEAM-11689. Can you sync and retry ?

reuvenlax · 2021-01-27T16:25:58Z

@chamikaramj I rebased, but I'm still getting that failure.

reuvenlax · 2021-01-27T16:28:29Z

Could not resolve all files for configuration ':sdks:java:io:hcatalog:provided'.
Could not resolve org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde.
Required by:
project :sdks:java:io:hcatalog > org.apache.hive:hive-exec:2.1.0 > org.apache.calcite:calcite-core:1.6.0
> Could not resolve org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde.
> Could not get resource 'https://repo.spring.io/plugins-release/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom'.
> Could not HEAD 'https://repo.spring.io/plugins-release/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom'. Received status code 401 from server: Unauthorized

ibzib · 2021-01-27T20:20:35Z

Could not resolve all files for configuration ':sdks:java:io:hcatalog:provided'.
Could not resolve org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde.
Required by:
project :sdks:java:io:hcatalog > org.apache.hive:hive-exec:2.1.0 > org.apache.calcite:calcite-core:1.6.0
Could not resolve org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde.
Could not get resource 'https://repo.spring.io/plugins-release/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom'.
Could not HEAD 'https://repo.spring.io/plugins-release/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom'. Received status code 401 from server: Unauthorized

Thanks for reporting this @reuvenlax , #13823 fixes this error (note that both branches compared by the linkage checker need to patch the fix for it to work).

reuvenlax · 2021-01-27T22:08:03Z

@ibzib unfortunately I'm still getting this error!

suztomo · 2021-01-27T22:10:42Z

@reuvenlax How about removing the local Maven repository (~/.m2/repository) and retry?

reuvenlax · 2021-01-27T22:38:54Z

@suztomo Unfortunately that did not help

suztomo · 2021-01-27T22:52:58Z

I see. I'll try your branch later today.

ibzib · 2021-01-27T23:07:20Z

@reuvenlax I got this to work using the following command. Can you try using commit hashes directly to make sure we're on the same page?

sdks/java/build-tools/beam-linkage-check.sh 870cc2253544bc7307bb11c3f3957cfdc6759318 576da5b1a61b80f81c2087299c01028744a9480a

Where 870cc22 is my patch and 576da5b1a61b80f81c2087299c01028744a9480a is your changes.

reuvenlax · 2021-01-28T00:49:29Z

The problem was that I had not yet pushed my client to github. after pushing, the error went away.

@suztomo I'm now getting the below errors. I'm not entirely sure how to interpret the output of this tool. Most of the errors seem related to grpc. Does this mean that some other component is pulling in a conflicting version of grpc?

com.google.cloud.tools.opensource.classpath.LinkageCheckResultException: Found 14 linkage errors
io.grpc.internal.ManagedChannelImpl (io.grpc:grpc-core:1.35.0)
at com.google.cloud.tools.opensource.classpath.LinkageCheckerMain.main(LinkageCheckerMain.java:76)
Cause:
Dependency conflict: io.grpc:grpc-api:1.34.1 does not define io.grpc.InternalConfigSelector$Result's method getInterceptor() but io.grpc:grpc-api:1.35.0 defines it.
selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / io.grpc:grpc-api:1.34.1 (compile)
unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-core:1.35.0 (compile) / io.grpc:grpc-api:1.35.0 (compile)
(io.grpc:grpc-api:1.34.1) io.grpc.LoadBalancer$Helper's method createResolvingOobChannelBuilder(String, io.grpc.ChannelCredentials) is not found;
referenced by 1 class file
io.grpc.util.ForwardingLoadBalancerHelper (io.grpc:grpc-core:1.35.0)
Cause:
Dependency conflict: io.grpc:grpc-api:1.34.1 does not define io.grpc.LoadBalancer$Helper's method createResolvingOobChannelBuilder(String, io.grpc.ChannelCredentials) but io.grpc:grpc-api:1.35.0 defines it.
selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / io.grpc:grpc-api:1.34.1 (compile)
unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-core:1.35.0 (compile) / io.grpc:grpc-api:1.35.0 (compile)
(io.grpc:grpc-api:1.34.1) io.grpc.LoadBalancer$Helper's method getChannelCredentials() is not found;
referenced by 1 class file
io.grpc.util.ForwardingLoadBalancerHelper (io.grpc:grpc-core:1.35.0)
Cause:
Dependency conflict: io.grpc:grpc-api:1.34.1 does not define io.grpc.LoadBalancer$Helper's method getChannelCredentials() but io.grpc:grpc-api:1.35.0 defines it.
selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / io.grpc:grpc-api:1.34.1 (compile)
unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-core:1.35.0 (compile) / io.grpc:grpc-api:1.35.0 (compile)
(io.grpc:grpc-api:1.34.1) io.grpc.LoadBalancer$Helper's method getUnsafeChannelCredentials() is not found;
referenced by 1 class file
io.grpc.util.ForwardingLoadBalancerHelper (io.grpc:grpc-core:1.35.0)
Cause:
Dependency conflict: io.grpc:grpc-api:1.34.1 does not define io.grpc.LoadBalancer$Helper's method getUnsafeChannelCredentials() but io.grpc:grpc-api:1.35.0 defines it.
selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / io.grpc:grpc-api:1.34.1 (compile)
unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-core:1.35.0 (compile) / io.grpc:grpc-api:1.35.0 (compile)
(io.netty:netty-buffer:4.1.51.Final) io.netty.buffer.PooledByteBufAllocator's method (boolean, int, int, int, int, int, int, boolean) is not found;
referenced by 1 class file
io.grpc.netty.Utils (io.grpc:grpc-netty:1.35.0)
Cause:
Dependency conflict: io.netty:netty-buffer:4.1.51.Final does not define io.netty.buffer.PooledByteBufAllocator's method (boolean, int, int, int, int, int, int, boolean) but io.netty:netty-buffer:4.1.52.Final defines it.
selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.netty:netty-handler:4.1.51.Final (compile) / io.netty:netty-buffer:4.1.51.Final (compile)
unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-netty:1.35.0 (compile) / io.netty:netty-codec-http2:4.1.52.Final (compile) / io.netty:netty-buffer:4.1.52.Final (compile)
(com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10) com.google.api.services.bigquery.model.Dataset$Access's method getRoutine() is not found;
referenced by 2 class files
com.google.cloud.bigquery.Acl (com.google.cloud:google-cloud-bigquery:1.126.3)
com.google.cloud.bigquery.DatasetInfo (com.google.cloud:google-cloud-bigquery:1.126.3)
Cause:
Dependency conflict: com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 does not define com.google.api.services.bigquery.model.Dataset$Access's method getRoutine() but com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 defines it.
selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 (compile)
unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / com.google.cloud:google-cloud-bigquery:1.126.3 (compile) / com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 (compile)
Dependency conflict: com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 does not define com.google.api.services.bigquery.model.Dataset$Access's method getRoutine() but com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 defines it.
selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 (compile)
unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / com.google.cloud:google-cloud-bigquery:1.126.3 (compile) / com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 (compile)
(com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10) com.google.api.services.bigquery.model.Dataset$Access's method setRoutine(com.google.api.services.bigquery.model.RoutineReference) is not found;
referenced by 1 class file
com.google.cloud.bigquery.Acl (com.google.cloud:google-cloud-bigquery:1.126.3)
Cause:
Dependency conflict: com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 does not define com.google.api.services.bigquery.model.Dataset$Access's method setRoutine(com.google.api.services.bigquery.model.RoutineReference) but com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 defines it.
selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 (compile)
unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / com.google.cloud:google-cloud-bigquery:1.126.3 (compile) / com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 (compile)
(com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10) com.google.api.services.bigquery.model.HivePartitioningOptions's method setRequirePartitionFilter(Boolean) is not found;
referenced by 1 class file
com.google.cloud.bigquery.HivePartitioningOptions (com.google.cloud:google-cloud-bigquery:1.126.3)
Cause:
Dependency conflict: com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 does not define com.google.api.services.bigquery.model.HivePartitioningOptions's method setRequirePartitionFilter(Boolean) but com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 defines it.
selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 (compile)
unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / com.google.cloud:google-cloud-bigquery:1.126.3 (compile) / com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 (compile)
(com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10) com.google.api.services.bigquery.model.HivePartitioningOptions's method getRequirePartitionFilter() is not found;
referenced by 1 class file
com.google.cloud.bigquery.HivePartitioningOptions (com.google.cloud:google-cloud-bigquery:1.126.3)
Cause:
Dependency conflict: com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 does not define com.google.api.services.bigquery.model.HivePartitioningOptions's method getRequirePartitionFilter() but com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 defines it.
selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 (compile)
unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / com.google.cloud:google-cloud-bigquery:1.126.3 (compile) / com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 (compile)
(io.netty:netty-buffer:4.1.51.Final) io.netty.buffer.ByteBufUtil's method ensureAccessible(io.netty.buffer.ByteBuf) is not found;
referenced by 2 class files
io.netty.handler.codec.http2.DefaultHttp2DataFrame (io.netty:netty-codec-http2:4.1.52.Final)
io.netty.handler.codec.spdy.DefaultSpdyDataFrame (io.netty:netty-codec-http:4.1.52.Final)
Cause:
Dependency conflict: io.netty:netty-buffer:4.1.51.Final does not define io.netty.buffer.ByteBufUtil's method ensureAccessible(io.netty.buffer.ByteBuf) but io.netty:netty-buffer:4.1.52.Final defines it.
selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.netty:netty-handler:4.1.51.Final (compile) / io.netty:netty-buffer:4.1.51.Final (compile)
unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-netty:1.35.0 (compile) / io.netty:netty-codec-http2:4.1.52.Final (compile) / io.netty:netty-buffer:4.1.52.Final (compile)
Dependency conflict: io.netty:netty-buffer:4.1.51.Final does not define io.netty.buffer.ByteBufUtil's method ensureAccessible(io.netty.buffer.ByteBuf) but io.netty:netty-buffer:4.1.52.Final defines it.
selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.netty:netty-handler:4.1.51.Final (compile) / io.netty:netty-buffer:4.1.51.Final (compile)
unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-netty:1.35.0 (compile) / io.netty:netty-codec-http2:4.1.52.Final (compile) / io.netty:netty-codec-http:4.1.52.Final (compile) / io.netty:netty-buffer:4.1.52.Final (compile)
(io.netty:netty-buffer:4.1.51.Final) io.netty.buffer.ByteBufUtil's method isAccessible(io.netty.buffer.ByteBuf) is not found;
referenced by 2 class files
io.netty.handler.codec.http.DefaultFullHttpResponse (io.netty:netty-codec-http:4.1.52.Final)
io.netty.handler.codec.http.DefaultFullHttpRequest (io.netty:netty-codec-http:4.1.52.Final)
Cause:
Dependency conflict: io.netty:netty-buffer:4.1.51.Final does not define io.netty.buffer.ByteBufUtil's method isAccessible(io.netty.buffer.ByteBuf) but io.netty:netty-buffer:4.1.52.Final defines it.
selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.netty:netty-handler:4.1.51.Final (compile) / io.netty:netty-buffer:4.1.51.Final (compile)
unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-netty:1.35.0 (compile) / io.netty:netty-codec-http2:4.1.52.Final (compile) / io.netty:netty-codec-http:4.1.52.Final (compile) / io.netty:netty-buffer:4.1.52.Final (compile)
Dependency conflict: io.netty:netty-buffer:4.1.51.Final does not define io.netty.buffer.ByteBufUtil's method isAccessible(io.netty.buffer.ByteBuf) but io.netty:netty-buffer:4.1.52.Final defines it.
selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.netty:netty-handler:4.1.51.Final (compile) / io.netty:netty-buffer:4.1.51.Final (compile)
unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-netty:1.35.0 (compile) / io.netty:netty-codec-http2:4.1.52.Final (compile) / io.netty:netty-codec-http:4.1.52.Final (compile) / io.netty:netty-buffer:4.1.52.Final (compile)
Problematic artifacts in the dependency tree:
io.grpc:grpc-api:1.34.1 is at:
org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / io.grpc:grpc-api:1.34.1 (compile)
and 2 other dependency paths.
io.grpc:grpc-core:1.35.0 is at:
org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-core:1.35.0 (compile)
and 5 other dependency paths.
io.netty:netty-buffer:4.1.51.Final is at:
org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.netty:netty-handler:4.1.51.Final (compile) / io.netty:netty-buffer:4.1.51.Final (compile)
and 3 other dependency paths.
io.grpc:grpc-netty:1.35.0 is at:
org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-netty:1.35.0 (compile)
com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 is at:
org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 (compile)
com.google.cloud:google-cloud-bigquery:1.126.3 is at:
org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / com.google.cloud:google-cloud-bigquery:1.126.3 (compile)
io.netty:netty-codec-http2:4.1.52.Final is at:
org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-netty:1.35.0 (compile) / io.netty:netty-codec-http2:4.1.52.Final (compile)
io.netty:netty-codec-http:4.1.52.Final is at:
org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-netty:1.35.0 (compile) / io.netty:netty-codec-http2:4.1.52.Final (compile) / io.netty:netty-codec-http:4.1.52.Final (compile)

suztomo · 2021-01-28T01:36:36Z

A Maven project that depends on "org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT" (your branch) would select the following 2 artifacts in its class path:

"io.grpc:grpc-api:1.34.1"
(from org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / io.grpc:grpc-api:1.34.1 (compile))
"io.grpc:grpc-core:1.35.0"
(from org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-core:1.35.0 (compile) / io.grpc:grpc-api:1.35.0 (compile))

These two artifacts are not compatible with each other.

Possible Solutions

A solution would be "beam-sdks-java-io-google-cloud-platform" module declaring grpc-api dependency, even though the module does not touch the class in grpc-api.

Another solution is that Beam users to use the newly-created Beam GCP BOM (#13737). This pins the transitive dependencies to the version listed in the BOM (1.35.0 for the gRPC libraries).

suztomo · 2021-01-28T02:24:27Z

The netty's problem is easy one.

(io.netty:netty-buffer:4.1.51.Final) io.netty.buffer.PooledByteBufAllocator's method <init>(boolean, int, int, int, int, int, int, boolean) is not found;
referenced by 1 class file
io.grpc.netty.Utils (io.grpc:grpc-netty:1.35.0)
Cause:
Dependency conflict: io.netty:netty-buffer:4.1.51.Final does not define io.netty.buffer.PooledByteBufAllocator's method <init>(boolean, int, int, int, int, int, int, boolean) but io.netty:netty-buffer:4.1.52.Final defines it.
selected: org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.29.0-SNAPSHOT / io.netty:netty-handler:4.1.51.Final (compile) / io.netty:netty-buffer:4.1.51.Final (compile)
unselected: org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.29.0-SNAPSHOT / io.grpc:grpc-netty:1.35.0 (compile) / io.netty:netty-codec-http2:4.1.52.Final (compile) / io.netty:netty-buffer:4.1.52.Final (compile)

"io.grpc:grpc-netty:1.35.0" is not compatible with "io.netty:netty-buffer:4.1.51.Final". Just upgrade the netty_version in BeamModulePlugin.groovy to "io.netty:netty-buffer:4.1.52.Final"

reuvenlax · 2021-01-28T04:06:10Z

Do you know why this would happen though? Is com.google.cloud:google-cloud-bigquerystorage bringing in the old version of gRPC? We're pulling that in through the BOM though, so I assumed that it would have the same gRPC version that the BOM depended on.

…

On Wed, Jan 27, 2021 at 5:36 PM Tomo Suzuki ***@***.***> wrote: A Maven project that depends on "org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT" (your branch) would select the following 2 artifacts in its class path: - "io.grpc:grpc-api:1.34.1" (from org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / io.grpc:grpc-api:1.34.1 (compile)) - "io.grpc:grpc-core:1.35.0" (from org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-core:1.35.0 (compile) / io.grpc:grpc-api:1.35.0 (compile)) These two artifacts are not compatible with each other. Possible Solutions A solution would be "beam-sdks-java-io-google-cloud-platform" module declaring grpc-api dependency, even though the module does not touch the class in grpc-api. Another solution is that Beam users to use the newly-created Beam GCP BOM ( #13737 <#13737>). This pins the transitive dependencies to the version listed in the BOM (1.35.0 for the gRPC libraries). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#13765 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFAYJVIC7QAT6JUTA6EV35LS4C5TFANCNFSM4WEZQAAA> .

reuvenlax · 2021-02-09T22:05:42Z

@suztomo I think it's saying that there's a conflict between google-api-services-bigquery and google-cloud-bigquerystorage, correct? Any ideas how to resolve this?

suztomo · 2021-02-09T22:24:18Z

Yes, I agree that there's a conflict between google-api-services-bigquery and google-cloud-bigquerystorage.

I see com.google.cloud:google-cloud-bigquery:1.126.3 uses "com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0".

Would you try upgrading the version of google-api-services-bigquery in BeamModulePlugin.groovy line 512? https://github.com/apache/beam/blob/705649d/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L512

google_api_services_bigquery : "com.google.apis:google-api-services-bigquery:v2-rev20200719-$google_clients_version",

(I would try "com.google.apis:google-api-services-bigquery:v2-rev20201030-$google_clients_version")

reuvenlax · 2021-02-09T23:52:00Z

@suztomo @chamikaramj succeded! No new linkage errors

chamikaramj · 2021-02-10T03:14:05Z

Great. Thanks.

LGTM

chamikaramj · 2021-02-10T03:14:19Z

Run SQL_Java11 PreCommit

chamikaramj · 2021-02-10T03:14:58Z

Retest this please

chamikaramj · 2021-02-10T03:20:12Z

Run Java PostCommit

chamikaramj · 2021-02-10T03:20:55Z

Run Dataflow ValidatesRunner

reuvenlax · 2021-02-10T03:45:58Z

@chamikaramj @kennknowles

:sdks:java:testing:nexmark:analyzeClassesDependencies is now failing with

No signature of method: ca.cutterslade.gradle.analyze.AnalyzeDependenciesTask$_action_closure3$_closure8.doCall() is applicable for argument types: (org.gradle.api.internal.artifacts.PreResolvedResolvableArtifact) values: [auto-value-annotations-1.7.4.jar (com.google.auto.value:auto-value-annotations:1.7.4)]
Possible solutions: findAll(), findAll(), isCase(java.lang.Object), isCase(java.lang.Object)

Is this a known issue? I don't believe that updating GCP deps should affect Nexmark

kennknowles · 2021-02-10T03:48:19Z

Since this fixes an independent bug, would you split it into its own commit?

kennknowles · 2021-02-10T03:50:00Z

That looks like it is in the IWYU dependency checker. It seems to occur on every module.

kennknowles · 2021-02-10T03:51:33Z

Run Java PreCommit

kennknowles · 2021-02-10T03:52:04Z

I'm curious if it is just a bad error message from an outage of the dep download. That's a totally random guess. No idea.

ibzib · 2021-02-10T05:36:57Z

@chamikaramj @kennknowles
:sdks:java:testing:nexmark:analyzeClassesDependencies is now failing with
No signature of method: ca.cutterslade.gradle.analyze.AnalyzeDependenciesTask$_action_closure3$_closure8.doCall() is applicable for argument types: (org.gradle.api.internal.artifacts.PreResolvedResolvableArtifact) values: [auto-value-annotations-1.7.4.jar (com.google.auto.value:auto-value-annotations:1.7.4)]
Possible solutions: findAll(), findAll(), isCase(java.lang.Object), isCase(java.lang.Object)
Is this a known issue? I don't believe that updating GCP deps should affect Nexmark

The bad error message is from a bug in the IWYU plugin (which should be fixed in the next release of the plugin): gradle-dependency-analyze/gradle-dependency-analyze#125

It looks like upgrading the bom upgrades auto-value-annotations from 1.7.2 to 1.7.4, so you need to sync the version here:

beam/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

Line 425 in bc1569c

def autovalue_version ="1.7.2"

kennknowles · 2021-02-10T17:05:39Z

Looks like the gradle analyze plugin was released a few hours after your comment, with that fix. Let's upgrade that in a separate commit.

reuvenlax · 2021-02-11T03:03:47Z

Run Java PreCommit

reuvenlax · 2021-02-11T04:11:15Z

Run Java_Examples_Dataflow PreCommit

reuvenlax · 2021-02-11T05:28:04Z

@kennknowles @chamikaramj any idea why Java_Examples_Dataflow is failing?

reuvenlax · 2021-02-11T05:28:11Z

Run Java_Examples_Dataflow PreCommit

TheNeuralBit · 2021-02-12T01:18:17Z

Unfortunately it looks like this broke the Java PostCommit, SpannerReadIT is failing: BEAM-11805

[BEAM-11805] Revert version updates from #13765

reuvenlax requested a review from chamikaramj January 16, 2021 00:26

reuvenlax force-pushed the update_bom branch from ff3bd39 to 576da5bCompare January 26, 2021 23:59

reuvenlax force-pushed the update_bom branch from 576da5b to 52bde06Compare January 27, 2021 23:48

reuvenlax force-pushed the update_bom branch from 3dbf340 to d561e56Compare February 9, 2021 23:51

kennknowles approved these changes Feb 10, 2021
View reviewed changes

kennknowles mentioned this pull request Feb 10, 2021
[BEAM-11205] bump com.google.cloud:libraries-bom to 16.3.0 #13937
Closed
4 tasks

reuvenlax added 2 commits February 10, 2021 17:25

update versions
e0ba4dc

update autovalue version
5136c4e

reuvenlax force-pushed the update_bom branch from d561e56 to 5136c4eCompare February 11, 2021 02:38

reuvenlax merged commit a5750ab into apache:masterFeb 11, 2021

TheNeuralBit mentioned this pull request Feb 12, 2021
[BEAM-11805] Revert version updates from #13765 #13972
Merged

TheNeuralBit added a commit that referenced this pull request Feb 13, 2021
[BEAM-11805] Revert version updates from#13765(#13972)
e092703
[BEAM-11805] Revert version updates from #13765

suztomo mentioned this pull request Feb 16, 2021
[BEAM-11770] bump google-cloud-pubsub to 1.110.3(google_cloud_platform_libraries_bom ) #13994
Merged
4 tasks

allenpradeep mentioned this pull request Feb 17, 2021
[BEAM-11805] Replace user-agent for spanner #13990
Merged
4 tasks

Uh oh!

Conversation

reuvenlax commented Jan 16, 2021

Uh oh!

reuvenlax commented Jan 16, 2021

Uh oh!

chamikaramj commented Jan 16, 2021

Uh oh!

codecovBot commented Jan 16, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

reuvenlax commented Jan 19, 2021

Uh oh!

chamikaramj commented Jan 19, 2021

Uh oh!

reuvenlax commented Jan 19, 2021

Uh oh!

suztomo commented Jan 19, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

suztomo commented Jan 19, 2021

Uh oh!

suztomo commented Jan 19, 2021

Uh oh!

kennknowles commented Jan 19, 2021

Uh oh!

reuvenlax commented Jan 27, 2021

Uh oh!

suztomo commented Jan 27, 2021

Uh oh!

reuvenlax commented Jan 27, 2021 via email

Uh oh!

reuvenlax commented Jan 27, 2021

Uh oh!

chamikaramj commented Jan 27, 2021

Uh oh!

reuvenlax commented Jan 27, 2021

Uh oh!

reuvenlax commented Jan 27, 2021

Uh oh!

ibzib commented Jan 27, 2021

Uh oh!

reuvenlax commented Jan 27, 2021

Uh oh!

suztomo commented Jan 27, 2021

Uh oh!

reuvenlax commented Jan 27, 2021

Uh oh!

suztomo commented Jan 27, 2021

Uh oh!

ibzib commented Jan 27, 2021

Uh oh!

reuvenlax commented Jan 28, 2021

Uh oh!

suztomo commented Jan 28, 2021

Possible Solutions

Uh oh!

suztomo commented Jan 28, 2021

Uh oh!

reuvenlax commented Jan 28, 2021 via email

Uh oh!

reuvenlax commented Feb 9, 2021

Uh oh!

suztomo commented Feb 9, 2021

Uh oh!

reuvenlax commented Feb 9, 2021

Uh oh!

chamikaramj commented Feb 10, 2021

Uh oh!

chamikaramj commented Feb 10, 2021

Uh oh!

chamikaramj commented Feb 10, 2021

Uh oh!

chamikaramj commented Feb 10, 2021

Uh oh!

chamikaramj commented Feb 10, 2021

Uh oh!

reuvenlax commented Feb 10, 2021

Uh oh!

codecovBot commented Jan 16, 2021•
edited
Loading

suztomo commented Jan 19, 2021•
edited
Loading