[BEAM-9444] Use GCP Libraries BOM for Google Cloud Dependencies by suztomo · Pull Request #11156 · apache/beam

suztomo · 2020-03-18T13:37:37Z

As per Subject: Proposal: Beam to use GCP Libraries BOM, specifying library versions through GCP Libraries BOM.

To use BOM, declaring the BOM dependency with "platform" (Gradle: Importing Maven BOMs)
Removing version part from Google Cloud dependencies that is in the BOM.
- List of Google Cloud libraries in the GCP Libraries BOM
- The versionless coordinates cannot be used for force function in Gradle.
- I found io.grpc:grpc-protobuf and io.grpc:grpc-protobuf-lite are unused.
Maven artifact POM generation logic showing "platform" dependency in dependnecyManagement section.

Linkage Check Result

No change. https://gist.github.com/suztomo/eadcdcac2b465a771cdefe27a29bbbc8

Artifact Version Changes

Resolved artifact version changes as part of this PR:

Unchanged

com.google.api:gax:1.54.0
com.google.api:gax-grpc:1.54.0
com.google.api:api-common:1.8.1
com.google.cloud:google-cloud-bigquerystorage:0.125.0-beta
com.google.cloud:google-cloud-datacatalog:0.32.1
io.grpc:grpc-auth:1.27.2
io.grpc:grpc-core:1.27.2
io.grpc:grpc-context:1.27.2
io.grpc:grpc-grpclb:1.27.2
io.grpc:grpc-netty:1.27.2
io.grpc:grpc-stub:1.27.2
com.google.api.grpc:proto-google-common-protos:1.17.0

Upgrade

com.google.auth:google-auth-library-credentials:0.20.0 from 0.19.0
com.google.auth:google-auth-library-oauth2-http:0.20.0 from 0.19.0
com.google.cloud:google-cloud-core:1.93.0 from 1.92.2
com.google.cloud:google-cloud-core-grpc:1.93.0 from 1.92.2
com.google.cloud:google-cloud-spanner:1.49.2 from 1.49.1
com.google.http-client:google-http-client:1.34.2 from 1.34.0
com.google.http-client:google-http-client-jackson2:1.34.2 from 1.34.0
com.google.http-client:google-http-client-protobuf:1.34.2 from 1.34.0
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:0.90.0 from 0.85.1
com.google.api.grpc:proto-google-cloud-bigtable-v2:1.10.0 from 1.9.1
com.google.api.grpc:proto-google-cloud-datastore-v1:0.85.2 from 0.85.0
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:1.49.2 from 1.49.1

Downgrade

com.google.cloud:google-cloud-bigquery:1.107.0 from 1.108.0 // GCP Libraries BOM 4.3.0 will have 1.108.1
com.google.api.grpc:grpc-google-cloud-pubsub-v1:1.85.0 from 1.85.1 // GCP Librareis BOM 4.3.0 will have 1.85.1
com.google.api.grpc:proto-google-cloud-pubsub-v1:1.85.0 from 1.85.1 // GCP Librareis BOM 4.3.0 will have 1.85.1

Removal

Removed unused fields

io.grpc:grpc-protobuf:1.27.2
io.grpc:grpc-protobuf-lite:1.27.2

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	SDK	Apex	Dataflow	Gearpump	Samza
Go		---	---	---	---
Java
Python		---		---	---
XLang	---	---	---	---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website
Non-portable
Portable	---		---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

iemejia · 2020-03-18T14:48:17Z

bang

iemejia · 2020-03-18T14:48:34Z

Run Java PostCommit

iemejia · 2020-03-18T14:48:40Z

Run Java HadoopFormatIO Performance Test

iemejia · 2020-03-18T14:48:46Z

Run BigQueryIO Streaming Performance Test Java

iemejia · 2020-03-18T14:48:52Z

Run Dataflow ValidatesRunner

iemejia · 2020-03-18T14:48:58Z

Run Spark ValidatesRunner

iemejia · 2020-03-18T14:49:04Z

Run SQL Postcommit

suztomo · 2020-03-18T14:54:06Z

@iemejia Thank you. Sonatype is unhealthy now. I'll check later today.

10:51:05 > Could not get resource 'https://oss.sonatype.org/content/repositories/staging/io/grpc/grpc-core/maven-metadata.xml'.
10:51:05 > Could not GET 'https://oss.sonatype.org/content/repositories/staging/io/grpc/grpc-core/maven-metadata.xml'.
10:51:05 > Read timed out

aaltay · 2020-03-24T22:33:53Z

Run Java PostCommit

aaltay · 2020-03-24T22:34:05Z

Run Java HadoopFormatIO Performance Test

aaltay · 2020-03-24T22:34:26Z

Run BigQueryIO Streaming Performance Test Java

aaltay · 2020-03-24T22:34:43Z

Run Dataflow ValidatesRunner

aaltay · 2020-03-24T22:35:00Z

Run Spark ValidatesRunner

aaltay · 2020-03-24T22:35:14Z

Run SQL Postcommit

aaltay · 2020-03-24T22:35:32Z

Run Java PostCommit

aaltay · 2020-03-27T01:43:07Z

Run Java PostCommit

aaltay · 2020-03-27T01:43:19Z

Run Java HadoopFormatIO Performance Test

aaltay · 2020-03-27T01:43:28Z

Run BigQueryIO Streaming Performance Test Java

aaltay · 2020-03-27T01:43:43Z

Run Dataflow ValidatesRunner

aaltay · 2020-03-27T01:43:52Z

Run Spark ValidatesRunner

aaltay · 2020-03-27T01:43:59Z

Run SQL Postcommit

kamilwu · 2020-03-30T15:07:45Z

Run Dataflow ValidatesRunner

kamilwu · 2020-03-30T15:08:41Z

Run Java PreCommit

suztomo · 2020-03-30T18:05:43Z

:runners:google-cloud-dataflow-java:validatesRunnerLegacyWorkerTest failed: https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_PR/252/

11:12:20 > Task :runners:google-cloud-dataflow-java:validatesRunnerLegacyWorkerTest
11:13:55 11:13:55 org.apache.beam.sdk.transforms.ParDoTest$TimerFamilyTests > testTimerWithMultipleTimerFamilyUnbounded FAILED
11:13:55 java.lang.RuntimeException at ParDoTest.java:4770
11:13:55 Caused by: java.lang.IllegalArgumentException at ParDoTest.java:4770
11:29:11 11:29:11 org.apache.beam.sdk.transforms.ParDoTest$TimerFamilyTests > testTimerFamilyEventTimeUnbounded FAILED
11:29:11 java.lang.RuntimeException at ParDoTest.java:4708
11:29:11 Caused by: java.lang.IllegalArgumentException at ParDoTest.java:4708

It seems other post commit jobs are also failing:
https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/4738/console

08:01:43 > Task :runners:google-cloud-dataflow-java:validatesRunnerLegacyWorkerTest
08:03:14 08:03:14 org.apache.beam.sdk.transforms.ParDoTest$TimerFamilyTests > testTimerWithMultipleTimerFamilyUnbounded FAILED
08:03:14 java.lang.RuntimeException at ParDoTest.java:4770
08:03:14 Caused by: java.lang.IllegalArgumentException at ParDoTest.java:4770
08:16:38 08:16:38 org.apache.beam.sdk.transforms.ParDoTest$TimerFamilyTests > testTimerFamilyEventTimeUnbounded FAILED
08:16:38 java.lang.RuntimeException at ParDoTest.java:4708
08:16:38 Caused by: java.lang.IllegalArgumentException at ParDoTest.java:4708
08:33:31 08:33:31 org.apache.beam.sdk.transforms.FlattenTest > testFlattenPCollectionsThenParDo FAILED
08:33:31 java.lang.RuntimeException at FlattenTest.java:118

Checking which PR started this failure. -> Found #11216 (comment)

chamikaramj

Thanks!

chamikaramj · 2020-04-02T23:43:34Z

+ google_auth_library_oauth2_http : "com.google.auth:google-auth-library-oauth2-http",
+ google_cloud_bigquery : "com.google.cloud:google-cloud-bigquery",
+ google_cloud_bigquery_storage : "com.google.cloud:google-cloud-bigquerystorage",
 google_cloud_bigtable_client_core : "com.google.cloud.bigtable:bigtable-client-core:1.13.0",


Have we confirmed that GCS artifacts not included in this BOM do not conflict with artifacts included in the BOM ?

chamikaramj · 2020-04-02T23:50:47Z

- force project.library.java.values()
+ // Filtering versionless coordinates that depend on BOM
+ def librariesWithVersion = project.library.java.values().findAll { it.split(':').size() > 2 }
+ force librariesWithVersion


Please force findings here just for the record.

suztomo

@chamikaramj PTAL. Added source code comment on force.

suztomo · 2020-04-03T03:21:41Z

- force project.library.java.values()
+ // Filtering versionless coordinates that depend on BOM
+ def librariesWithVersion = project.library.java.values().findAll { it.split(':').size() > 2 }
+ force librariesWithVersion


Added comment with ticket ID BEAM-9542.

suztomo · 2020-04-03T03:25:04Z

+ google_auth_library_oauth2_http : "com.google.auth:google-auth-library-oauth2-http",
+ google_cloud_bigquery : "com.google.cloud:google-cloud-bigquery",
+ google_cloud_bigquery_storage : "com.google.cloud:google-cloud-bigquerystorage",
 google_cloud_bigtable_client_core : "com.google.cloud.bigtable:bigtable-client-core:1.13.0",


Yes, Linkage Checker detects the discrepancy if there's any.
Note that our existing integration tests are not that useful here since they mostly check one GCP service at a time while users may use a combination.
This change is not intended to fix compatibility with users' libraries. It's about how Beam maintains the versions of these Google libraries with ease.

suztomo · 2020-04-27T22:04:52Z

Gradle's updateOfflineRepository task (via https://github.com/mdietrichstein/gradle-offline-dependencies-plugin) does not work with Maven BOMs. The plugin is marked as not maintained. Closing this for now.

suztomo · 2020-10-08T23:05:59Z

 configuration: (configuration.shadowClosure ? 'shadow' : 'compile'), scope: 'compile')
 generateDependenciesFromConfiguration(configuration: 'provided', scope: 'provided')

+ if (!boms.isEmpty()) {


When this Beam module uses a BOM, we need to set dependencyManagent element

suztomo added 9 commits March 17, 2020 21:22

google-cloud-core to BOM
fd6f4ef

More libraries in BOM
6c3ed18

Removed unused variables
fa532dc

Comments
f7490ea

google_cloud_pubsub_v1 to use BOM
5bacaaf

gRPC versions to use BOM
fbeac78

Trying to create dependencyManagement
7931d7e

proto-google-cloud-spanner-admin-database
fec5361

type pom
712954c

probot-autolabelerBot added build dataflow examples extensions gcp google-cloud-platform-core io java runners sql labels Mar 18, 2020

suztomo added 3 commits March 18, 2020 10:20

Format
0174bc4

Format
483600d

Removed versions in protos-
f86ef3d

suztomo added 3 commits March 23, 2020 13:46

Clarified how versions are set
b715356

URL
c040c41

Merge remote-tracking branch 'origin/master' into BEAM-9444-gcp-bom
b4b094e

chamikaramj reviewed Apr 2, 2020
View reviewed changes

suztomo added 2 commits April 2, 2020 23:20

Added comment
d90ab86

Merge remote-tracking branch 'origin/master' into BEAM-9444-gcp-bom
ab124d0

suztomo commented Apr 3, 2020
View reviewed changes

Merge remote-tracking branch 'origin/master' into BEAM-9444-gcp-bom
81a1d3d

This was referenced Apr 10, 2020
[BEAM-9444] rebased version of PR 11156 #11387
Closed
[BEAM-9444] Clean version of PR 11156 (no need to review) #11388
Closed

suztomo closed this Apr 27, 2020

suztomo commented Oct 8, 2020
View reviewed changes

kileys mentioned this pull request Oct 12, 2020
[BEAM-9444] Use GCP BOM to set package versions #13075
Merged
4 tasks

Uh oh!

Conversation

suztomo commented Mar 18, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Linkage Check Result

Artifact Version Changes

Unchanged

Upgrade

Downgrade

Removal

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

Uh oh!

iemejia commented Mar 18, 2020

Uh oh!

iemejia commented Mar 18, 2020

Uh oh!

iemejia commented Mar 18, 2020

Uh oh!

iemejia commented Mar 18, 2020

Uh oh!

iemejia commented Mar 18, 2020

Uh oh!

iemejia commented Mar 18, 2020

Uh oh!

iemejia commented Mar 18, 2020

Uh oh!

suztomo commented Mar 18, 2020

Uh oh!

aaltay commented Mar 24, 2020

Uh oh!

aaltay commented Mar 24, 2020

Uh oh!

aaltay commented Mar 24, 2020

Uh oh!

aaltay commented Mar 24, 2020

Uh oh!

aaltay commented Mar 24, 2020

Uh oh!

aaltay commented Mar 24, 2020

Uh oh!

aaltay commented Mar 24, 2020

Uh oh!

aaltay commented Mar 27, 2020

Uh oh!

aaltay commented Mar 27, 2020

Uh oh!

aaltay commented Mar 27, 2020

Uh oh!

aaltay commented Mar 27, 2020

Uh oh!

aaltay commented Mar 27, 2020

Uh oh!

aaltay commented Mar 27, 2020

Uh oh!

kamilwu commented Mar 30, 2020

Uh oh!

kamilwu commented Mar 30, 2020

Uh oh!

suztomo commented Mar 30, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chamikaramj left a comment

Choose a reason for hiding this comment

Uh oh!

chamikaramjApr 2, 2020

Choose a reason for hiding this comment

Uh oh!

chamikaramjApr 2, 2020

Choose a reason for hiding this comment

Uh oh!

suztomo left a comment

Choose a reason for hiding this comment

Uh oh!

suztomoApr 3, 2020

Choose a reason for hiding this comment

Uh oh!

suztomoApr 3, 2020

Choose a reason for hiding this comment

Uh oh!

suztomo commented Mar 18, 2020•
edited
Loading

suztomo commented Mar 30, 2020•
edited
Loading