BigQuery: adds models API implementation. by shollyman · Pull Request #5021 · googleapis/google-cloud-java

shollyman · 2019-04-25T19:41:37Z

BigQuery: adds models API implementation.

This change adds support for BigQuery ML models as an API resource independent from the previous
ML-As-A-Table variant that existed during beta. It mirrors much of the interactions with Tables, so there's a corresponding ModelId for encapsulating the model reference, ModelInfo for the model metadata, and Model which has service awareness and can be invoked for operations like get/reload.

This initial change does NOT include the extended statistical information from models related to training runs, nor does it include ML feature and labelling column information. That will come in a subsequent change. This change exposes the model type (e.g. LINEAR_REGRESSION), and the ability to do operations like mutate the description/friendlyname/labels.

I've also marked one of the BigQuery methods deprecated due to ambiguous type signature.
boolean delete(String datasetId, String tableId);

When tables were the only kind of resource in a dataset, being able to specify the table via a string was sufficient. However, now that we have independent resource collection kinds within a dataset container, the signature for deletion needs to use more strongly qualified identifiers.

REVIEWER NOTE
I've overridden pom.xml temporarily to pin to an older version of google-api-services-bigquery which will allow testing. The BigQuery service is rolling out the changes necessary so that this no longer needs pinning, but this change is not appropriate for submission until that is complete.

tswast · 2019-04-25T20:06:52Z

+ * ModelInfo modelInfo = beforeModel.toBuilder()
+ * .setDescription(newDescription)
+ * .build();
+ * Model afterModel = bigquery.update(moddelInfo);


Typo: moddel
Note: We have a little program at https://github.com/googleapis/google-cloud-java/blob/master/utilities/snippets.go for copying code samples into JavaDocs, if you want to try that out.

Fixed the typo. Inclined to add the snippets in a followup and redo the instrumentation then, unless you want this change to get even bigger.

tswast · 2019-04-25T20:14:33Z

+ break;
+ }
+ }
+ assertTrue(found);


You sure you want to assert this since listing resources is eventually consistent? Could be flakey.

Good news, everyone! This should be strongly consistent, per conversation with one of the backend engineers who works on the metadata layer.

tswast · 2019-05-01T21:50:58Z

I've also marked one of the BigQuery methods deprecated due to ambiguous type signature.
boolean delete(String datasetId, String tableId);

Thought: Should this be a separate PR so that it shows up nicely in the changelog?

tswast · 2019-05-01T22:24:21Z

+ LABELS("labels"),
+ LAST_MODIFIED_TIME("lastModifiedTime"),
+ LOCATION("location"),
+ MODEL_REFERENCE("modelReference"),


I see model type in the Model class, but not here. At first I thought you were waiting for the enum from Apiary, but I guess not?

I called it TYPE, but perhaps MODEL_TYPE is better for the enum?

Oh, you're right. TYPE seems fine. Less redundant.

…1.28.0

tswast · 2019-05-08T21:09:24Z

+ LABELS("labels"),
+ LAST_MODIFIED_TIME("lastModifiedTime"),
+ LOCATION("location"),
+ MODEL_REFERENCE("modelReference"),


Oh, you're right. TYPE seems fine. Less redundant.

codecov · 2019-05-09T16:08:37Z

Codecov Report

Merging #5021 into master will increase coverage by 0.01%.
The diff coverage is 65.09%.

@@ Coverage Diff @@## master #5021 +/- ##
============================================
+ Coverage 50.38% 50.4% +0.01% - Complexity 23736 23783 +47 
============================================
Files 2248 2251 +3 Lines 226471 226789 +318 Branches 24954 24968 +14 ============================================
+ Hits 114109 114314 +205 - Misses 103770 103870 +100 - Partials 8592 8605 +13

Impacted Files	Coverage Δ	Complexity Δ
.../com/google/cloud/bigquery/spi/v2/BigQueryRpc.java	`77.77% <ø> (ø)`	`0 <0> (ø)`	⬇️
.../google/cloud/bigquery/spi/v2/HttpBigQueryRpc.java	`6.1% <0%> (-0.83%)`	`2 <0> (ø)`
...e/cloud/bigquery/testing/RemoteBigQueryHelper.java	`62.5% <0%> (-1.14%)`	`6 <0> (ø)`
...n/java/com/google/cloud/bigquery/BigQueryImpl.java	`80.21% <60.75%> (-3.89%)`	`58 <5> (+5)`
...src/main/java/com/google/cloud/bigquery/Model.java	`75.51% <75.51%> (ø)`	`12 <12> (?)`
...main/java/com/google/cloud/bigquery/ModelInfo.java	`78.84% <78.84%> (ø)`	`18 <18> (?)`
...c/main/java/com/google/cloud/bigquery/ModelId.java	`79.16% <79.16%> (ø)`	`13 <13> (?)`
.../main/java/com/google/cloud/bigquery/BigQuery.java	`78.18% <80.76%> (+0.48%)`	`0 <0> (ø)`	⬇️
...able/gaxx/reframing/ReframingResponseObserver.java	`88.99% <0%> (-1.84%)`	`29% <0%> (-1%)`
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 20912a4...47957b6. Read the comment docs.

shollyman · 2019-05-09T17:35:18Z

The failing kokoro OSX Java 8 test is related to datastore, not BQ. Will proceed with submit.

shollymanand others added 10 commits April 22, 2019 08:14

initial model
ecd1703

Merge branch 'models' ofhttps://github.com/shollyman/google-cloud-java
2b55784

more impl
b890c81

more modelling
2a01f8d

Merge branch 'master' ofhttps://github.com/googleapis/google-cloud-java
6d9aa15

Merge branch 'master' into models
7cb0247

more modelling
b4cd4e6

model testing
cb8961f

more tests and comments
506d44c

more tests, mark bigquery.delete(string, string, string) deprecated
f100375

shollyman added api: bigquery Issues related to the BigQuery API. do not merge Indicates a pull request not ready for merge, due to either quality or timing. labels Apr 25, 2019

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Apr 25, 2019

shollyman changed the title ~~Models~~BigQuery: adds models API implementation.Apr 25, 2019

tswast reviewed Apr 25, 2019
View reviewed changes

shollyman added 3 commits April 29, 2019 10:50

Merge branch 'master' ofhttps://github.com/googleapis/google-cloud-java
3caef2c

Merge branch 'master' into models
b5a6569

typo
715a99a

tswast approved these changes May 1, 2019
View reviewed changes

Merge branch 'master' ofhttps://github.com/googleapis/google-cloud-java
ca1f0e0

yoshi-automation added the 🚨 critical P0 critical issue. Requires immediate fix label May 2, 2019

shollyman added 3 commits May 8, 2019 10:52

Merge branch 'master' ofhttps://github.com/googleapis/google-cloud-java
7896589

Merge branch 'master' into models and unpin
bf6fa35

address review comments
16d3ab8

shollyman marked this pull request as ready for review May 8, 2019 18:06

shollyman requested a review from a teamMay 8, 2019 18:06

shollyman added 3 commits May 8, 2019 11:22

formatter + typo
83769ce

update pinning of bigquery in google-cloud-clients to v2-rev20190423-…
47c1fca
…1.28.0

step down from 1.28.0 to 1.27.0 due to bounding on deps
ae12428

tswast approved these changes May 8, 2019
View reviewed changes

shollyman added 3 commits May 9, 2019 09:07

backout local pin, will pick up new pin from master
dc0cc2c

Merge branch 'master' ofhttps://github.com/googleapis/google-cloud-java
09913dd

Merge branch 'master' into models
47957b6

shollyman merged commit 57ac06a into googleapis:masterMay 9, 2019

shollyman removed 🚨 critical P0 critical issue. Requires immediate fix do not merge Indicates a pull request not ready for merge, due to either quality or timing. labels May 9, 2019

chingor13 mentioned this pull request May 15, 2019
Release google-cloud-java v0.92.0 #5189
Merged

Uh oh!

Conversation

shollyman commented Apr 25, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tswast commented May 1, 2019

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecovBot commented May 9, 2019

Codecov Report

Uh oh!

shollyman commented May 9, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants