[BEAM-13806] Add x-lang BigQuery IO integration test to Go SDK. by youngoli · Pull Request #16818 · apache/beam

youngoli · 2022-02-11T02:44:29Z

Also includes piping in flags for BigQuery IO through integration test script, and a small file for creating bigquery tables that expire after a day.

This was manually tested, but can't be tested on Jenkins at the moment because it only works on Dataflow and there currently isn't a test suite that runs Go xlang transforms on Dataflow. That should be fixed shortly and then this can be tested on Jenkins. It shouldn't be submitted until then.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

`ValidatesRunner` compliance status (on master branch)

Lang	ULR	Twister2
Go	---	---
Java
Python	---	---
XLang		---

Examples testing status on various runners

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---	---	---	---	---	---	---
Java	---		---	---	---	---	---
Python	---	---	---	---	---	---	---
XLang	---	---	---	---	---	---	---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go	Java	Python

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

youngoli · 2022-02-11T02:44:56Z

R: @lostluck @riteshghorse

lostluck

The big comment is about the output PCollections for Writes, but I don't expect that change in this PR. I beleive It's a best practice to always have those for write transforms so it's possible to block on them with side inputs. I clearly missed that in reviews last week.

lostluck · 2022-02-11T17:55:42Z

Does the bigqueryIO Write not return a pcollection we can have the read block on? As it stands, I think this will have both happen at the same time.

No, but I split the pipeline into two separate pipelines that will be run in sequence, which should avoid this issue.

lostluck · 2022-02-15T00:54:17Z

consider printing the table name using t.Log, to make it clearer which resources map to this test.

lostluck · 2022-02-15T00:56:57Z

Assuming the test is successful could we please also delete the table manually?
I see that we have limited it's lifespan, but I'd prefer that we don't accrue a rolling pile of these over the course of the 24 hour window.

Sure, done.

aaltay · 2022-02-25T00:56:26Z

@youngoli - What is the next step on this PR?

aaltay · 2022-03-10T17:52:40Z

Could this be merged?

youngoli · 2022-03-10T19:10:53Z

Sorry I haven't been checking this. Don't merge this yet, I need to test it on Jenkins (which requires getting in #16819 first). I'll try to find time to work on these and get them in.

youngoli · 2022-04-20T20:29:39Z

Run XVR_GoUsingJava_Dataflow PostCommit

youngoli · 2022-04-20T20:33:41Z

I rebased to fix merge errors and account for the changes in our integration testing script, but the functionality of the first commit should be unchanged, only the test framework and expansion service setup should be any different.

The second commit contains all the fixup addressing the comments.

youngoli · 2022-04-20T22:45:07Z

Run XVR_GoUsingJava_Dataflow PostCommit

youngoli · 2022-04-21T00:15:18Z

Run XVR_GoUsingJava_Dataflow PostCommit

aaltay · 2022-04-28T17:46:05Z

What is the next step on this PR?

robertwb · 2022-05-25T22:06:16Z

Also test reading from a query.

I did. As an update to what we talked about online, I'm not getting any permissions errors testing locally. (I am getting a different unrelated error, but it's because of a bug in how I wrote the test. Will commit once I fix that.)

Also includes piping in flags for BigQuery IO through integration test script, and a small file for creating bigquery tables that expire after a day.

Splits the integration test into two pipelines to run sequentially. Also drops table after a successful test and logs table names.

CreateRows wasn't creating the same elements in both read and write pipelines after splitting the two pipelines. Adjusted it to use a consistent seed in all pipelines.

codecov · 2022-06-03T21:05:19Z

Codecov Report

Merging #16818 (8db67b4) into master (9e5d085) will not change coverage.
The diff coverage is n/a.

@@ Coverage Diff @@## master #16818 +/- ##
=======================================
Coverage 74.10% 74.10% =======================================
Files 698 698 Lines 92486 92486 =======================================
Hits 68541 68541 Misses 22690 22690 Partials 1255 1255

Flag	Coverage Δ
go	`50.86% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9e5d085...8db67b4. Read the comment docs.

youngoli · 2022-06-03T21:07:40Z

I added a test that reads from a Query, and while testing in my own environment I'm getting weird results. The elements output from a query read have all fields as pointers because SQL queries don't match the table's schema. That's pretty normal. But when I made an equivalent struct in Go, I get errors when decoding into that struct, even though all the fields seem to be matching exactly:

panic: reflect: Call using struct { Counter *int64 "beam:\"counter\""; Rand_data *struct { Flip *bool "beam:\"flip\""; Num *int64 "beam:\"num\""; Word *string "beam:\"word\"" } "beam:\"rand_data\"" } as type bigquery.TestRowPtrs
Full error:
while executing Process for Plan[s08-67]:
2: DataSink[S[ptransform-65@localhost:12371]] Coder:W;coder-80<LP;coder-81<R[bigquery.TestRow]>>!GWC
3: PCollection[pcollection-72] Out:[2]
4: ParDo[bigquery.ReadFromQueryPipeline.func1] Out:[2]
1: DataSource[S[ptransform-64@localhost:12371], 0] Coder:W;coder-76<LP;coder-77<R[struct { Counter *int64 "beam:\"counter\""; Rand_data *struct { Flip *bool "beam:\"flip\""; Num *int64 "beam:\"num\""; Word *string "beam:\"word\"" } "beam:\"rand_data\"" }]>>!GWC Out:4
caused by:
panic: reflect: Call using struct { Counter *int64 "beam:\"counter\""; Rand_data *struct { Flip *bool "beam:\"flip\""; Num *int64 "beam:\"num\""; Word *string "beam:\"word\"" } "beam:\"rand_data\"" } as type bigquery.TestRowPtrs goroutine 60 [running]:
runtime/debug.Stack()
...

youngoli · 2022-06-03T22:25:33Z

Run XVR_GoUsingJava_Dataflow PostCommit

lostluck

If we document the type alias work around for "lost" types, I'm very comfortable getting this in. We should file an issue for whatever is happening with Query. We likely need to simply hard replace the coder java thinks it has in the CrossLanguage handler, or figure out why a new one is derived in the first place.

youngoli · 2022-06-10T06:45:44Z

Run XVR_GoUsingJava_Dataflow PostCommit

lostluck · 2022-06-10T17:12:58Z

Is this the error @robertwb ran into? Missing ProjectID?

:39:18 --- FAIL: TestBigQueryIO_BasicWriteRead (418.27s)
00:39:18 panic: tried cross-language for beam:transform:org.apache.beam:schemaio_bigquery_read:v1 against localhost:37821 and failed
00:39:18 expanding external transform
00:39:18 expanding transform with ExpansionRequest: components:{environments:{key:"go" value:{}}} transform:{unique_name:"External" spec:{urn:"beam:transform:org.apache.beam:schemaio_bigquery_read:v1" payload:"\nX\n\x0e\n\x08location\x1a\x02\x10\x07\n\x0c\n\x06config\x1a\x02\x10\t\n\x12\n\ndataSchema\x1a\x04\x08\x01\x10\t\x12$3fc24beb-ef0b-4fd0-b491-64340b33ca1e\x12k\x03\x01\x04\x00f\x04\x01\rbSELECT * FROM `apache-beam-testing.beam_bigquery_io_test_temp.go_bqio_it_temp_1654844426145960792`"} environment_id:"go"} namespace:"fPIRkaLpTJ"
00:39:18 expansion failed
00:39:18 caused by:
00:39:18 org.apache.beam.sdk.io.gcp.bigquery.BigQuerySchemaRetrievalException: Exception while trying to retrieve schema of query
00:39:18 at org.apache.beam.sdk.io.gcp.bigquery.BigQueryQuerySourceDef.getBeamSchema(BigQueryQuerySourceDef.java:179)
00:39:18 at org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$TypedRead.expand(BigQueryIO.java:1222)
00:39:18 at org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$TypedRead.expand(BigQueryIO.java:743)
00:39:18 at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:548)
00:39:18 at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:482)
00:39:18 at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
00:39:18 at org.apache.beam.sdk.io.gcp.bigquery.BigQuerySchemaIOProvider$BigQuerySchemaIO$1.expand(BigQuerySchemaIOProvider.java:183)
00:39:18 at org.apache.beam.sdk.io.gcp.bigquery.BigQuerySchemaIOProvider$BigQuerySchemaIO$1.expand(BigQuerySchemaIOProvider.java:165)
00:39:18 at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:548)
00:39:18 at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:499)
00:39:18 at org.apache.beam.sdk.expansion.service.ExpansionService$TransformProvider.apply(ExpansionService.java:396)
00:39:18 at org.apache.beam.sdk.expansion.service.ExpansionService.expand(ExpansionService.java:515)
00:39:18 at org.apache.beam.sdk.expansion.service.ExpansionService.expand(ExpansionService.java:595)
00:39:18 at org.apache.beam.model.expansion.v1.ExpansionServiceGrpc$MethodHandlers.invoke(ExpansionServiceGrpc.java:220)
00:39:18 at org.apache.beam.vendor.grpc.v1p43p2.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182)
00:39:18 at org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:340)
00:39:18 at org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:866)
00:39:18 at org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
00:39:18 at org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
00:39:18 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
00:39:18 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
00:39:18 at java.lang.Thread.run(Thread.java:748)
00:39:18 Caused by: java.lang.NullPointerException: Required parameter projectId must be specified.
00:39:18 at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:907)
00:39:18 at com.google.api.client.util.Preconditions.checkNotNull(Preconditions.java:138)
00:39:18 at com.google.api.services.bigquery.Bigquery$Jobs$Insert.<init>(Bigquery.java:1737)
00:39:18 at com.google.api.services.bigquery.Bigquery$Jobs.insert(Bigquery.java:1687)
00:39:18 at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.dryRunQuery(BigQueryServicesImpl.java:473)
00:39:18 at org.apache.beam.sdk.io.gcp.bigquery.BigQueryQueryHelper.dryRunQueryIfNeeded(BigQueryQueryHelper.java:73)
00:39:18 at org.apache.beam.sdk.io.gcp.bigquery.BigQueryQuerySourceDef.getBeamSchema(BigQueryQuerySourceDef.java:168)
00:39:18 ... 21 more

lostluck · 2022-06-10T17:24:47Z

Some of the "default" tests are also failing (prefix, multi, partition), but they're hard dying after apparently sending a bundle to the Java side (I see successful Go side messages). Definitely unrelated, but separately worth investigating.

youngoli · 2022-06-10T22:59:14Z

Yes, that looks like the same issue Robert ran into. It seems to be failing because the expansion service is unaware of the GCP project from pipeline options. But this was discussed earlier this week and it should be a relatively easy fix. I think for now though I might split the query and non-query reads into separate tests and sickbay the query read temporarily.

lostluck · 2022-06-10T23:23:33Z

SGTM. I'll give it a once over after and then we can merge.

…

On Fri, Jun 10, 2022, 3:59 PM Daniel Oliveira ***@***.***> wrote: Yes, that looks like the same issue Robert ran into. It seems to be failing because the expansion service is unaware of the GCP project from pipeline options. But this was discussed earlier this week and it should be a relatively easy fix. I think for now though I might split the query and non-query reads into separate tests and sickbay the query read temporarily. — Reply to this email directly, view it on GitHub <#16818 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADKDOFKPU73WTHQS5ONDVXTVOPCE3ANCNFSM5OC2L5JA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

lostluck · 2022-06-11T02:32:02Z

Run XVR_GoUsingJava_Dataflow PostCommit

lostluck · 2022-06-13T23:46:40Z

Prior to the merge conflicts (that I'll manually fix in a bit) I kicked off which should be one final run of these (less the filtered out Query test). Assuming this passes, a quick fix to merge the integration filters later, this will be good to merge.
https://ci-beam.apache.org/job/beam_PostCommit_XVR_GoUsingJava_Dataflow_PR/11/

lostluck · 2022-06-14T02:07:41Z

The run did pass, modulo the 3 unrelated tests that are currently failing. I'm guessing something changed to make those "built in" transforms to no longer be built in, so they're failing. They're working for python however, so there's some mechanism that's failing with how Go specifies them.

youngoli · 2022-06-14T04:22:22Z

Nice, merged in. Thanks for doing that last bit of checking, I appreciate it.

…he#16818) * [BEAM-13806] Add x-lang BigQuery IO integration test to Go SDK. Also includes piping in flags for BigQuery IO through integration test script, and a small file for creating bigquery tables that expire after a day. * [BEAM-13806] Splitting BigQuery IT into read and write pipelines. Splits the integration test into two pipelines to run sequentially. Also drops table after a successful test and logs table names. * Fixup: Fix gradle build and undo VR script changes. * Fixup: Add Query test and fix deterministic random element generation. CreateRows wasn't creating the same elements in both read and write pipelines after splitting the two pipelines. Adjusted it to use a consistent seed in all pipelines. * Fixup: Avoiding inline functions * Workaround to coder issue, plus some debugging code * Polishing workaround with documentation and removing debug prints. * Move pipeline code to test file * Split Query test from non-query test Co-authored-by: Robert Burke <lostluck@users.noreply.github.com>

github-actionsBot added the go label Feb 11, 2022

lostluck approved these changes Feb 15, 2022
View reviewed changes

youngoli force-pushed the dataflowit branch from ad02c5d to b532bc5Compare April 20, 2022 02:42

robertwb reviewed May 25, 2022
View reviewed changes

youngoli added 4 commits June 1, 2022 15:45

[BEAM-13806] Add x-lang BigQuery IO integration test to Go SDK.
21aae4f
Also includes piping in flags for BigQuery IO through integration test script, and a small file for creating bigquery tables that expire after a day.

[BEAM-13806] Splitting BigQuery IT into read and write pipelines.
02ccd10
Splits the integration test into two pipelines to run sequentially. Also drops table after a successful test and logs table names.

Fixup: Fix gradle build and undo VR script changes.
5ca09d3

Fixup: Add Query test and fix deterministic random element generation.
c9ce532
CreateRows wasn't creating the same elements in both read and write pipelines after splitting the two pipelines. Adjusted it to use a consistent seed in all pipelines.

youngoli force-pushed the dataflowit branch from b656410 to c9ce532Compare June 3, 2022 21:02

github-actionsBot added the io label Jun 3, 2022

youngoli added 2 commits June 8, 2022 15:29

Fixup: Avoiding inline functions
c4f62c8

Workaround to coder issue, plus some debugging code
3cabe4e

lostluck requested changes Jun 9, 2022
View reviewed changes

youngoli added 2 commits June 9, 2022 19:22

Polishing workaround with documentation and removing debug prints.
d723e08

Move pipeline code to test file
ca91b83

youngoliand others added 2 commits June 10, 2022 18:21

Split Query test from non-query test
fbcbf15

Merge branch 'master' into dataflowit
ac34ec1

Merge branch 'master' into dataflowit
8db67b4

youngoli merged commit 080f54a into apache:masterJun 14, 2022

Uh oh!

Conversation

youngoli commented Feb 11, 2022

ValidatesRunner compliance status (on master branch)

Examples testing status on various runners

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

Uh oh!

youngoli commented Feb 11, 2022

Uh oh!

lostluck left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aaltay commented Feb 25, 2022

Uh oh!

aaltay commented Mar 10, 2022

Uh oh!

youngoli commented Mar 10, 2022

Uh oh!

youngoli commented Apr 20, 2022

Uh oh!

youngoli commented Apr 20, 2022

Uh oh!

youngoli commented Apr 20, 2022

Uh oh!

youngoli commented Apr 21, 2022

Uh oh!

aaltay commented Apr 28, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecovBot commented Jun 3, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

youngoli commented Jun 3, 2022

Uh oh!

youngoli commented Jun 3, 2022

Uh oh!

lostluck left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

youngoli commented Jun 10, 2022

Uh oh!

lostluck commented Jun 10, 2022

Uh oh!

lostluck commented Jun 10, 2022

Uh oh!

youngoli commented Jun 10, 2022

Uh oh!

lostluck commented Jun 10, 2022 via email

Uh oh!

lostluck commented Jun 11, 2022

Uh oh!

lostluck commented Jun 13, 2022

Uh oh!

lostluck commented Jun 14, 2022

Uh oh!

youngoli commented Jun 14, 2022

Uh oh!

Reviewers

`ValidatesRunner` compliance status (on master branch)

codecovBot commented Jun 3, 2022•
edited
Loading