Support Map and Arrays of Maps in BQ for StorageWrites for Beam Rows by prodriguezdefino · Pull Request #22179 · apache/beam

prodriguezdefino · 2022-07-07T01:25:55Z

Currently BigQuery table schema utility, and the implementation for StorageWrites for Beam Rows, does not support sending records with Maps as part of their schema.

This PR adds that functionality transforming the Map into a Message type which contains two fields key and value respecting the types coming from upstream while mimicking the behavior when using TableRows to the BigQueryIO PTransform.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

prodriguezdefino · 2022-07-09T23:27:44Z

Run Java PreCommit

github-actions · 2022-09-08T13:16:33Z

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.

prodriguezdefino · 2022-10-12T22:23:35Z

fixes #23618

prodriguezdefino · 2022-11-21T17:28:04Z

.remove-labels stale

reuvenlax · 2022-11-30T21:48:23Z

Can you rebase and resolve, since that file has since been refactored?

prodriguezdefino · 2022-12-01T00:39:49Z

Can you rebase and resolve, since that file has since been refactored?

done

reuvenlax · 2022-12-01T00:34:12Z

 if (fieldDescriptor.isOptional()) {
 return null;
+ } else if (fieldDescriptor.isRepeated()) {
+ return Lists.newArrayList();


Collections.emptyList()

reuvenlax · 2022-12-01T00:43:41Z

+ arrayElementType.getTypeName().isCollectionType()
+ || arrayElementType.getTypeName().isMapType()
+ ? true
+ : false;


remove redundant ternary operator. Also no need for Boolean

Done (what was I thinking)

reuvenlax · 2022-12-01T00:45:09Z

+
+ if (shouldFlatMap) {
+ valueStream = valueStream.flatMap(vs -> ((List) vs).stream());
+ }


explain why flatMap is correct here?

Because BQ does not support arrays of maps, this code is making the decision of flattening those structures (the if condition is computed based on that particular scenario).
[
map1 [k1,v2] [k2,v2] ,
map2 [k3,v3],
map3 [k4,v4] [k5,v5],
]
------------------------- to
[
record {key:k1, value:v1},
record {key:k2, value:v2},
record {key:k3, value:v3},
record {key:k4, value:v4},
record {key:k5, value:v5}
]
It respects the order in the array and the inherent order of iteration in the maps, but it won't check for repeated keys across the maps in the original array.

reuvenlax · 2022-12-01T00:51:26Z

+ FieldType keyFieldType,
+ FieldType valueFieldType,
+ Map.Entry<Object, Object> entryValue) {
+


remove extra line

reuvenlax · 2022-12-01T00:53:03Z

+ FieldDescriptor keyFieldDescriptor =
+ Preconditions.checkNotNull(descriptor.findFieldByName("key"));
+ @Nullable Object key = toProtoValue(keyFieldDescriptor, keyFieldType, entryValue.getKey());
+ if (key != null) {


are null keys allowed?

Good question, AFAICT the backing Map in the Row object is a HashMap (with expected size) so I would say that null keys are allowed in a map property for a Row object.
This code only ignores setting the value in the proto if null was the value present on the map's key.

prodriguezdefino · 2022-12-01T23:47:12Z

Run Java PreCommit

prodriguezdefino · 2022-12-01T23:47:23Z

Run Java_GCP_IO_Direct PreCommit

prodriguezdefino · 2022-12-02T18:49:31Z

Is there a way to rerun Java Tests / Java Unit Tests (ubuntu-latest) (pull_request) those are failing because:

org.apache.beam.fn.harness.jmh.ProcessBundleBenchmarkTest > testStateWithoutCaching FAILED
[176](https://github.com/apache/beam/actions/runs/3595444602/jobs/6054940468#step:5:177)
java.util.concurrent.ExecutionException at ProcessBundleBenchmarkTest.java:47
[177](https://github.com/apache/beam/actions/runs/3595444602/jobs/6054940468#step:5:178)
Caused by: java.lang.RuntimeException at ProcessBundleBenchmark.java:175
[178](https://github.com/apache/beam/actions/runs/3595444602/jobs/6054940468#step:5:179)
Caused by: java.lang.RuntimeException at UnboundedScheduledExecutorService.java:316
[179](https://github.com/apache/beam/actions/runs/3595444602/jobs/6054940468#step:5:180)
Caused by: java.lang.InterruptedException at FutureTask.java:418

Not related with this changes, but not sure how to ensure this works without pushing more commits.

reuvenlax · 2022-12-02T21:43:33Z

Has this been tested e2e (with BigQuery)?

prodriguezdefino · 2023-01-31T00:51:46Z

Has this been tested e2e (with BigQuery)?

yes.

map, job id: 2023-01-30_16_31_29-9471872084021240643

list/array of maps job id: 2023-01-30_16_32_57-10318556278839602747
schema on BQ:

reuvenlax · 2023-01-31T01:49:44Z

Run Java_GCP_IO_Direct PreCommit

prodriguezdefino · 2023-01-31T03:34:17Z

Run Java_GCP_IO_Direct PreCommit

github-actions · 2023-04-01T12:42:41Z

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.

github-actions · 2023-06-08T12:41:31Z

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.

github-actions · 2023-06-15T12:42:16Z

This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

JohnZZGithub · 2024-09-20T01:03:22Z

Could we merge the PR? This is necessary for us to ingest data to BigQuery.

prodriguezdefino · 2024-09-20T01:37:53Z

This PR has been closed for a while, I can take some time to migrate the changes to a new branch from master.

…

On Thu, Sep 19, 2024 at 6:03 PM zz ***@***.***> wrote: Could we merge the PR? This is necessary for us to ingest data to BigQuery. — Reply to this email directly, view it on GitHub <#22179 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2HMFYXRBLP2I6TEUYYPJTZXNX67AVCNFSM6AAAAABORALTZWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRSGUYDSMJTGQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

JohnZZGithub · 2024-09-20T22:08:28Z

This PR has been closed for a while, I can take some time to migrate the changes to a new branch from master.
…
On Thu, Sep 19, 2024 at 6:03 PM zz @.> wrote: Could we merge the PR? This is necessary for us to ingest data to BigQuery. — Reply to this email directly, view it on GitHub <#22179 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA2HMFYXRBLP2I6TEUYYPJTZXNX67AVCNFSM6AAAAABORALTZWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRSGUYDSMJTGQ . You are receiving this because you authored the thread.Message ID: @.>

Thanks a lot Pablo!

adding support for maps and array of maps to BQ when using StorageWrites
307ce71

github-actionsBot added gcp io java labels Jul 7, 2022

github-actionsBot added the stale label Sep 8, 2022

prodriguezdefinoand others added 3 commits October 12, 2022 09:36

Merge branch 'master' into support_maps_plus_arrayofmap_bq_storagewrites
00ecf6b

adding tests for new functionality
10aa27a

spotless apply
7c41bee

prodriguezdefino marked this pull request as ready for review October 12, 2022 22:25

prodriguezdefino changed the title ~~Support Map and Arrays of Maps in BQ for StorageWrites~~Support Map and Arrays of Maps in BQ for StorageWrites for Beam RowsOct 31, 2022

fixes checkstyle error
c9e22d6

prodriguezdefino added 2 commits November 30, 2022 15:27

Merge branch 'master' into support_maps_plus_arrayofmap_bq_storagewrites
27683f8

spotless apply
7cf633b

reuvenlax reviewed Dec 1, 2022
View reviewed changes

prodriguezdefino added 2 commits December 1, 2022 09:40

addressing comments
5e450e9

fixing spotless expected format
5f588e5

prodriguezdefino requested a review from reuvenlax December 6, 2022 05:09

Merge branch 'master' into support_maps_plus_arrayofmap_bq_storagewrites
e6d7d56

github-actionsBot removed the stale label Jan 27, 2023

ran spotless apply
cda4069

prodriguezdefino added a commit to prodriguezdefino/apache-beam-streaming-tests that referenced this pull request Jan 31, 2023
testingapache/beam#24274andapache/beam#22179
9196d4a

prodriguezdefino mentioned this pull request Jan 31, 2023
Changes after Avro related changes on BigQueryIO prodriguezdefino/apache-beam-streaming-tests#1
Merged

github-actionsBot added stale and removed stale labels Apr 1, 2023

github-actionsBot added the stale label Jun 8, 2023

github-actionsBot closed this Jun 15, 2023

prodriguezdefino mentioned this pull request Sep 20, 2024
Support Map in BQ for StorageWrites API for Beam Rows #32512
Merged
3 tasks

Uh oh!

Conversation

prodriguezdefino commented Jul 7, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GitHub Actions Tests Status (on master branch)

Uh oh!

prodriguezdefino commented Jul 9, 2022

Uh oh!

github-actionsBot commented Sep 8, 2022

Uh oh!

prodriguezdefino commented Oct 12, 2022

Uh oh!

prodriguezdefino commented Nov 21, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

reuvenlax commented Nov 30, 2022

Uh oh!

prodriguezdefino commented Dec 1, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

prodriguezdefinoDec 1, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

prodriguezdefino commented Dec 1, 2022

Uh oh!

prodriguezdefino commented Dec 1, 2022

Uh oh!

prodriguezdefino commented Dec 2, 2022

Uh oh!

reuvenlax commented Dec 2, 2022

Uh oh!

prodriguezdefino commented Jan 31, 2023

Uh oh!

reuvenlax commented Jan 31, 2023

Uh oh!

prodriguezdefino commented Jan 31, 2023

Uh oh!

github-actionsBot commented Apr 1, 2023

Uh oh!

github-actionsBot commented Jun 8, 2023

Uh oh!

github-actionsBot commented Jun 15, 2023

Uh oh!

JohnZZGithub commented Sep 20, 2024

Uh oh!

prodriguezdefino commented Sep 20, 2024 via email

Uh oh!

JohnZZGithub commented Sep 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

prodriguezdefino commented Jul 7, 2022•
edited
Loading

prodriguezdefino commented Nov 21, 2022•
edited
Loading

prodriguezdefinoDec 1, 2022•
edited
Loading