Skip to content

[BEAM-12482] Respect schemaUpdateOptions during BigQuery load with temporary tables - #16303

Closed
MiguelAnzoWizeline wants to merge 2 commits into
apache:masterfrom
MiguelAnzoWizeline:BEAM-12482_bigqueryio_schema_update_options
Closed

[BEAM-12482] Respect schemaUpdateOptions during BigQuery load with temporary tables#16303
MiguelAnzoWizeline wants to merge 2 commits into
apache:masterfrom
MiguelAnzoWizeline:BEAM-12482_bigqueryio_schema_update_options

Conversation

@MiguelAnzoWizeline

Copy link
Copy Markdown
Contributor

Adding a zero row job to bigquery writeTable to update the schema when using temporary tables


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

ValidatesRunner compliance status (on master branch)

LangULRDataflowFlinkSamzaSparkTwister2
Go---Build StatusBuild StatusBuild StatusBuild Status---
JavaBuild StatusBuild Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Python---Build Status
Build Status
Build Status
Build Status
Build Status
Build StatusBuild Status---
XLangBuild StatusBuild StatusBuild StatusBuild StatusBuild Status---

Examples testing status on various runners

LangULRDataflowFlinkSamzaSparkTwister2
Go---------------------
Java---Build Status
Build Status
Build Status
---------------
Python---------------------
XLang---------------------

Post-Commit SDK/Transform Integration Tests Status (on master branch)

GoJavaPython
Build StatusBuild StatusBuild Status
Build Status
Build Status

Pre-Commit Tests Status (on master branch)

---JavaPythonGoWebsiteWhitespaceTypescript
Non-portableBuild Status
Build Status
Build Status
Build Status
Build Status
Build StatusBuild StatusBuild StatusBuild Status
Portable---Build StatusBuild Status---------

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests

See CI.md for more information about GitHub Actions CI.

@MiguelAnzoWizeline

Copy link
Copy Markdown
ContributorAuthor

Run Java PostCommit

@MiguelAnzoWizeline

Copy link
Copy Markdown
ContributorAuthor

Run Java PreCommit

2 similar comments
@MiguelAnzoWizeline

Copy link
Copy Markdown
ContributorAuthor

Run Java PreCommit

@MiguelAnzoWizeline

Copy link
Copy Markdown
ContributorAuthor

Run Java PreCommit

@MiguelAnzoWizeline

Copy link
Copy Markdown
ContributorAuthor

R: @chamikaramj Hi, I'm working on a fix for https://issues.apache.org/jira/browse/BEAM-12482 that I see you were commenting on, could you give it a review or know who can do it? The PR is an attempt to replicate the fix done in #14113 for python.

@aaltay

Copy link
Copy Markdown
Member

@chamikaramj - Could you please review this change?

@kileys

Copy link
Copy Markdown
Contributor

R: @pabloem


String[] values = {"meow", "bark"};
TableRow rowToInsert =
new TableRow().set("new_field", values[0]).set("required_field", values[1]);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is a single row enough to cause two loads to happen? Can we please add another Row so that we can force a two-stage load?

@aaltay

Copy link
Copy Markdown
Member

@MiguelAnzoWizeline - could you please respond to open comments?

@fernando-wizeline

Copy link
Copy Markdown
Contributor

@MiguelAnzoWizeline - could you please respond to open comments?

Hi @aaltay! @MiguelAnzoWizeline is no longer in our team.
We in the team will be taking care of this and reply to the comments posted before.
Thanks!

@aaltay

Copy link
Copy Markdown
Member

@MiguelAnzoWizeline - could you please respond to open comments?

Hi @aaltay! @MiguelAnzoWizeline is no longer in our team. We in the team will be taking care of this and reply to the comments posted before. Thanks!

Ack. Thank you. Good luck @MiguelAnzoWizeline and thank you for everything :)

And we are all Beam contributors, you are all always welcome here.

@github-actions

Copy link
Copy Markdown
Contributor

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.

@pabloem

Copy link
Copy Markdown
Member

@fernando-wizeline do we have plans for this PR?

@fernando-wizeline

Copy link
Copy Markdown
Contributor

@fernando-wizeline do we have plans for this PR?

Hi @pabloem!
@MarcoRob is currently working on this PR. He's just ironing out some issues related to the insertion.

@MarcoRob

Copy link
Copy Markdown
Contributor

@fernando-wizeline do we have plans for this PR?

Hi @pabloem! @MarcoRob is currently working on this PR. He's just ironing out some issues related to the insertion.

Hi @pabloem I already finished the changes I will be submitting a new PR with the changes on this fix, I am just pulling the tests since the new changes and flow affected the tests on BigQuerySchemaUpdateOptionsIT.

@MarcoRob

MarcoRob commented Apr 13, 2022

Copy link
Copy Markdown
Contributor

Link to the new PR-17365, this PR can be closed, since it will no longer be needed.

@aaltayaaltay closed this Apr 20, 2022
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@MiguelAnzoWizeline@aaltay@kileys@fernando-wizeline@pabloem@MarcoRob