Skip to content

[BEAM-12482] Ensure that we ignore schema update options when loading temporary tables that don't exist. - #15635

Merged
lukecwik merged 2 commits into
apache:masterfrom
lukecwik:beam12482
Oct 6, 2021
Merged

[BEAM-12482] Ensure that we ignore schema update options when loading temporary tables that don't exist.#15635
lukecwik merged 2 commits into
apache:masterfrom
lukecwik:beam12482

Conversation

@lukecwik

Copy link
Copy Markdown
Member

Note that this doesn't solve schema update completely and only works if the destination table doesn't exist but expands slightly the set of use cases supported.

This depends on the destination table having a schema that matches that of the temp tables effectively making the SchemaUpdateOptions moot.


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

ValidatesRunner compliance status (on master branch)

LangULRDataflowFlinkSamzaSparkTwister2
Go---Build StatusBuild StatusBuild StatusBuild Status---
JavaBuild StatusBuild Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Python---Build Status
Build Status
Build Status
Build Status
Build Status
Build StatusBuild Status---
XLangBuild StatusBuild StatusBuild StatusBuild StatusBuild Status---

Examples testing status on various runners

LangULRDataflowFlinkSamzaSparkTwister2
Go---------------------
Java---Build Status
Build Status
Build Status
---------------
Python---------------------
XLang---------------------

Post-Commit SDK/Transform Integration Tests Status (on master branch)

GoJavaPython
Build StatusBuild StatusBuild Status
Build Status
Build Status

Pre-Commit Tests Status (on master branch)

---JavaPythonGoWebsiteWhitespaceTypescript
Non-portableBuild Status
Build Status
Build Status
Build Status
Build Status
Build StatusBuild StatusBuild StatusBuild Status
Portable---Build StatusBuild Status---------

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests

See CI.md for more information about GitHub Actions CI.

@lukecwik

Copy link
Copy Markdown
MemberAuthor

R: @chamikaramj
CC: @reuvenlax

@chamikaramj

Copy link
Copy Markdown
Contributor

Do we need something similar to what was done for Python: #14113
(running a zero row load job against the destination table)

@lukecwik

Copy link
Copy Markdown
MemberAuthor

Do we need something similar to what was done for Python: #14113 (running a zero row load job against the destination table)

Yes. This is only a partial fix as in jobs that create the table will work and pushes the error further down the road as per the PR description.

@chamikaramj

Copy link
Copy Markdown
Contributor

LGTM. Thanks.

@lukecwik
lukecwik merged commit 49a96b9 into apache:masterOct 6, 2021
steveniemitz pushed a commit to twitter-forks/beam that referenced this pull request Oct 18, 2021
… temporary tables that don't exist. (apache#15635)
* [BEAM-12482] Ensure that we ignore schema update options when loading temporary tables that don't exist.
* fixup! Add comment.
steveniemitz pushed a commit to twitter-forks/beam that referenced this pull request Nov 1, 2021
… temporary tables that don't exist. (apache#15635)
* [BEAM-12482] Ensure that we ignore schema update options when loading temporary tables that don't exist.
* fixup! Add comment.
dmitriikuzinepam pushed a commit to dmitriikuzinepam/beam that referenced this pull request Nov 2, 2021
… temporary tables that don't exist. (apache#15635)
* [BEAM-12482] Ensure that we ignore schema update options when loading temporary tables that don't exist.
* fixup! Add comment.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@lukecwik@chamikaramj