Skip to content

BigQuery: Allow subset of schema to be passed into load_table_from_dataframe. - #9064

Merged
tswast merged 9 commits into
googleapis:masterfrom
tswast:issue8140-partial-schema
Aug 22, 2019
Merged

BigQuery: Allow subset of schema to be passed into load_table_from_dataframe.#9064
tswast merged 9 commits into
googleapis:masterfrom
tswast:issue8140-partial-schema

Conversation

@tswast

@tswasttswast commented Aug 20, 2019•
edited
Loading

Copy link
Copy Markdown
Contributor

Closes#8140

  • Allow subset of schema to be passed into load_table_from_dataframe.
  • The types of any remaining columns will be auto-detected.
  • When the type of any remaining column can't be determined, warn that the schema was ignored because the type of column X could not be determined.
  • Update snippets to show overriding schema for columns whose types can't be autodetected. Blocked by BigQuery: Load to table from dataframe without index #5572 (comment).

This PR is branched from #9049.

@googlebotgooglebot added the cla: yes This human has signed the Contributor License Agreement. label Aug 20, 2019
@tswast
tswastforce-pushed the issue8140-partial-schema branch from 09bb705 to dba55a4CompareAugust 21, 2019 17:55
@tswast
tswastforce-pushed the issue8140-partial-schema branch from dba55a4 to 105ebc9CompareAugust 21, 2019 18:58
@tswast
tswast marked this pull request as ready for review August 21, 2019 18:59
@tswast
tswast requested review from a team and plamutAugust 21, 2019 18:59
@tswasttswast mentioned this pull request Aug 21, 2019
Closed
@tswasttswast mentioned this pull request Aug 21, 2019
Closed
@tswast

tswast commented Aug 22, 2019•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Blocked by #9074

Test failures are because the schema in the snippets samples cannot be autodetected and we send a schema of {"fields": []}, instead of None.

@tswasttswast added api: bigquery Issues related to the BigQuery API. do not merge Indicates a pull request not ready for merge, due to either quality or timing. labels Aug 22, 2019
plamut
plamut reviewed Aug 22, 2019
View reviewed changes

@plamutplamut left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, I can see how this can be useful to users. 👍

Had a few sanity check remarks/questions, please just check if any of them apply.

Edit: What about the last point in the PR checklist, i.e. updating snippets?

Comment threadbigquery/tests/unit/test_client.py
Comment threadbigquery/tests/system.py Outdated
Comment threadbigquery/google/cloud/bigquery/_pandas_helpers.py
Comment threadbigquery/tests/system.py
@tswasttswast mentioned this pull request Aug 22, 2019
Closed
@tswast

tswast commented Aug 22, 2019

Copy link
Copy Markdown
ContributorAuthor

On second thought, since the snippets show writing the indexes of a DataFrame, I can't actually update them until #5572 (comment) is fixed. I'll work on that issue next.

@tswast
tswast requested a review from plamutAugust 22, 2019 18:21
@tswasttswast removed the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Aug 22, 2019
plamut
plamut approved these changes Aug 22, 2019
View reviewed changes

@plamutplamut left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, and I agree with leaving out the snippets until their blocker is resolved.

Edit: One of the unit tests is failing, needs a fix.

@tswast
tswast merged commit adefce3 into googleapis:masterAug 22, 2019
@tswast
tswast deleted the issue8140-partial-schema branch August 22, 2019 19:36
HemangChothani pushed a commit to HemangChothani/google-cloud-python that referenced this pull request Aug 29, 2019
@tswast
Allow subset of schema to be passed intoload_table_from_dataframe. (…
a4739bd
…googleapis#9064)
* Allow subset of schema to be passed into `load_table_from_dataframe`.
The types of any remaining columns will be auto-detected.
* Warn when it's not possible to determine a column type.
emar-kar pushed a commit to MaxxleLLC/google-cloud-python that referenced this pull request Sep 18, 2019
@tswast@emar-kar
Allow subset of schema to be passed intoload_table_from_dataframe. (…
e3dcf4a
…googleapis#9064)
* Allow subset of schema to be passed into `load_table_from_dataframe`.
The types of any remaining columns will be auto-detected.
* Warn when it's not possible to determine a column type.
@tswasttswast mentioned this pull request Nov 12, 2019
Open
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@plamutplamutplamut approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@plamutplamut

Labels

api: bigqueryIssues related to the BigQuery API.cla: yesThis human has signed the Contributor License Agreement.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

BigQuery: Allow partial schema in load_table_from_dataframe

3 participants

@tswast@plamut@googlebot