Skip to content

fix(bigquery): write pandas datetime[ns] columns to BigQuery TIMESTAMP columns - #10028

Merged
plamut merged 8 commits into
googleapis:masterfrom
tswast:issue9996-load_table_from_dataframe-sample-tests
Jan 11, 2020
Merged

fix(bigquery): write pandas datetime[ns] columns to BigQuery TIMESTAMP columns#10028
plamut merged 8 commits into
googleapis:masterfrom
tswast:issue9996-load_table_from_dataframe-sample-tests

Conversation

@tswast

@tswasttswast commented Dec 30, 2019•
edited
Loading

Copy link
Copy Markdown
Contributor

Also:

  • Enable TIMESTAMP and DATETIME unit tests for _pandas_helpers.
  • Add more data types to load dataframe sample.

Fixes#9996 🦕

@googlebotgooglebot added the cla: yes This human has signed the Contributor License Agreement. label Dec 30, 2019
@tswasttswast mentioned this pull request Dec 30, 2019
Closed
@tswasttswast changed the title doc(bigquery): add more data types to load dataframe samplefix(bigquery): interpret datetime columns from pandas dataframe as nanosecondsJan 2, 2020
@tswast
tswastforce-pushed the issue9996-load_table_from_dataframe-sample-tests branch from e10f755 to ee48df8CompareJanuary 2, 2020 20:32
@tswast
fix(bigquery): write pandas datetime[ns] columns to BigQuery TIMESTAM…
6965558
…P columns
Also:
* Enable TIMESTAMP and DATETIME unit tests for `_pandas_helpers`.
* Add more data types to load dataframe sample.
@tswast
tswastforce-pushed the issue9996-load_table_from_dataframe-sample-tests branch from ee48df8 to 6965558CompareJanuary 2, 2020 23:13
@tswasttswast changed the title fix(bigquery): interpret datetime columns from pandas dataframe as nanosecondsfix(bigquery): write pandas datetime[ns] columns to BigQuery TIMESTAMP columnsJan 2, 2020
@tswast
tswast marked this pull request as ready for review January 2, 2020 23:18
@tswast
tswast requested review from a team and plamutJanuary 2, 2020 23:18
@tswast
update client tests
36fee78
@tswast
doc: show timezone conversions for timestamp columns
4fda7f5
Pandas doesn't automatically convert datetime objects to UTC time, so
show how to do this in the code sample.
@tswast

tswast commented Jan 3, 2020

Copy link
Copy Markdown
ContributorAuthor

Failure is for the docs session and appears to be just a temporary flake with the gax docs.

sphinx.errors.SphinxWarning: failed to reach any of the inventories with the following issues:
intersphinx inventory 'https://gax-python.readthedocs.org/en/latest/objects.inv' not fetchable due to <class 'requests.exceptions.ConnectionError'>: HTTPSConnectionPool(host='gax-python.readthedocs.org', port=443): Max retries exceeded with url: /en/latest/objects.inv (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fdef467d050>: Failed to establish a new connection: [Errno 110] Connection timed out'))

@plamutplamut added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jan 9, 2020
@yoshi-kokoroyoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jan 9, 2020
plamut
plamut suggested changes Jan 9, 2020
View reviewed changes

@plamutplamut left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to fix an undefined name that causes the snippets and lint checks to fail. Seems good otherwise.

Comment threadbigquery/samples/tests/test_load_table_dataframe.py
@tswast
tswast requested a review from plamutJanuary 10, 2020 15:38
plamut
plamut approved these changes Jan 11, 2020
View reviewed changes

@plamutplamut left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@plamut
plamut merged commit d897d56 into googleapis:masterJan 11, 2020
@tswast
tswast deleted the issue9996-load_table_from_dataframe-sample-tests branch January 13, 2020 14:29
This was referenced Jan 29, 2020
Merged
Merged
Merged
Merged
Merged
This was referenced Jan 29, 2020
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Closed
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@plamutplamutplamut approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

cla: yesThis human has signed the Contributor License Agreement.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

BigQuery: DATETIME columns invalid when uploaded with load_table_from_dataframe

4 participants

@tswast@plamut@googlebot@yoshi-kokoro