Skip to content

BigQuery Storage API sample for reading pandas dataframe - #1994

Merged
tswast merged 10 commits into
masterfrom
tswast-bqstorage-pandas
Feb 7, 2019
Merged

BigQuery Storage API sample for reading pandas dataframe#1994
tswast merged 10 commits into
masterfrom
tswast-bqstorage-pandas

Conversation

@tswast

@tswasttswast commented Feb 4, 2019

Copy link
Copy Markdown
Contributor

How to get a pandas DataFrame, fast!

The first two examples use the existing BigQuery client. These examples
create a thread pool and read in parallel. The final example shows using
just the new BigQuery Storage client, but only shows how to read with a
single thread.

@tswasttswast added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Feb 4, 2019
@googlebotgooglebot added the cla: yes This human has signed the Contributor License Agreement. label Feb 4, 2019
@tswasttswast removed the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Feb 5, 2019
@tswast
tswastforce-pushed the tswast-bqstorage-pandas branch from 0eaca4e to a3a48c0CompareFebruary 5, 2019 00:37
shollyman
shollyman approved these changes Feb 5, 2019
View reviewed changes

@shollymanshollyman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, modulo the open question on small results

Comment threadbigquery_storage/to_dataframe/main.py Outdated
# [START bigquerystorage_pandas_read_query_results]
import uuid

# Due to a known issue in the BigQuery Storage API (TODO: link to

@shollymanshollymanFeb 5, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we consider simply running a large query that emits enough data to avoid the inline? Pros: better demonstrates the perf of the new API, and avoids us having to revisit the sample. Cons: test time overhead and potential pitfalls for people kicking tires with small results. Part of this is dependent on how the team will be maintaining their KI list.

@tswasttswastFeb 5, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cons: test time overhead.

Long test time is my main reason for avoiding queries that return big results. I guess it's not so bad since this repo can test the different directories independently.

Cons: potential pitfalls for people kicking tires with small results.

The current failure case is rather bad: it returns a successful response, but gives you an empty result set. That's a pretty big pitfall, because catching it requires noticing that you didn't get the data you thought you were going to get.

@shollymanshollymanFeb 5, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works for me. Let's confirm how the KIs will be maintained so you can link them.

alixhami
alixhami reviewed Feb 5, 2019
View reviewed changes
Comment threadbigquery_storage/to_dataframe/main.py Outdated
Comment threadbigquery_storage/to_dataframe/main.py Outdated
Comment threadbigquery_storage/to_dataframe/main.py Outdated
Comment threadbigquery_storage/to_dataframe/main.py Outdated
alixhami
alixhami reviewed Feb 5, 2019
View reviewed changes
Comment threadbigquery_storage/to_dataframe/main_test.py Outdated
alixhami
alixhami reviewed Feb 5, 2019
View reviewed changes
Comment threadbigquery_storage/to_dataframe/main_test.py Outdated
Comment threadbigquery_storage/to_dataframe/main_test.py Outdated
Comment threadbigquery_storage/to_dataframe/main_test.py Outdated
Comment threadbigquery_storage/to_dataframe/main_test.py Outdated
Comment threadbigquery_storage/to_dataframe/main_test.py
@tswast
BigQuery Storage API sample for reading pandas dataframe
8ed8334
How to get a pandas DataFrame, fast!
The first two examples use the existing BigQuery client. These examples
create a thread pool and read in parallel. The final example shows using
just the new BigQuery Storage client, but only shows how to read with a
single thread.
@tswast
Add tests.
b7fae9d
@tswast
Add fixture to create temporary dataset.
6ded541
Blacken sample.
@tswast
Remove unnecessary CLI logic.
3ccc002
* Move imports inside region tags.
* Adjust query indentation to match region tags.
@tswast
Move sample code to test file.
2274a87
@tswast
Rename project_id to your_project_id
1be1a6e
Move duplicate imports out of region tags.
Add region tag for the whole sample.
@tswast
Remove check for no streams. Let index accesssor fail, instead.
0dbcca6
@tswast
Move temporary dataset creation...
398a2de
to just above the sample where it is used. This
makes the complete source code for the sample
make more sense (bigquerystorage_pandas_tutorial_all)
@tswast
Trim line length.
a4b6740
@tswast
Add region tags for use in cleanup section.
925fe3b
@tswast
tswastforce-pushed the tswast-bqstorage-pandas branch from f064277 to 925fe3bCompareFebruary 7, 2019 18:47
@tswast
tswast merged commit e9bc7de into masterFeb 7, 2019
@tswast
tswast deleted the tswast-bqstorage-pandas branch February 7, 2019 18:53
plamut pushed a commit to plamut/python-bigquery-storage that referenced this pull request Sep 2, 2020
@tswast
BigQuery Storage API sample for reading pandas dataframe [(#1994)](Go…
e73d7d4
…ogleCloudPlatform/python-docs-samples#1994)
* BigQuery Storage API sample for reading pandas dataframe
How to get a pandas DataFrame, fast!
The first two examples use the existing BigQuery client. These examples
create a thread pool and read in parallel. The final example shows using
just the new BigQuery Storage client, but only shows how to read with a
single thread.
plamut pushed a commit to googleapis/python-bigquery-storage that referenced this pull request Sep 10, 2020
@tswast@plamut
BigQuery Storage API sample for reading pandas dataframe [(#1994)](Go…
e0b1b25
…ogleCloudPlatform/python-docs-samples#1994)
* BigQuery Storage API sample for reading pandas dataframe
How to get a pandas DataFrame, fast!
The first two examples use the existing BigQuery client. These examples
create a thread pool and read in parallel. The final example shows using
just the new BigQuery Storage client, but only shows how to read with a
single thread.
Linchin pushed a commit that referenced this pull request Aug 18, 2025
@tswast@plamut
BigQuery Storage API sample for reading pandas dataframe [(#1994)](#1994
81bb6f7
)
* BigQuery Storage API sample for reading pandas dataframe
How to get a pandas DataFrame, fast!
The first two examples use the existing BigQuery client. These examples
create a thread pool and read in parallel. The final example shows using
just the new BigQuery Storage client, but only shows how to read with a
single thread.
@LinchinLinchin mentioned this pull request Aug 18, 2025
Merged
9 tasks
parthea pushed a commit to googleapis/google-cloud-python that referenced this pull request Aug 21, 2025
@tswast@plamut
BigQuery Storage API sample for reading pandas dataframe [(#1994)](Go…
7a8f31b
…ogleCloudPlatform/python-docs-samples#1994)
* BigQuery Storage API sample for reading pandas dataframe
How to get a pandas DataFrame, fast!
The first two examples use the existing BigQuery client. These examples
create a thread pool and read in parallel. The final example shows using
just the new BigQuery Storage client, but only shows how to read with a
single thread.
parthea pushed a commit to googleapis/google-cloud-python that referenced this pull request Sep 16, 2025
@tswast@parthea
BigQuery Storage API sample for reading pandas dataframe [(#1994)](Go…
21e91ba
…ogleCloudPlatform/python-docs-samples#1994)
* BigQuery Storage API sample for reading pandas dataframe
How to get a pandas DataFrame, fast!
The first two examples use the existing BigQuery client. These examples
create a thread pool and read in parallel. The final example shows using
just the new BigQuery Storage client, but only shows how to read with a
single thread.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@shollymanshollymanshollyman approved these changes
@alixhamialixhamialixhami approved these changes

Assignees

No one assigned

Labels

bigquerycla: yesThis human has signed the Contributor License Agreement.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@tswast@shollyman@alixhami@googlebot