Support of Storage API

[yan-hic]

Any plan on supporting the bigquery storage api ?

---

Edit Sub-steps:

• Update BQ DB-API to support BigQuery Storage API [BigQuery, BigQuery Storage]: Add option to use BigQuery Storage API to download results in BigQuery DB-API python-bigquery#16
• Update create_connect_args to (optionally) return a BQ Storage API client alongside the BQ client.
• Add a use_bqstorage_api option to the connection URL parsing logic.

[tswast]

This requires adding support for the BigQuery Storage API to the DB-API adapter in google-cloud-bigquery, first.

I assume you want to improve the performance of downloading large results?

[yan-hic]

You are right on with your comment in https://github.com/googleapis/google-cloud-python/issues/9465, it is indeed to test/use with superset.

As a workaround, I can try pyodbc with the BQ ODBC driver.

[yan-hic]

@tswast does this close it ? : googleapis/python-bigquery#36 (comment)
If so, some documentation would be useful. Thanks

[tswast]

It unblocks this feature from being completed, but doesn't quite fix it yet.

We need to update create_connect_args to (optionally) return a BQ Storage API client alongside the BQ client.

https://github.com/mxmzdlv/pybigquery/blob/f427dbe0040cbdf3681c1937699af0bea2b89af6/pybigquery/sqlalchemy_bigquery.py#L318

As I understand it, the return value of create_connect_args is passed to the BigQuery DB-API connect function.

This will also require adding a use_bqstorage_api option to the connection URL parsing logic https://github.com/mxmzdlv/pybigquery/blob/master/pybigquery/parse_url.py

[alonme]

Can you guys add some more details about the usage of this feature?
I would like to implement it, but i feel like i won't be able to do it in the best way possible and test that it works without understanding this issue a bit more

[yan-hic]

@alonme nothing to document since not implemented yet.
In the meantime you can use the standard way to read rows, with hope for the turbo mode soon.

[alonme]

Maybe I wasn't clear,
I would be happy to contribute this feature.
On order to do that, id like to understand it better

[tswast]

@alonme Thank you for volunteering to help.

To verify the updates suggested in #41 (comment) make a query to table where the expected result set is moderately large (100 MB or more). When use_bqstorage_api=True is included in the connection string, you should notice a considerable (4-16x or more) speedup.

[alonme]

@tswast cool, got a PoC to work.
hope to get a nice branch for PR in the next days,
please assign to me

[yan-hic]

closing as per #61 (comment)