refactor(bigquery): to_dataframe uses faster to_arrow + to_pandas when pyarrow is available

[tswast]

…pyarrow is available

Related to similar PR #9997 but for the google-cloud-bigquery library.

Fixes https://issuetracker.google.com/140579733

[plamut]

Looks fine code-wise IMO.

Two remarks:

• Do we have a common representative table at hand to verify the stated performance gains and compare the results? If not, I can still manually create a dummy table with 10M floats, just like in the bug description.
• The coverage check failure is legitimate and should be fixed.

[tswast]

In https://friendliness.dev/2019/07/29/bigquery-arrow/, we sampled the tlc_green_trips public data by running a SQL query like:

SELECT *
FROM table_name
WHERE RAND() < 0.06

and writing to a destination table so that we can read directly from the table (making query time not part of the benchmark).

[tswast]

Using the same (n1-standard-4) instance from #9997, I tested the speedup in the same way.

cd google-cloud-python/bigquery
git checkout b140579733-bq-to_dataframe-part-2
pip3 install -e .
cd ../..

benchmark_bq.py

importsysfromgoogle.cloudimportbigqueryclient=bigquery.Client()
table_id="swast-scratch.to_dataframe_benchmark.tlc_green_{}pct".format(sys.argv[1])
dataframe=client.list_rows(table_id).to_dataframe(create_bqstorage_client=True)
print("Got {} rows.".format(len(dataframe.index)))

After:

swast@instance-1:~$ time python3 benchmark_bq.py 6
Got 950246 rows.
real 0m6.856s
user 0m3.644s
sys 0m1.132s
swast@instance-1:~$ time python3 benchmark_bq.py 12_5
Got 1980854 rows.
real 0m10.847s
user 0m6.012s
sys 0m1.464s
swast@instance-1:~$ time python3 benchmark_bq.py 25
Got 3963696 rows.
real 0m18.387s
user 0m11.240s
sys 0m2.564s
swast@instance-1:~$ time python3 benchmark_bq.py 50
Got 7917713 rows.
real 0m31.600s
user 0m21.752s
sys 0m4.432s

Before:

swast@instance-1:~$ time python3 benchmark_bq.py 6
Got 950246 rows.
real 0m8.255s
user 0m8.008s
sys 0m1.324s
swast@instance-1:~$ time python3 benchmark_bq.py 12_5
Got 1980854 rows.
real 0m14.623s
user 0m15.284s
sys 0m1.980s
swast@instance-1:~$ time python3 benchmark_bq.py 25
Got 3963696 rows.
real 0m26.064s
user 0m30.008s
sys 0m3.344s
swast@instance-1:~$ time python3 benchmark_bq.py 50
Got 7917713 rows.
real 0m48.729s
user 1m1.356s
sys 0m7.452s

The speedup is a bit better than what we saw in #9997 at 1.542x here. Still not quite the 2x I was seeing in the summer, though. I think the reason we're seeing a larger difference here is that we're reading from multiple streams in parallel. This makes the download time go faster and maybe more dataframes to concatenate for the final dataframe result.

[plamut]

Similarly to #9997, the observed performance gains are not huge on my 50 Mbps internet connection (network I/O consumes most of the time), but nevertheless more or less consistently reproducible.