increase the default chunk size for uploads

[tswast]

See: googleapis/google-resumable-media-python#238

The BigQuery client library currently defaults to 1MB chunks.

python-bigquery/google/cloud/bigquery/client.py

Line 98 in 36fe86f

_DEFAULT_CHUNKSIZE = 1048576 # 1024 * 1024 B = 1 MB

The bq CLI currently defaults to 100MB chunks (via googleapiclient). https://github.com/googleapis/google-api-python-client/blob/11d78e0fe9290d1a8c516a0e2f2e019dbd1877f9/googleapiclient/http.py#L69

Perhaps this could be the cause of the 10x performance difference for file uploads?

CC @KevinTydlacka

[tswast]

Looks like the Python GCS client defaults to 40MB, so this might not be the whole story. https://github.com/googleapis/python-storage/blob/c027ccf4279fb05e041754294f10744b7d81beea/google/cloud/storage/fileio.py#L27

[tseaver]

That is the default for the BlobReader / BlobWriter wrappers: the default for the underlying blob methods is 100 Mb:

_DEFAULT_CHUNKSIZE = 104857600 # 1024 * 1024 B * 100 = 100 MB

[tswast]

fixed by #799