BigQuery upload_from_file unicode file-like must be opened in binary-mode if it's more than RESUMABLE_UPLOAD_THRESHOLD, otherwise str-mode.

[joar]

Steps to reproduce

import sys
from gcloud.bigquery import Client, SchemaField
from gcloud.bigquery.job import CreateDisposition, WriteDisposition

csv_filename = 'sandwiches.csv'

if len(sys.argv) > 1:
 csv_filename = sys.argv[1]

bq = Client()

ds = bq.dataset('test_unicode')
ds.location = 'EU'


def test_unicode_upload(filename, mode):

 if not ds.exists():
 print('Creating dataset: {}'.format(ds.name))
 ds.create()

 fields = [
 SchemaField('name', 'STRING'),
 SchemaField('main_ingredient', 'STRING'),
 ]

 table = ds.table('sandwiches', fields)

 print('Uploading CSV: {}, mode={!r}'.format(csv_filename, mode))
 table.upload_from_file(
 open(filename, mode),
 encoding='UTF-8',
 source_format='CSV',
 write_disposition=WriteDisposition.WRITE_TRUNCATE,
 create_disposition=CreateDisposition.CREATE_IF_NEEDED)

test_unicode_upload(csv_filename, 'r') # Works

test_unicode_upload(csv_filename, 'rb')
# Fails in http.client.HTTPConnection()._send_request
#
# /usr/lib/python3.4/http/client.py in _send_request(self, method, url,
# body, headers)
# 1178 if isinstance(body, str):
# 1179 # RFC 2616 Section 3.7.1 says that text default has a
# 1180 # default charset of iso-8859-1.
# -> 1181 body = body.encode('iso-8859-1')
# 1182 self.endheaders(body)
#
# UnicodeEncodeError: 'latin-1' codec can't encode characters in position
#649-650: ordinal not in range(256)

sandwiches.csv

name,main_ingredient
Räksmörgås,Räkor
Baguette,Bröd

Expected behavior

When i send in a binary-mode file-like I expect upload_from_file to pass the data through to BigQuery as-is, and that the BigQuery load job will decode it for me using encoding=.

Enviroment

$ python --version
Python 3.4.3+
$ pip freeze | egrep 'httplib2|gcloud'
gcloud==0.13.0
httplib2==0.9.2

[dhermes]

It seems the binary file object is just getting passed off to our vendored in streaming package; not quite sure how it handles the encoding / what the right answer is.

Thanks a lot for reporting!

[dhermes]

If it helps, I stepped into the debugger and found that the characters get mangled before being sent:

ipdb> print(body)
--===============4443280759287164299==
Content-Type: application/json
MIME-Version: 1.0

{"configuration": {"load": {"createDisposition": "CREATE_IF_NEEDED", "sourceFormat": "CSV", "writeDisposition": "WRITE_TRUNCATE", "schema": {"fields": [{"mode": "NULLABLE", "name": "name", "type": "STRING"}, {"mode": "NULLABLE", "name": "main_ingredient", "type": "STRING"}]}, "encoding": "UTF-8", "destinationTable": {"tableId": "sandwiches", "datasetId": "test_unicode", "projectId": "PROJECT"}}}}
--===============4443280759287164299==
Content-Type: application/octet-stream
MIME-Version: 1.0
Content-Transfer-Encoding: binary

name,main_ingredient
R��ksm��rg��s,R��kor
Baguette,Br��d

--===============4443280759287164299==--

e.g.

ipdb> body[-78:-43]
'R��ksm��rg��s,R��kor\nBaguette,Br��d'
ipdb> body[-77]
'�'
ipdb> type(body[-77])
<class 'str'>
ipdb> body[-77].encode('utf-8')
b'\xef\xbf\xbd'
ipdb> body[-76]
'�'
ipdb> body[-76].encode('utf-8')
b'\xef\xbf\xbd'

[joar]

@dhermes It seem that the file must be read in binary-mode when the upload strategy is RESUMABLE_UPLOAD, and in str-mode otherwise.

[joar]

Here's a new script to reproduce

import os
import sys
from itertools import product

import math
from gcloud.streaming.transfer import RESUMABLE_UPLOAD_THRESHOLD
from gcloud.bigquery import Client, SchemaField
from gcloud.bigquery.job import CreateDisposition, WriteDisposition

csv_filename = 'sandwiches.csv'
large_csv_filename = 'large-sandwiches.csv'

if len(sys.argv) > 1:
 csv_filename = sys.argv[1]

bq = Client()

ds = bq.dataset('test_unicode')
ds.location = 'EU'


def create_large_sandwiches(in_path, out_path):
 target_size = RESUMABLE_UPLOAD_THRESHOLD

 # Assume we get all the content
 in_fd = open(in_path, 'rb')
 in_bytes = in_fd.read() # type: bytes

 in_fd.seek(0)
 lines = in_fd.readlines()
 last_line = lines[-1]

 repeat = math.ceil((target_size - len(in_bytes)) / len(last_line))

 open(out_path, 'wb').writelines(last_line for _ in range(0, repeat))


def test_unicode_upload(filename, mode):
 if not ds.exists():
 print('Creating dataset: {}'.format(ds.name))
 ds.create()

 fields = [
 SchemaField('name', 'STRING'),
 SchemaField('main_ingredient', 'STRING'),
 ]

 table = ds.table('sandwiches', fields)

 stat = os.stat(filename)

 _log.info('Uploading CSV: filename=%r, mode=%r, st_size=%d',
 filename, mode, stat.st_size)
 table.upload_from_file(
 open(filename, mode),
 encoding='UTF-8',
 source_format='CSV',
 write_disposition=WriteDisposition.WRITE_TRUNCATE,
 create_disposition=CreateDisposition.CREATE_IF_NEEDED)


if __name__ == '__main__':
 import logging
 logging.basicConfig(level=logging.INFO)
 _log = logging.getLogger(__name__)

 create_large_sandwiches(csv_filename, large_csv_filename)

 for args in product([csv_filename, large_csv_filename], ['r', 'rb']):
 try:
 test_unicode_upload(*args)
 except:
 _log.exception('Upload failed')

[dhermes]

@tseaver Can you PTAL? I assume you have more familiarity with gcloud.streaming / apitools than I do.

[tseaver]

Logically, the stream should be opened in binary mode: the back-end requires that it be encoded as either UTF-8 or ISO-8859-1, and assumes UTF-8 if no value is passed.

For multipart uploads (those smaller than RESUMABLE_UPLOAD_THRESHOLD), our vendored-in fork handles the stream precisely as apitools does.

@craigcitro, have you seen issues with multipart uploads blowing up when passed non-ASCII streams opened in binary mode?

[craigcitro]

I have not -- but @thobrla is much more likely to have run into this kind of thing. Travis, does this sound familiar?

[thobrla]

Not familiar - gsutil opens all files as binary.

[craigcitro]

@dhermes if you've got an easy repro on hand -- can you set a breakpoint where @tseaver pointed in transfer.py and look at self.stream and http_request? eg is the bad translation happening in that chunk of transfer, or is something getting mismatched earlier?

[tseaver]

@thobrla does gsutil do anything to force small uploads to use the resumable strategy? Or does it ever trigger the "mulitpart" upload (metadata + binary file)?