Download Fails Gzip Decompression

[jjangsangy]

I have files that have been uploaded using the gcloud command line interface with the -z flag which applies gzip content-encoding on files during transfer.

gcloud -m cp -z text gs://<bucket>

It appears that running the blob.download_to_file() and blob.download_to_string() raise a httplib2.FailedToDecompressContent exception on about 10% of the files.

Traceback (most recent call last):
 File "manage.py", line 10, in <module>
 execute_from_command_line(sys.argv)
 File "/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
 utility.execute()
 File "/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 346, in execute
 self.fetch_command(subcommand).run_from_argv(self.argv)
 File "/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/django/core/management/base.py", line 394, in run_from_argv
 self.execute(*args, **cmd_options)
 File "/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/django/core/management/base.py", line 445, in execute
 output = self.handle(*args, **options)
 File "/Users/sanghan/Projects/memexcadastre/download/management/commands/download.py", line 44, in handle
 file.download_to_filename(filename)
 File "/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/gcloud/storage/blob.py", line 329, in download_to_filename
 self.download_to_file(file_obj, client=client)
 File "/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/gcloud/storage/blob.py", line 314, in download_to_file
 download.initialize_download(request, client._connection.http)
 File "/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/gcloud/streaming/transfer.py", line 347, in initialize_download
 self.bytes_http or http, http_request)
 File "/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/gcloud/streaming/http_wrapper.py", line 405, in make_api_request
 check_response_func=check_response_func)
 File "/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/gcloud/streaming/http_wrapper.py", line 353, in _make_api_request_no_retry
 redirections=redirections, connection_type=connection_type)
 File "/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/oauth2client/client.py", line 622, in new_request
 redirections, connection_type)
 File "/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/httplib2/__init__.py", line 1609, in request
 (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
 File "/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/httplib2/__init__.py", line 1351, in _request
 (response, content) = self._conn_request(conn, request_uri, method, body, headers)
 File "/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/httplib2/__init__.py", line 1337, in _conn_request
 content = _decompressContent(response, content)
 File "/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/httplib2/__init__.py", line 410, in _decompressContent
 raise FailedToDecompressContent(_("Content purported to be compressed with %s but failed to decompress.") % response.get('content-encoding'), response, content)
httplib2.FailedToDecompressContent: Content purported to be compressed with gzip but failed to decompress.

Here is some context on the platform I'm running this on.

>>> import pkg_resources
>>> pkg_resources.get_build_platform()
'macosx-10.11-x86_64'

$ pip show gcloud

Metadata-Version: 2.0
Name: gcloud
Version: 0.12.0
Summary: API Client library for Google Cloud
Home-page: https://github.com/GoogleCloudPlatform/gcloud-python
Author: Google Cloud Platform
Author-email: jjg+gcloud-python@google.com
Installer: pip
License: Apache 2.0
Location: /Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages
Requires: pyOpenSSL, six, httplib2, protobuf, oauth2client, googleapis-common-protos

$ python --version
Python 2.7.11

The files that do decompress correctly seem to be successful in decompressing, but the ones that fail also fail every time. This also seems to be filesize independent as I have both large and files successfully decompress on download.

Thanks!

[dhermes]

Whoa. That seems really not good. I'll try to reproduce.

We're slowly trying to move away from httplib2 (see #1214). You might try to use httplib2shim suggested there and see if it resolves this issue.

I'm going to upload some gzip-ed files and see if I can get the error. Can you also provide gcloud --version in case the issue actually is the content of the files blobs?

[jjangsangy]

Sure

gcloud --version
Google Cloud SDK 105.0.0

alpha 2016.01.12
app-engine-java 1.9.34
app-engine-python 1.9.35
beta 2016.01.12
bq 2.0.24
bq-nix 2.0.24
core 2016.04.11
core-nix 2016.03.28
gcd-emulator v1beta3-1.0.0
gcloud
gsutil 4.18
gsutil-nix 4.18
kubectl
kubectl-darwin-x86_64 1.2.0

[dhermes]

I don't use the gcloud CLI very often, was gcloud -m cp -z text gs://<bucket> the actual command you used?

[jjangsangy]

Oh there is a typo, it should be. gsutil -m cp -z text *.txt gs://<bucket>, but more or less the same.

Here is an exerp from running the help dialog that was what I used to figure out what all the flags did.

$ gsutil cp --help


 -z <ext,...> Applies gzip content-encoding to file uploads with the given
 extensions. This is useful when uploading files with
 compressible content (such as .js, .css, or .html files)
 because it saves network bandwidth and space in Google Cloud
 Storage, which in turn reduces storage costs.

 When you specify the -z option, the data from your files is
 compressed before it is uploaded, but your actual files are
 left uncompressed on the local disk. The uploaded objects
 retain the Content-Type and name of the original files but are
 given a Content-Encoding header with the value "gzip" to
 indicate that the object data stored are compressed on the
 Google Cloud Storage servers.

 For example, the following command:

 gsutil cp -z html -a public-read cattypes.html gs://mycats

 will do all of the following:

 - Upload as the object gs://mycats/cattypes.html (cp command)
 - Set the Content-Type to text/html (based on file extension)
 - Compress the data in the file cattypes.html (-z option)
 - Set the Content-Encoding to gzip (-z option)
 - Set the ACL to public-read (-a option)
 - If a user tries to view cattypes.html in a browser, the
 browser will know to uncompress the data based on the
 Content-Encoding header, and to render it as HTML based on
 the Content-Type header.

 Note that if you download an object with Content-Encoding:gzip
 gsutil will decompress the content before writing the local
 file.

[dhermes]

@jjangsangy Is this deterministic? I.e. if you upload the same file twice, does the newly created blob become a failed blob?

[dhermes]

I used a script to generate a bunch of random text files:

import string
import random

import six


MIN_SIZE = 1024
MAX_SIZE = 65536
NUM_FILES = 256


def main():
 for index in six.moves.xrange(NUM_FILES):
 file_size = random.randint(MIN_SIZE, MAX_SIZE)
 contents = [random.choice(string.printable)
 for _ in six.moves.xrange(file_size)]
 filename = 'foo%04d.txt' % (index,)
 with open(filename, 'wb') as file_obj:
 file_obj.write(''.join(contents))

and then uploaded via

gsutil -m cp -z text *.txt gs://<bucket>

then tried to reproduce the failure via

import gcloud.storage
client = gcloud.storage.Client(project='<foo>')
bucket = client.bucket('<bucket>')
for blob in bucket.list_blobs():
 just_check = blob.download_as_string()

but it succeeded 100% of the time. I didn't try to repro on a Mac, so maybe that's the problem? Or maybe it's the specific contents of 10% of your files?

[jjangsangy]

Sure I'll try to create a new batch of files and see if I can reproduce.

[jjangsangy]

Yes, I was able to reproduce the result.

 File "/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/httplib2/__init__.py", line 1337, in _conn_request
 content = _decompressContent(response, content)
 File "/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/httplib2/__init__.py", line 410, in _decompressContent
 raise FailedToDecompressContent(_("Content purported to be compressed with %s but failed to decompress.") % response.get('content-encoding'), response, content)
httplib2.FailedToDecompressContent: Content purported to be compressed with gzip but failed to decompress.

I also ran this through a try:except block and It looks like it is deterministic and the same files are not being decompressed.

[dhermes]

Could you provide an example file for me to reproduce with? You can email it to me if need be.

[jjangsangy]

I would like to, but the nature of this data is sensitive.

I can try reproducing the issue with some other files that do not have this restriction and send them to you.

[dhermes]

OK thanks. Are the text files ASCII or UTF-8? You could do something like:

with open('foo.txt', 'r') as file_obj:
 all_chars = set(file_obj.read())

on each file and then see if your failing files have novel characters that aren't in the others by taking a set difference.

[jjangsangy]

@dhermes so I did find a file, it is data from a public domain so I can disclose it. What would be the best way of getting it to you?

[dhermes]

You could upload it as a gist: https://gist.github.com/