Gzip fails for file over 2**32 bytes

[BenCipollini]

BPO	25626
Nosy	@Yhg1s, @vadmium, @serhiy-storchaka
Files	zlib-size_t.patch: Cap zlib to UINT_MAX zlib-size_t.2.patch zlib-Py_ssize_t.patch zlib-Py_ssize_t.2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/vadmium'
closed_at = <Date 2015-11-21.11:03:41.734>
created_at = <Date 2015-11-14.19:48:21.730>
labels = ['extension-modules', 'type-bug', 'library']
title = 'Gzip fails for file over 2**32 bytes'
updated_at = <Date 2015-11-21.11:03:41.732>
user = 'https://bugs.python.org/BenCipollini'

bugs.python.org fields:

activity = <Date 2015-11-21.11:03:41.732>
actor = 'martin.panter'
assignee = 'martin.panter'
closed = True
closed_date = <Date 2015-11-21.11:03:41.734>
closer = 'martin.panter'
components = ['Extension Modules', 'Library (Lib)']
creation = <Date 2015-11-14.19:48:21.730>
creator = 'Ben Cipollini'
dependencies = []
files = ['41053', '41090', '41096', '41107']
hgrepos = []
issue_num = 25626
keywords = ['patch', '3.5regression']
message_count = 18.0
messages = ['254668', '254684', '254686', '254717', '254727', '254754', '254865', '254884', '254887', '254891', '254955', '254963', '254972', '254976', '255032', '255044', '255048', '255049']
nosy_count = 7.0
nosy_names = ['twouters', 'nadeem.vawda', 'python-dev', 'martin.panter', 'serhiy.storchaka', 'Matthew.Brett', 'Ben Cipollini']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue25626'
versions = ['Python 3.5', 'Python 3.6']

[BenCipollini]

Gzip fails when opening a file more than 2**32 bytes. This is a new issue in Python 3.5.

We hit this opening large neuroimaging files from the Human Connectome Project. See nipy/nibabel#362 for more details.

When size is > 2**32, we get the following error on Python 3.5:

/usr/lib/python3.5/gzip.py in read(self, size)
467 buf = self._fp.read(io.DEFAULT_BUFFER_SIZE)
468
--> 469 uncompress = self._decompressor.decompress(buf, size)
470 if self._decompressor.unconsumed_tail != b"":
471 self._fp.prepend(self._decompressor.unconsumed_tail)

OverflowError: Python int too large for C unsigned int

[vadmium]

Thanks for the report. Can you confirm if this demo illustrates your problem? For me, I only have 2 GiB of memory so I get a MemoryError, which seems reasonable for my situation.

from gzip import GzipFile
from io import BytesIO
file = BytesIO()
writer = GzipFile(fileobj=file, mode="wb")
writer.write(b"data")
writer.close()
file.seek(0)
reader = GzipFile(fileobj=file, mode="rb")
data = reader.read(2**32) # Ideally this should return b"data"

Assuming that triggers the OverflowError, then the heart of the problem is that the zlib.decompressobj.decompress() method does not accept such large numbers for the length limit:

>>> import zlib
>>> decompressor = zlib.decompressobj(wbits=16 + 15)
>>> decompressor.decompress(file.getvalue(), 2**32)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
OverflowError: Python int too large for C unsigned int

I think the ideal fix would be to cap the limit at 2**32 - 1 in the zlib library. Would this be okay for a 3.5.1 bug fix release, or would it be considered a feature change?

Failing that, another option would be to cap the limit in the gzip library, and just document the zlib limitation. I already have a patch in bpo-23200 documenting another quirk when max_length=0.

The same problem may also apply to the LZMA and bzip2 modules; I need to check.

[BenCipollini]

@martin: yes, that reproduces the problem.

I think the ideal fix would be to cap the limit at 2**32 - 1 in the zlib library.

If this cap is implemented, is there any workaround how we can efficiently open gzipped files over 4GB? Such files exist, exist in high-priority, government-funded datasets, and neuroinformatics in Python requires a way to open them efficiently.

another option would be to cap the limit in the gzip library

Again, these limitations are new in Python 3.5 and currently block us from using Python 3.5 to run neuroinformatics efficiently. Is there any chance to revert this new behavior, or any known memory-efficient workaround?

[vadmium]

Ben: By adding the cap, it means when the lower-level zlib module is asked to decode say 5 GiB, it will decode at most 4 GiB (rather than raising an exception). The 4 GiB limit (UINT_MAX to be precise) is due to the underlying zlib library; it’s not Python’s doing, and the limit was present before 3.5 as well.

In 3.5, if you ask GzipFile.read() to decompress 5 GiB, it will actually feed the decompressor 8 KiB (io.DEFAULT_BUFFER_SIZE) of compressed data, which will decompress to at most 8 MB or so (max compression ratio is 1:1032). Then it will feed the next 8 KiB chunk. So it does not matter whether the limit is 4 or 5 GiB because there can only be 8ish MB data per internal decompress() call.

Before 3.5, it first feeds a 1 KiB chunk, but exponentially increases the chunk size up to 10 MiB in each internal iteration.

If you think the new 3.5 implementation (with OverflowError fixed) is significantly less efficient, I can look at ways of improving it. But I am a bit skeptical. Usually I find once you process 10s or 100s of kB of data at once, any increases in the buffer size have negligible effect on the little time spent in native Python code, and times actually get worse as your buffers get too big, probably due to ineffective use of fast memory caching.

===

I propose this patch for the 3.5 branch, which implements the internal capping in the zlib module. It changes the zlib module to accept sizes greater than UINT_MAX, but internally limits them. I think this is the most practical way forward, because it does not require Python code to know the value of UINT_MAX (which could in theory vary by platform).

I would be good to get a review of my patch. In particular, is this change okay for the 3.5 maintainence branch? It would also be good to confirm that my patch fixes the original problem, i.e. read(2**32) works on computers with more than 4 GiB of memory (which I cannot test).

This regression is caused by bpo-23529, where I make sure that GzipFile.read(size) doesn’t buffer huge amounts of data from a decompression bomb.

[serhiy-storchaka]

Is 3.4 affected? A uint_converter was added in bpo-18294.

[vadmium]

3.4 shouldn’t be affected by the gzip regression. In 3.4 the gzip module does not set max_length to limit the decompression size.

In 3.4 the underlying zlib module will also raise OverflowError if max_lenth=2**32, and my patch could be applied, but I don’t think it is worth it at this stage.

[vadmium]

I am inclined to commit my patch to get this fixed in the upcoming 3.5.1 release, unless anyone thinks that could be a problem.

[serhiy-storchaka]

Could you please add tests for other two functions? And tests for the gzip module?

[vadmium]

Okay. For the gzip module, I cannot easily test this myself. Quickly looking at other cases, I guess it would look something like this, but I need to spend some time understanging the bigmemtest decorator properly:

@unittest.skipIf(sys.maxsize < _4G, "Requires non-32-bit system")
@test.support.bigmemtest(_4G, 1, dry_run=False)
def test_large_read(self, size):
...
data = reader.read(size) # Should not raise OverflowError
self.assertEqual(data, b"data")

[serhiy-storchaka]

You can commit your patch right now (it shouldn't make things worse) and add new tests later.