gh-89550: Buffer GzipFile.write to reduce execution time by ~15% by CCLDArjun · Pull Request #101251 · python/cpython

CCLDArjun · 2023-01-22T22:47:54Z

Currently, all of this is done for every write() call:
https://github.com/python/cpython/blob/main/Lib/gzip.py#L266-L288

This pr stores GzipFile.write() data into a buffer to postpone most of the compression work to when the buffer flushes. This way, we can call into zlib and File.write() less frequently.

benchmarking with timeit:

../python-mtimeit-s 'importgzip' 'withgzip.open("./Python-3.11.1.tgz", "rb") asinfile:
' 'withgzip.open("./Python-3.11.1new.tgz", "wb") asoutfile:
' 'forlineininfile:
' 'outfile.write(line)
' '
'
1loop, bestof5: 6.36secperloop

vs

../python-mtimeit-s 'importgzipoldasgzip' 'withgzip.open("./Python-3.11.1.tgz", "rb") asinfile:
' 'withgzip.open("./Python-3.11.1new.tgz", "wb") asoutfile:
' 'forlineininfile:
' 'outfile.write(line)
' '
'
1loop, bestof5: 7.68secperloop

Issue: GzipFile.write should be buffered #89550

…cpython into buffered-gzip-writes

bedevere-bot · 2023-01-22T22:47:58Z

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

rhpvorderman · 2023-01-24T09:36:54Z

Wow @CCLDArjun such a simple and effective solution. My solution was to split out a _GzipWriter (just like _GzipReader) and try to work from there, but I never got that to work easily and it was quite some extra code too. I am so glad you made this. I hope this will get merged soon!

CCLDArjun · 2023-01-24T17:42:28Z

My solution was to split out a _GzipWriter (just like _GzipReader) and try to work from there

Initially that was my idea too, where I subclassed _pyio.BufferedWriter to override its flush methods but that also wasn't working easily and was pretty messy: 943ca9c

rhpvorderman · 2023-01-25T08:27:57Z

I am glad that you were able to come up with a better idea. I have been hacking at the gzip module pretty thoroughly but I could not come up with this. I am in awe 😀 . I will port this to python-isal and I have recently started on making a zlib-ng module too. That way people will not have to wait for 3.12 to ship to get all this fast compression.

orsenthil · 2023-01-27T04:52:02Z

This looks good to me. A core developer in the expert list will review before merging.

rhpvorderman · 2023-02-08T12:34:48Z

@orsenthil This is still labeled "awaiting review" rather then "awaiting core review" does that matter?

arhadthedev · 2023-02-08T12:40:56Z

@rhpvorderman Only review messages from the Python team count.

@Yhg1s, @gpshead as zipfile experts (not quite the same but still).

gpshead

Overall I like this change. It's great how cleanly this can be added. :)

bedevere-bot · 2023-02-18T00:13:17Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

…cpython into buffered-gzip-writes

CCLDArjun · 2023-02-22T07:44:15Z

@gpshead tests are passing! Can you re review?

rhpvorderman · 2023-05-08T14:40:54Z

ping

It would be great if this gets into 3.12. The measured execution difference of 15% is at gzip level 9 which is the default. At a more modest compression level such as 1, the difference can be a couple of hundreds of percents.

CCLDArjun · 2023-05-08T16:50:12Z

pinging @gpshead for a final review

python#101251) Use `io.BufferedWriter` to buffer gzip writes. --------- Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com> Co-authored-by: Gregory P. Smith <greg@krypto.org>

* main: (47 commits) pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188) pythonGH-104308: socket.getnameinfo should release the GIL (python#104307) pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311) pythongh-99113: A Per-Interpreter GIL! (pythongh-104210) pythonGH-104284: Fix documentation gettext build (python#104296) pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251) pythongh-104223: Fix issues with inheriting from buffer classes (python#104227) pythongh-99108: fix typo in Modules/Setup (python#104293) pythonGH-104145: Use fully-qualified cross reference types for the bisect module (python#104172) pythongh-103193: Improve `getattr_static` test coverage (python#104286) Trim trailing whitespace and test on CI (python#104275) pythongh-102500: Remove mention of bytes shorthand (python#104281) pythongh-97696: Improve and fix documentation for asyncio eager tasks (python#104256) pythongh-99108: Replace SHA3 implementation HACL* version (python#103597) pythongh-104273: Remove redundant len() calls in argparse function (python#104274) pythongh-64660: Don't hardcode Argument Clinic return converter result variable name (python#104200) pythongh-104265 Disallow instantiation of `_csv.Reader` and `_csv.Writer` (python#104266) pythonGH-102613: Improve performance of `pathlib.Path.rglob()` (pythonGH-104244) pythongh-103650: Fix perf maps address format (python#103651) pythonGH-89812: Churn `pathlib.Path` methods (pythonGH-104243) ...

* main: (29 commits) pythongh-104276: Make `_struct.unpack_iterator` type use type flag instead of custom constructor (python#104277) pythongh-97696: Move around and update the whatsnew entry for asyncio eager task factory (python#104298) pythongh-103193: Fix refleaks in `test_inspect` and `test_typing` (python#104320) require-pr-label.yml: Add missing "permissions:" (python#104309) pythongh-90656: Add platform triplets for 64-bit LoongArch (LA64) (python#30939) pythongh-104180: Read SOCKS proxies from macOS System Configuration (python#104181) pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188) pythonGH-104308: socket.getnameinfo should release the GIL (python#104307) pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311) pythongh-99113: A Per-Interpreter GIL! (pythongh-104210) pythonGH-104284: Fix documentation gettext build (python#104296) pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251) pythongh-104223: Fix issues with inheriting from buffer classes (python#104227) pythongh-99108: fix typo in Modules/Setup (python#104293) pythonGH-104145: Use fully-qualified cross reference types for the bisect module (python#104172) pythongh-103193: Improve `getattr_static` test coverage (python#104286) Trim trailing whitespace and test on CI (python#104275) pythongh-102500: Remove mention of bytes shorthand (python#104281) pythongh-97696: Improve and fix documentation for asyncio eager tasks (python#104256) pythongh-99108: Replace SHA3 implementation HACL* version (python#103597) ...

* main: (156 commits) pythongh-97696 Add documentation for get_coro() behavior with eager tasks (python#104304) pythongh-97933: (PEP 709) inline list/dict/set comprehensions (python#101441) pythongh-99889: Fix directory traversal security flaw in uu.decode() (python#104096) pythongh-104184: fix building --with-pydebug --enable-pystats (python#104217) pythongh-104139: Add itms-services to uses_netloc urllib.parse. (python#104312) pythongh-104240: return code unit metadata from codegen (python#104300) pythongh-104276: Make `_struct.unpack_iterator` type use type flag instead of custom constructor (python#104277) pythongh-97696: Move around and update the whatsnew entry for asyncio eager task factory (python#104298) pythongh-103193: Fix refleaks in `test_inspect` and `test_typing` (python#104320) require-pr-label.yml: Add missing "permissions:" (python#104309) pythongh-90656: Add platform triplets for 64-bit LoongArch (LA64) (python#30939) pythongh-104180: Read SOCKS proxies from macOS System Configuration (python#104181) pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188) pythonGH-104308: socket.getnameinfo should release the GIL (python#104307) pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311) pythongh-99113: A Per-Interpreter GIL! (pythongh-104210) pythonGH-104284: Fix documentation gettext build (python#104296) pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251) pythongh-104223: Fix issues with inheriting from buffer classes (python#104227) pythongh-99108: fix typo in Modules/Setup (python#104293) ...

* main: (35 commits) pythongh-97696 Add documentation for get_coro() behavior with eager tasks (python#104304) pythongh-97933: (PEP 709) inline list/dict/set comprehensions (python#101441) pythongh-99889: Fix directory traversal security flaw in uu.decode() (python#104096) pythongh-104184: fix building --with-pydebug --enable-pystats (python#104217) pythongh-104139: Add itms-services to uses_netloc urllib.parse. (python#104312) pythongh-104240: return code unit metadata from codegen (python#104300) pythongh-104276: Make `_struct.unpack_iterator` type use type flag instead of custom constructor (python#104277) pythongh-97696: Move around and update the whatsnew entry for asyncio eager task factory (python#104298) pythongh-103193: Fix refleaks in `test_inspect` and `test_typing` (python#104320) require-pr-label.yml: Add missing "permissions:" (python#104309) pythongh-90656: Add platform triplets for 64-bit LoongArch (LA64) (python#30939) pythongh-104180: Read SOCKS proxies from macOS System Configuration (python#104181) pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188) pythonGH-104308: socket.getnameinfo should release the GIL (python#104307) pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311) pythongh-99113: A Per-Interpreter GIL! (pythongh-104210) pythonGH-104284: Fix documentation gettext build (python#104296) pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251) pythongh-104223: Fix issues with inheriting from buffer classes (python#104227) pythongh-99108: fix typo in Modules/Setup (python#104293) ...

GzipFile.flush() to not flush the compressor (nor pass along the zip_mode argument).

…h() to not flush the compressor (nor pass along the zip_mode argument).

Fix a regression introduced in GH-101251, causing GzipFile.flush() to not flush the compressor (nor pass along the zip_mode argument).

…onGH-105910) Fix a regression introduced in pythonGH-101251, causing GzipFile.flush() to not flush the compressor (nor pass along the zip_mode argument). (cherry picked from commit 1858db7) Co-authored-by: T. Wouters <thomas@python.org>

#105920) GH-105808: Fix a regression introduced in GH-101251 (GH-105910) Fix a regression introduced in GH-101251, causing GzipFile.flush() to not flush the compressor (nor pass along the zip_mode argument). (cherry picked from commit 1858db7) Co-authored-by: T. Wouters <thomas@python.org>

CCLDArjun added 4 commits January 22, 2023 09:26

initial testing to see if it's faster
943ca9c

Merge branch 'python:main' into buffered-gzip-writes
6f4701b

use io.BufferedWriter to buffer gzip writes
8653faf

:Merge branch 'buffered-gzip-writes' ofhttps://github.com/ccldarjun/…
1c36625
…cpython into buffered-gzip-writes

bedevere-bot added the awaiting review label Jan 22, 2023

CCLDArjun changed the title ~~Buffer gzip writes to reduce execution time by ~15%~~gh-89550: Buffer GzipFile.write to reduce execution time by ~15%Jan 22, 2023

bedevere-bot mentioned this pull request Jan 22, 2023
GzipFile.write should be buffered #89550
Closed

AlexWaygood added the performance Performance or resource usage label Jan 22, 2023

news blurb
bc4d929

rhpvorderman mentioned this pull request Jan 25, 2023
Buffer compression writers pycompression/xopen#78
Merged

Merge branch 'main' into buffered-gzip-writes
ef9dd50

This was referenced Feb 6, 2023
Buffer writing gzipfile pycompression/python-isal#131
Closed
Buffered writing for GzipFile pycompression/python-zlib-ng#8
Closed

arhadthedev added the stdlib Standard Library Python modules in the Lib/ directory label Feb 8, 2023

svisser mentioned this pull request Feb 17, 2023
Slow performance due to lack of buffering for GzipFile.write piskvorky/smart_open#760
Closed

gpshead self-requested a review February 17, 2023 23:54

gpshead requested changes Feb 18, 2023
View reviewed changes

Comment threadLib/gzip.py Outdated
Comment threadLib/gzip.py Outdated
Comment threadLib/gzip.py

bedevere-bot added awaiting changes and removed awaiting review labels Feb 18, 2023

gpshead self-assigned this Feb 18, 2023

CCLDArjun added 2 commits February 17, 2023 23:15

added buffer_size for file writes
b3aa1f7

:Merge branch 'buffered-gzip-writes' ofhttps://github.com/ccldarjun/…
ad2dc9b
…cpython into buffered-gzip-writes

svisser reviewed Feb 23, 2023
View reviewed changes

Comment threadLib/gzip.py

CCLDArjun added 2 commits February 23, 2023 14:54

use constant for write buffer size
fe31588

Merge branch 'main' into buffered-gzip-writes
875ebd8

Merge branch 'main' into buffered-gzip-writes
e33fc19

improve news wording and ReSTify
5ef4674

gpshead approved these changes May 8, 2023
View reviewed changes

bedevere-bot added awaiting merge and removed awaiting change review labels May 8, 2023

Merge branch 'main' into buffered-gzip-writes
c0f3d2a

gpshead enabled auto-merge (squash) May 8, 2023 17:29

gpshead merged commit 9af4854 into python:mainMay 8, 2023

bedevere-bot removed the awaiting merge label May 8, 2023

bdarnell mentioned this pull request Jun 15, 2023
GzipFile.flush doesn't flush compressor in 3.12 beta #105808
Closed

Yhg1s added a commit to Yhg1s/cpython that referenced this pull request Jun 19, 2023
pythonGH-105808: Fix a regression introduced inpythonGH-101251, causing
b7ecf06
GzipFile.flush() to not flush the compressor (nor pass along the zip_mode argument).

Yhg1s added a commit to Yhg1s/cpython that referenced this pull request Jun 19, 2023
Fix a regression introduced inpythonGH-101251, causing GzipFile.flus…
5b3a658
…h() to not flush the compressor (nor pass along the zip_mode argument).

Yhg1s mentioned this pull request Jun 19, 2023
GH-105808: Fix a regression introduced in GH-101251 #105910
Merged

gpshead pushed a commit that referenced this pull request Jun 19, 2023
GH-105808: Fix a regression introduced inGH-101251(#105910)
1858db7
Fix a regression introduced in GH-101251, causing GzipFile.flush() to not flush the compressor (nor pass along the zip_mode argument).

miss-islington mentioned this pull request Jun 19, 2023
[3.12] GH-105808: Fix a regression introduced in GH-101251 (GH-105910) #105920
Merged

effigies mentioned this pull request Aug 18, 2023
GzipFile.seek makes invalid write if buffer is not flushed in Python 3.12rc1 #108111
Closed
2 tasks

qiyunzhu mentioned this pull request Nov 17, 2023
Scikit-bio installation with Python 3.12 causes failures when testing scikit-bio/scikit-bio#1897
Closed

Uh oh!

Uh oh!

Conversation

CCLDArjun commented Jan 22, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-bot commented Jan 22, 2023

Uh oh!

rhpvorderman commented Jan 24, 2023

Uh oh!

CCLDArjun commented Jan 24, 2023

Uh oh!

rhpvorderman commented Jan 25, 2023

Uh oh!

orsenthil commented Jan 27, 2023

Uh oh!

rhpvorderman commented Feb 8, 2023

Uh oh!

arhadthedev commented Feb 8, 2023

Uh oh!

gpshead left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bedevere-bot commented Feb 18, 2023

Uh oh!

CCLDArjun commented Feb 22, 2023

Uh oh!

Uh oh!

rhpvorderman commented May 8, 2023

Uh oh!

CCLDArjun commented May 8, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

CCLDArjun commented Jan 22, 2023•
edited
Loading