gh-134635: add zlib.{adler32,crc32}_combine to combine checksums

[picnixz]

• Issue: Export zlib.{adler,crc}32_combine #134635

---

📚 Documentation preview 📚: https://cpython-previews--134650.org.readthedocs.build/

[emmatyping]

Have a few suggestions/thoughts but overall looks great!

[picnixz]

Thanks for the feedback! I'm not on my dev session so I won't be able to change the clinic-related code as I'll need to regen the files, but I'll address the rest in the meantime.

[thatch]

Overall looks good, thanks!

[picnixz]

Thanks for the reviews both of you!

[emmatyping]

Two suggestions for wording but otherwise looks great! Thank you for adding these.

[picnixz]

I've edited your suggestion a bit but thank you!

[morotti]

However, looking at the fastzip code, they are concurrently calculating the checksums and assembling them afterwards, so they cannot chain calls to zlib.crc32.

Just a note that the function is probably obsolete. The ticket goes back to 2019.
It's fine to add, though it might be misleading for suggesting that computing crc in parallel is a thing.

The python interpreter has switched to zlib-ng on windows (other OS to follow). the zlib.crc is computing crc with hardware instructions on x64, which are insanely fast. binascii.crc is just an alias to zlib.crc.
seeing up to 420GB per second on small data that fit in cpu cache on my desktop. (not a typo, it's insane!). nearly 100 GB per second otherwise.

there is little point in computing crc32 in parallel. In fact we should probably remove or adjust the crc code to not release the GIL, as it's counterproductive on small data.

[emmatyping]

@morotti

The python interpreter has switched to zlib-ng on windows (other OS to follow).

I don't think adopting zlib-ng everywhere is settled, and there is a long tail of old Linux distributions that want to link against a system zlib that do not ship zlib-ng yet. I don't think we can assume zlib-ng performance. That being said even zlib crc32 is pretty fast. However, even with a fast sequential crc32 I think this could be useful for concurrent checksuming for say, parallel chunked file downloads.

In fact we should probably remove or adjust the crc code to not release the GIL, as it's counterproductive on small data.

I would want to see convincing benchmarks before making this change and proper motivation, otherwise I'm not sure if this is worth the complexity. (if you have those though please do open an issue!)

[morotti]

hello @emmatyping and sorry for pinging this old ticket

That being said even zlib crc32 is pretty fast.

Nope, zlib crc32 is horribly slow. I've been working on optimizing pip and whole percents of the duration of "pip install" is computing crc!

You can find benchmarks (both compression and crc) in the zlib-ng thread:
TL;DR around 20GB/s crc on slow laptop, 80GB/s crc on fast desktop, up to 420GB/s on very small data that can fit in cache.
#91349 (comment)
#91349 (comment)

I don't think adopting zlib-ng everywhere is settled, and there is a long tail of old Linux distributions that want to link against a system zlib that do not ship zlib-ng yet. I don't think we can assume zlib-ng performance.

See the thread, it's settled and there is agreement to do it.
The issue is nobody has figured out how to modify the build system on Linux/MacOS :D (and even if modified, some distributions will dynamic link the library and load zlib from the OS instead, as you say)

Anyway, we can get better crc function aside of zlib/zlib-ng. The python interpreter can use the crc32 function from liblzma instead.
This PR is exposing the function #131721