gh-91349: Expose the crc32 function from the lzma library

[morotti]

Hello,

This is a simple PR to expose the crc32 function from the lzma library.
The code and tests are very similar to the crc in binascii and zlib. I don't think it's too controversial.

I came across this while looking at compression libraries for #91349
crc32 is always available from the lzma library (xz-utils/liblzma.so). I think it should be exposed.
there are more hashing functions available (namely crc64) but they need compilation/runtime checks to verify whether they are available.

Regards.

• Issue: Use zlib-ng (fast!) rather than mainline stale zlib in binary releases #91349

---

📚 Documentation preview 📚: https://cpython-previews--131721.org.readthedocs.build/

[morotti]

sorry for delay,
@merwok@AA-Turner I started over from the main branch and made all the adjustments.

all the docstring/comments were copy pasted from the zlib function. I guess the old function had an old style that is no longer wanted.

for the return type in the docstring, we can settle with "positive integer"

https://docs.python.org/3/library/zlib.html#zlib.crc32
the docstring on zlib.crc32 says "The result is an unsigned 32-bit integer." but there is no concept of unsigned or 32 bits for python int.

[AA-Turner]

Thanks!

[morotti]

@AA-Turner , I merged all your suggestions.

for the versionadded: next, I thought changes on main would ship in 3.14? I am too late for 3.14 release?

[AA-Turner]

'next' is updated by the Release Manager for each version, it's a convenience tool that we use internally. We can still get this in for 3.14, yes.

A

[morotti]

regarding performance, don't mind that for now, I will come back later.

in principle

• zlib is abandoned and the slowest.
• liblzma is 2 or 3 times faster.
• zlib-ng is two orders of magnitude faster. it's using hardware instructions.

liblzma had code to start using hardware crc instructions soon ... and I see they just made a new stable release few days ago https://github.com/tukaani-project/xz/releases
I will check what's better and come back with more PR to use the best available crc function.

[morotti]

it appears the Ubuntu (free-threading) / build and test (ubuntu-24.04-arm is flaky

[AA-Turner]

I'll re-run it.

[morotti]

build passed, ready to merge

[AA-Turner]

Do you want to include your man.com email in the commit? Currently GH lists it as a co-authored-by, but the commit itself will be under your GMail address.

[morotti]

Please don't mind the email.
My work and my personal emails can both show up depending how the commits and PRs are opened. It doesn't matter which one shows up. My organization signed the CLA for the cpython interpreter, I have permissions to contribute.

[morotti]

ping, can this be merged?

[morotti]

@merwok@AA-Turner are you able to review?

I've rebased on main and redid the chance, since it's been 3 months and main had changed.

[morotti]

@gpshead you've replied multiple times to my questions on improving compression code with "PR welcome". maybe you would be interested in reviewing this PR?

[github-actions]

This PR is stale because it has been open for 30 days with no activity.

[merwok]

I’m a little confused by the issue linked from this PR.

The issue is about using zlib-ng as the underlying library for our zlib module.

This PR exposes a function from lzma. What is the link to zlib-ng or zlib module?

[morotti]

I’m a little confused by the issue linked from this PR.

The issue is about using zlib-ng as the underlying library for our zlib module.

This PR exposes a function from lzma. What is the link to zlib-ng or zlib module?

Hello,

(first part)
This PR is to expose the crc32 function from the lzma library, as stated in the title. (I hope you know what's a CRC and I don't need to go this far in the explanation ^^).

It's pretty simple. There is no downside or issue that can come from exposing this simple function that's been stable for decades.

(second part)
In the bigger scheme of things, zlib also has a CRC function.
zlib compression and lzma compression are using the same CRC algorithm, the result of both functions is the same.

zlib has been unmaintained for most of the decade, it's not using hardware instructions to calculate the CRC and it's never gonna be updated. It started being replaced by zlib-ng for massive performance improvements, see the linked issue, unfortunately there are plenty of systems/OS that will remain stuck on zlib for a long time.
liblzma is maintained and using hardware instructions, it's 10 to 100 times faster than zlib to calculate the same CRC.

Once the lzma crc function is exposed, this will allow applications to use it instead of the slow zlib crc.
Then we can do a follow up PR to use the crc function from lzma instead of zlib. (it's used in a few places like binascii, zipfile, tarfile).

[gpshead]

We do not want to increase our reliance on liblzma due to its license and general trust in the project. It'd be far easier to add our own fast internal crc32 implementation if that is desired at this point.

[morotti]

We do not want to increase our reliance on liblzma due to its license and general trust in the project. It'd be far easier to add our own fast internal crc32 implementation if that is desired at this point.

I'm sorry but this replies makes no sense whatsoever.

How is there any problem with xz-utils license or trust? especially in the context of one CRC function? It's free software and it's the baseline of multiple Linux distributions to distribute Linux packages.
(If you're referring to a malicious bug that was introduced years ago, this is not worse than people trying to put vulnerabilities into the kernel or other major projects occasionally, and this has no relevance to exposing a CRC function).

Python making its own CRC implementation is not on the table, this would require to write advanced C code with intrinsic/assembly per platform, which is completely outside of the scope of python, where maintainers can't even agree on the signature of one function. It's absurd to claim that the code should be in Python.
The optimized CRC code belongs in zlib-ng and liblzma. It should simply be exposed by python.

EDIT: if the concern is not wanting the python stdlib (binascii and co) to use lzma.crc because zlib.crc is more free, I'm happy to just expose the function, and not any make further PRs to the interpreter to use it.

[thesamesam]

We do not want to increase our reliance on liblzma due to its license and general trust in the project. It'd be far easier to add our own fast internal crc32 implementation if that is desired at this point.

Sorry, I don't understand this. I have absolutely no stake in this bug, but:

1. the licence isn't public domain anymore, it's 0BSD since a few years ago, because some regions don't recognise public domain (at least consistently). I assume that's the issue you're referring to?
2. I don't think the trust comment is fair. Extended social engineering of the maintainer could happen to any project, and singling out xz doesn't feel on. There are dependencies that CPython has which are seemingly maintained by one person, but I wouldn't lobby CPython to drop those either.

Feel free to reject it on other rationale (as I said, I have no opinion on this) but I don't think these two sound right. If there's something that we need to do more on the xz side, please let me know as well.

[gpshead]

Sorry, a quick search was showing me GPL, looking at the repo its a big mix of things and the strictly liblzma part is not GPL.

regardless, my larger point is that we should not depend on a compression library for crc32 - it isn't the primary purpose of the library. we should just ship our own performant functions.