pycrc32 is a Python module for SIMD-accelerated CRC32 checksum computation.
Big thanks to rust-crc32fast - this project just provides Python bindings for their Rust implementation.
pip install pycrc32frompycrc32importcrc32data=b"123456789"print(f"crc32 for {data!r} is {crc32(data)}")For scenarios that require more flexibility, such as processing large amounts of data or computing the checksum in stages, you can use the Hasher class:
frompycrc32importHasher# Create a new Hasher instancehasher=Hasher()
# Update the hasher with data chunkshasher.update(b"123456")
hasher.update(b"789")
# Finalize the computation and get the checksumchecksum=hasher.finalize()
print(f"Checksum: {checksum}")
# Reset the hasher to compute another checksumhasher.reset()
hasher.update(b"The quick brown fox jumps over the lazy dog")
new_checksum=hasher.finalize()
print(f"New checksum: {new_checksum}")You can also initialize a Hasher with a specific initial CRC32 state:
initial_crc=12345678hasher=Hasher.with_initial(initial_crc)
hasher.update(b"additional data")
final_checksum=hasher.finalize()
print(f"Final checksum with initial state: {final_checksum}")To combine checksums from different data blocks without needing to concatenate the data, use the combine method:
hasher1=Hasher()
hasher1.update(b"Data block 1")
checksum1=hasher1.finalize()
hasher2=Hasher()
hasher2.update(b"Data block 2")
checksum2=hasher2.finalize()
# Combine checksums from hasher1 into hasher2hasher1.combine(hasher2) # Combine the state of hasher2 into hasher1# The final checksum after combinationcombined_checksum=hasher1.finalize()
print(f"Combined checksum: {combined_checksum}")The performance of pycrc32 has been benchmarked on a trusty old Intel i7-8550U using 32MB of random input data. Below is a comparison of the median computation times across different libraries:
| Library | Median Time (s) |
|---|---|
pycrc32 | 0.002703 |
zlib | 0.019796 |
fastcrc | 0.071426 |
We reach almost 10x performance improvements compared to the zlib baseline implementation.
See scripts/bench.py for more details.