Use standard libary Zstandard for Python 3.14+

[mollymorphous]

Summary

PEP 784 add a Zstandard implementation to the Python standard library under compression.zstd, and is scheduled for release in Python 3.14. This PR adapts ZStandardDecoder to work with either the standard library implementation or the implementation from the zstandard package.

This has the implication that Zstandard content decoding is available by default on Python 3.14 and later, without the need to install the zstd extra.

Checklist

• I understand that this PR may be closed in case there was no previous discussion. (This doesn't apply to typos!)
• I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • Testing this requires Python 3.14, but the existing zstd unit tests pass with the standard library implementation. I'm always happy to add more tests if needed!
• I've updated the documentation accordingly.

[lovelydinosaur]

Ooh interesting, thanks.

Any idea on how widely zstd is currently supported? (Wrt both browsers and servers.)

Related to this... compression is one of the currently unimplemented features in the httpx 1.0 prerelease... https://www.encode.io/httpnext/

[lovelydinosaur]

Could you explain this part?

[mollymorphous]

This is admittedly a little awkward. Python upstreamed the pyzstd package into compress.zstd because it's API was closer to existing standard library compression APIs. The zstandard package provides a ZstdDecompressObj facade that implements the standard library style API.

An alternative would be to import the libraries under separate names so the method would look more like this:

def_new_compressor(self) ->None:
ifcompression_zstdisnotNone:
self.decompressor=compression_zstd.ZstdDecompressor()
else:
self.decompressor=zstandard.ZstdDecompressor().decompressobj()

What do you think?

[ddelange]

if the requirement in pyproject.toml is changed to the 3.9+ backport at backports.zstd published by the pyzstd maintainer (which got moved into cpython 3.14 standard library):

backports-zstd==1.0.0 ; python_version < "3.14"

this diff would get much simpler right?

maybe with a simpler diff, this PR has a better chance of landing?

reading from chunked streams gets streamlined like that: aio-libs/aiohttp@df8ad83

[lilydjwg]

Any idea on how widely zstd is currently supported? (Wrt both browsers and servers.)

At least crates.io, packagist and sourceforge support it. (My tests blow up because I cache http responses and there are zstd responses from other Python versions; the "zstandard" module doesn't support 3.14 yet.)

[mollymorphous]

Thanks for the review! Firefox and Chrome currently support zstd (caniuse). Safari does not yet, but plans to: WebKit/standards-positions#168

I wasn't able to find a hard number on server-side deployments, the answer seems to be not a lot, but CloudFlare recently added support to the CDN.

[mollymorphous]

This is admittedly a little awkward. Python upstreamed the pyzstd package into compress.zstd because it's API was closer to existing standard library compression APIs. The zstandard package provides a ZstdDecompressObj facade that implements the standard library style API.

An alternative would be to import the libraries under separate names so the method would look more like this:

def_new_compressor(self) ->None:
ifcompression_zstdisnotNone:
self.decompressor=compression_zstd.ZstdDecompressor()
else:
self.decompressor=zstandard.ZstdDecompressor().decompressobj()

What do you think?

[lovelydinosaur]

From a bit of time reviewing this I've not been able to track down good examples of URLs to use for comparison purposes here.

Eg...

• CloudFlare blog pages don't appear to use compression for hosted images.
• GitHub pages sites don't appear to use compression for hosted images.
• WikiPedia supports gzip compression throughout.

Here's one example of a URL that does support zstd, though in this particular case it appears less efficient than gzip...

$ curl https://help.netflix.com/en/node/30081 -H "Accept-Encoding: br" --output netflix.br $ curl https://help.netflix.com/en/node/30081 -H "Accept-Encoding: gzip" --output netflix.gzip
$ curl https://help.netflix.com/en/node/30081 -H "Accept-Encoding: zstd" --output netflix.zstd
$ curl https://help.netflix.com/en/node/30081 -H "Accept-Encoding: identity" --output netflix
$ wc -c netflix*
84920 netflix
84920 netflix.br # Not supported on this URL
12602 netflix.gzip
16761 netflix.zstd

The CloudFlare pitch for zstd https://blog.cloudflare.com/new-standards/ isn't neccessarily convincing... gzip is essentially just as fast, and looks to have slightly less efficient though notably more stable compression ratios.

I'm reviewing this for the purposes of httpx 1.0, and I'm expecting that only supporting gzip might be a reasonable default.

Does anyone have some useful real world examples that'd help verify if that is / isn't a good policy?

[cclauss]

I suggest adding Python 3.14 to the test suite as in:

• GitHub Actions: Add Python 3.14 to test matrix #3645

[tuffnatty]

I would suggest dropping zstandard and switch to stdlib-compatible backports.zstd on Python 3.9-3.13, as httpx has dropped Python 3.8 support already.

[tuffnatty]

The CloudFlare pitch for zstd https://blog.cloudflare.com/new-standards/ isn't neccessarily convincing... gzip is essentially just as fast, and looks to have slightly less efficient though notably more stable compression ratios.

gzip is definitely not just as fast (without hardware offloading), it's just that they measure the whole response time, where the compression speed difference does not seem to matter much on average.

I'm reviewing this for the purposes of httpx 1.0, and I'm expecting that only supporting gzip might be a reasonable default.

Does anyone have some useful real world examples that'd help verify if that is / isn't a good policy?

I had used this to stream huge JSONL chunks over HTTP, and while it does not make traffic much less or a great overall speed improvement, the CPU load situation has improved a lot (relative to gzip).

[cclauss]

@tuffnatty Would you be willing to create an alternative pull request that uses the backport and adds automated tests on Py3.14 like

• GitHub Actions: Add Python 3.14 to test matrix #3645

[lovelydinosaur]

Okay, so my review of this was that supporting gzip only would be a sensible policy.
That's what we'll go for in 1.0. Let's not spend any more time rejigging zstd here.

[lilydjwg]

Okay, so my review of this was that supporting gzip only would be a sensible policy.
That's what we'll go for in 1.0. Let's not spend any more time rejigging zstd here.

Would there be a mechanism to add support for other compression methods via third-party code then? I'm a bit worried about interoperability with not-so-good servers and proxies.

[lovelydinosaur]

That's a good question... ...I can't answer that fully at the moment.

There might not be any API explicitly for that purpose. Here's how interop. with the streams class would be...

# A custom stream on top of the streams API...classZstdStream(httpx.Stream):
def__init__(self, stream: httpx.Stream):
self._wrapped=stream# Implement `.read()` and `.close()`# Usage...stream=ZStdStream(response.stream)
body=stream.read()

We probably don't want specific dials to "support for other compression methods via third-party code", since the less config the better. However we do want the tooling itself to be flexible enough to support customisation.

There's a current example in the docs demo'ing a custom client which is vastly more simple than with httpx 0.28. Perhaps building out that example to also demo custom response classes would be a good way to point users towards adaptability without increasing API surface area.

[tuffnatty]

@tuffnatty Would you be willing to create an alternative pull request that uses the backport

@cclauss Thanks but the issue has been resolved in another way.

[ddelange]

Okay, so my review of this was that supporting gzip only would be a sensible policy. That's what we'll go for in 1.0. Let's not spend any more time rejigging zstd here.

fwiw, the major alternatives to httpx support zstd (and for sure brotli):

• aiohttp
• urllib3
  • requests

supporting only gzip/deflate might be insufficient in 2025. especially brotli accounts for 33% of compressed http responses already in 2021 (Figure 22.4) and 44% of compressed javascript versus 41% gzip in 2024 (Figure 1.21)

[cclauss]

the issue has been resolved in another way.

How was the issue resolved?

[tuffnatty]

the issue has been resolved in another way.

How was the issue resolved?

#3613 (comment)

[ddelange]

disregard my last comment, I see that they're all supported 👍