Add `asyncio` backend by MarkusSintonen · Pull Request #930 · encode/httpcore

MarkusSintonen · 2024-06-16T18:27:20Z

Summary

Adds back native asyncio backend based on commit where it was removed here. Adjusts it to work with the new interfaces. This also adds some additional integration testing.

Doesn't make the new backend the default.

AnyIO has considerable overhead so this allows again using the native backend. Also this is going to allow removal of the AnyIO as the dependency (when the synchronization PR goes in also).

Overhead is about 1.45x here (the synchronization has a lot more overhead in the other PR).

Previously (is master):

Now:

Related discussion https://github.com/encode/httpx/issues/3215#issuecomment-2155630018.

Checklist

I understand that this PR may be closed in case there was no previous discussion. (This doesn't apply to typos!)
I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
I've updated the documentation accordingly.

MarkusSintonen · 2024-06-16T18:54:37Z

+ return backend_name, options
+
+
+class Server(uvicorn.Server):


Seems the httpbin fixture doesnt work well for pool related testing as it always closes the connections immeaditely... So adding this for testing those aspects in integration testing file

lovelydinosaur · 2024-06-17T09:50:58Z

In order to land this PR I'd suggest adding an asyncio native backend without any other changes.

The default backend should not change as part of the pull request, and the existing anyio backend should not be modified. I'd expect the only modified code files to be __init__.py and httpcore/_backends/asyncio.py.

The documentation for https://www.encode.io/httpcore/network-backends/#async-network-backends should also be modified to include the native asyncio backend, demo'ing it's usage with...

network_backend=httpcore.AsyncIOBackend()
asyncwithhttpcore.AsyncConnectionPool(network_backend=network_backend) ashttp:
...

Similar to the existing AnyIOBackend example.

(Minimal incremental changes ftw)

MarkusSintonen · 2024-06-17T12:25:14Z

The default backend should not change as part of the pull request

Yep that I didnt change as you previously suggested

MarkusSintonen · 2024-06-17T13:02:55Z

In order to land this PR I'd suggest adding an asyncio native backend without any other changes.
The default backend should not change as part of the pull request, and the existing anyio backend should not be modified. I'd expect the only modified code files to be __init__.py and httpcore/_backends/asyncio.py.

This is now done, so there is only the new backend and export. + The tests I used add the full coverage and verify it works

MarkusSintonen · 2024-06-17T13:48:48Z

The documentation for https://www.encode.io/httpcore/network-backends/#async-network-backends should also be modified to include the native asyncio backend, demo'ing it's usage with

Added this

MarkusSintonen · 2024-06-18T08:26:37Z

Asyncio-backend performs also better when all other current optimization PRs are applied (benchmark params REPEATS=100 REQUESTS=50 CONCURRENCY=20):

With AnyIO-backend:

With asyncio-backend:

lovelydinosaur · 2024-06-18T08:52:23Z

The code & docs changes look reasonable.
Not obvious that the tests cover the functionality added?

MarkusSintonen · 2024-06-18T13:15:42Z

Not obvious that the tests cover the functionality added?

Previously the @pytest.mark.anyio runned the tests as parametrized by asyncio-anyio and trio (which are the default). But now via the conftest, we specify the test parametrization to also include the plain asyncio test mode (as documented in AnyIO fixture docs https://anyio.readthedocs.io/en/stable/testing.html#specifying-the-backends-to-run-on).

With this and the added integration testing (eg UDS, socket options) we get close to 100% coverage for the new network backend. (Actually we get even more coverage for anyio/trio backends so there is some no longer needed nocoverage)

lovelydinosaur

Looks great, thanks so much.

Naming... should we use AsyncIOStream and AsyncIOBackend or AsyncioStream and AsyncioBackend?

bdraco · 2024-09-29T02:10:56Z

Thanks for doing this. I think this will close out the discussion I opened #893

MarkusSintonen · 2024-10-01T07:53:24Z

Looks great, thanks so much.
Naming... should we use AsyncIOStream and AsyncIOBackend or AsyncioStream and AsyncioBackend?

No problem! Its now renamed to AsyncIOBackend. Thanks

lovelydinosaur · 2024-10-01T10:44:06Z

Fantastic, appreciate your time on this @MarkusSintonen.
Following SEMVER we'll want this in a 1.1.0 release. (New functionality)

We could either get #957 released first then merge this, or include this now and bump the version.

MarkusSintonen · 2024-10-01T11:03:42Z

@tomchristie no problem! Would it be possible to include also the asyncio-based synchronization changes as part of this?

lovelydinosaur · 2024-10-01T11:40:18Z

+ self._read_lock = asyncio.Lock()
+ self._write_lock = asyncio.Lock()


We don't use locking at the stream layer, let's remove these.
(I think we had locking at this layer in the past, the locking is now handled at the HTTP11Connection/HTTP2Connection layer)

Done, they are now gone

lovelydinosaur · 2024-10-01T11:40:31Z

+ self._read_lock = asyncio.Lock()
+ self._write_lock = asyncio.Lock()


Suggested change
self._read_lock=asyncio.Lock()
self._write_lock=asyncio.Lock()

lovelydinosaur · 2024-10-01T11:41:24Z

+ async with self._read_lock:
+ with map_exceptions(exc_map):
+ try:
+ return await asyncio.wait_for(
+ self._stream_reader.read(max_bytes), timeout
+ )
+ except AttributeError as exc: # pragma: nocover
+ if "resume_reading" in str(exc):
+ # Python's asyncio has a bug that can occur when a
+ # connection has been closed, while it is paused.
+ # See: https://github.com/encode/httpx/issues/1213
+ #
+ # Returning an empty byte-string to indicate connection
+ # close will eventually raise an httpcore.RemoteProtocolError
+ # to the user when this goes through our HTTP parsing layer.
+ return b""
+ raise


Suggested change
asyncwithself._read_lock:
withmap_exceptions(exc_map):
try:
returnawaitasyncio.wait_for(
self._stream_reader.read(max_bytes), timeout
)
exceptAttributeErrorasexc: # pragma: nocover
if"resume_reading"instr(exc):
# Python's asyncio has a bug that can occur when a
# connection has been closed, while it is paused.
# See: https://github.com/encode/httpx/issues/1213
#
# Returning an empty byte-string to indicate connection
# close will eventually raise an httpcore.RemoteProtocolError
# to the user when this goes through our HTTP parsing layer.
returnb""
raise
withmap_exceptions(exc_map):
try:
returnawaitasyncio.wait_for(
self._stream_reader.read(max_bytes), timeout
)
exceptAttributeErrorasexc: # pragma: nocover
if"resume_reading"instr(exc):
# Python's asyncio has a bug that can occur when a
# connection has been closed, while it is paused.
# See: https://github.com/encode/httpx/issues/1213
#
# Returning an empty byte-string to indicate connection
# close will eventually raise an httpcore.RemoteProtocolError
# to the user when this goes through our HTTP parsing layer.
returnb""
raise

lovelydinosaur · 2024-10-01T11:41:47Z

+ async with self._write_lock:
+ with map_exceptions(exc_map):
+ self._stream_writer.write(data)
+ return await asyncio.wait_for(self._stream_writer.drain(), timeout)


Suggested change
asyncwithself._write_lock:
withmap_exceptions(exc_map):
self._stream_writer.write(data)
returnawaitasyncio.wait_for(self._stream_writer.drain(), timeout)
withmap_exceptions(exc_map):
self._stream_writer.write(data)
returnawaitasyncio.wait_for(self._stream_writer.drain(), timeout)

lovelydinosaur · 2024-10-01T11:46:31Z

Would it be possible to include also the asyncio-based synchronization changes as part of this?

Thanks for the ask, let's not do that. Mixing too many different concerns.
This is a self-contained PR and we can deal with it in isolation.

(Minimal incremental changes ftw)

stale · 2025-04-26T04:20:51Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

MarkusSintonen force-pushed the asyncio-backend2 branch 2 times, most recently from 0d3a1f4 to db58896Compare June 16, 2024 18:37

MarkusSintonen marked this pull request as ready for review June 16, 2024 18:51

MarkusSintonen mentioned this pull request Jun 16, 2024
Connection pool optimization: move socket polling from expiry checks to connection usage #928
Closed
3 tasks

MarkusSintonen commented Jun 16, 2024
View reviewed changes

MarkusSintonen force-pushed the asyncio-backend2 branch 3 times, most recently from 009249a to 56447f4Compare June 17, 2024 09:50

Add asyncio backend. Add integration testing
81b6b1f

MarkusSintonen force-pushed the asyncio-backend2 branch from 56447f4 to 81b6b1fCompare June 17, 2024 12:20

Simplify patch set
828c491

Update doc
b47b4c2

Add network backend benchmark param
3814cf4

MarkusSintonen force-pushed the asyncio-backend2 branch from 9f13ff5 to 3814cf4Compare June 18, 2024 12:17

Merge branch 'master' into asyncio-backend2
ac533ed

lovelydinosaur reviewed Sep 27, 2024
View reviewed changes

Comment threadhttpcore/_backends/asyncio.py Outdated

lovelydinosaur reviewed Sep 27, 2024
View reviewed changes

Rename to AsyncIOBackend
30f9c85

Merge branch 'master' into asyncio-backend2
0038a0a

lovelydinosaur reviewed Oct 1, 2024
View reviewed changes

Comment threaddocs/network-backends.md Outdated

lovelydinosaur reviewed Oct 1, 2024
View reviewed changes

Comment threaddocs/network-backends.md Outdated

lovelydinosaur reviewed Oct 1, 2024
View reviewed changes

Comment threaddocs/network-backends.md Outdated

Apply suggestions from code review
07772b1

lovelydinosaur reviewed Oct 1, 2024
View reviewed changes

Comment threadhttpcore/_backends/asyncio.py Outdated

Remove unneeded locks
ae741c0

RyanMarten mentioned this pull request Dec 4, 2024
[Bug]: Cannot get past 50 RPS BerriAI/litellm#6592
Closed

staleBot added the wontfix This will not be worked on label Apr 26, 2025

staleBot closed this Jun 27, 2025

Kludex mentioned this pull request May 12, 2026
httpx.AsyncClient has much worse performance than aiohttp.ClientSession with concurrent requests pydantic/httpx2#827
Open

		self._read_lock = asyncio.Lock()
		self._write_lock = asyncio.Lock()

	self._read_lock=asyncio.Lock()
	self._write_lock=asyncio.Lock()

-asyncwithself._write_lock:
-withmap_exceptions(exc_map):
-self._stream_writer.write(data)
-returnawaitasyncio.wait_for(self._stream_writer.drain(), timeout)
+withmap_exceptions(exc_map):
+self._stream_writer.write(data)
+returnawaitasyncio.wait_for(self._stream_writer.drain(), timeout)

Uh oh!

Conversation

MarkusSintonen commented Jun 16, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

Uh oh!

MarkusSintonenJun 16, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lovelydinosaur commented Jun 17, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MarkusSintonen commented Jun 17, 2024

Uh oh!

MarkusSintonen commented Jun 17, 2024

Uh oh!

MarkusSintonen commented Jun 17, 2024

Uh oh!

MarkusSintonen commented Jun 18, 2024

Uh oh!

lovelydinosaur commented Jun 18, 2024

Uh oh!

MarkusSintonen commented Jun 18, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

lovelydinosaur left a comment

Choose a reason for hiding this comment

Uh oh!

bdraco commented Sep 29, 2024

Uh oh!

MarkusSintonen commented Oct 1, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lovelydinosaur commented Oct 1, 2024

Uh oh!

MarkusSintonen commented Oct 1, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lovelydinosaur commented Oct 1, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

staleBot commented Apr 26, 2025

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

MarkusSintonen commented Jun 16, 2024•
edited
Loading

MarkusSintonenJun 16, 2024•
edited
Loading

lovelydinosaur commented Jun 17, 2024•
edited
Loading

MarkusSintonen commented Jun 18, 2024•
edited
Loading

lovelydinosaur commented Oct 1, 2024•
edited
Loading