Fix support for connection Upgrade and CONNECT when some data in the stream has been read.

[MtkN1]

Summary

• ClosesSupport connection Upgrade and CONNECT. #872

Checklist

• I understand that this PR may be closed in case there was no previous discussion. (This doesn't apply to typos!)
• I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
• I've updated the documentation accordingly.

[MtkN1]

I have created a draft pull request to begin the work!
First, Upgrade/CONNECT (incorrectly) raises an exception.

My understanding is that the following work needs to be done...

1. Implement a network stream that can handle trailing data like this advice (Trailing Data in WebSocket Upgrade Requests #871 (comment))
2. Generates its network stream at the step that raised the exception
3. Switch variables to be stored in network_stream.
   

   httpcore/httpcore/_async/http11.py

   Line 125 in 79fa6bf

   "network_stream": self._network_stream,

@tomchristie I would appreciate any advice you can give me!

[lovelydinosaur]

Ah fantastic thanks, shall we start with a test?...

1. There's an existing Upgrade test here, that could be replicated into a new test_upgrade_with_trailing_data test case.

httpcore/tests/_async/test_http11.py

Line 238 in 79fa6bf

asyncdeftest_http11_upgrade_connection():

I think the next step after that will be...

2. We probably want to change receive_response_headers so that it also returns the trailing_data as part of the response info, include the trailing data in the raised exception.

And then...

3. We'll want an HTTPUpgradeStream(SyncNetworkStream) class. This have an __init__ method taking a network stream and trailing data. Most of the methods should delegate to the underlying network stream, except read which should initially return the trailing data. We'll return that in the extensions for the Upgrade/CONNECT cases.

[MtkN1]

Thank you for your guidance 🙏 I'll give it a try.

[MtkN1]

Work completed, I marked PR as ready. Could you please review this?

• Added HTTP11UpgradeStream.
  • The read() method allows trailing data to be received.
  • Other methods bypass the original NetworkStream.
• For Upgrade and CONNECT, switch the network_stream extension to HTTP11UpgradeStream.
• Added tests. 100% coverage.
• I confirm that this fix resolves my discussion (Trailing Data in WebSocket Upgrade Requests #871 (comment)) and httpx-ws issue (Connection hangs forever if the WebSocket server just sends a message and close frankie567/httpx-ws#7 (comment)).
• I am not adding documentation because this change would be the behavior the user expects.

[lovelydinosaur]

This one is a fantastic fix, thanks. ✨
Minor suggestion wrt. the tests.

---

Later follow-ups we can consider...

• Should all HTTP requests include this behaviour or only Upgrade/CONNECT?
• Should all HTTP requests return network_stream or only Upgrade/CONNECT?
• What might a nice __repr__ be here?

(I'm just jotting these down here for reference)

[T-256]

LGTM, Thank You!

[lovelydinosaur]

Nice work.

[MtkN1]

Thank you for your approval 🚀 It is a pleasure to contribute to this project.