Parity with requests: support https:// proxied over https://

[mborsetti]

Executing the following using HTTPX:

import httpx
response = httpx.get('https://api.ipify.org', proxies='https://proxyname.com')
print(response.status_code)

Throws a RuntimeError ("not supported by httpcore package"):

Traceback (most recent call last):
File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.1.3\plugins\python-ce\helpers\pydev\pydevd.py", line 1500, in _exec
pydev_imports.execfile(file, globals, locals) # execute the script
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.1.3\plugins\python-ce\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "C:\Users\anonymous\AppData\Roaming\JetBrains\PyCharmCE2023.2\scratches\scratch_2.py", line 4, in <module>
response = httpx.get('https://api.ipify.org', proxies='https://proxyname.com')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_api.py", line 189, in get
return request(
^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_api.py", line 100, in request
return client.request(
^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 814, in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 901, in send
response = self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 929, in _send_handling_auth
response = self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 966, in _send_handling_redirects
response = self._send_single_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 1002, in _send_single_request
response = transport.handle_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_transports\default.py", line 228, in handle_request
resp = self._pool.handle_request(req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpcore\_sync\connection_pool.py", line 262, in handle_request
raise exc
File "C:\Program Files\Python\Python312\Lib\site-packages\httpcore\_sync\connection_pool.py", line 245, in handle_request
response = connection.handle_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpcore\_sync\http_proxy.py", line 317, in handle_request
stream = stream.start_tls(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpcore\_backends\sync.py", line 149, in start_tls
raise RuntimeError(
RuntimeError: Attempted to add a TLS layer on top of the existing TLS stream, which is not supported by httpcore package

Similar code using requests works just fine:

import requests
response = requests.get('https://api.ipify.org', proxies={'https': 'https://proxyname.com'})
print(response.status_code)

Result:

200

[karpetrosyan]

Just upgrade your httpx version to 0.25.0.

[mborsetti]

Hello @karpetrosyan,

Thanks for your reply.

Just upgrade your httpx version to 0.25.0.

I am running 0.25.0, as confirmed by the below code:

import httpx
response = httpx.get('https://api.ipify.org', proxies='https://proxyname.com')
print(response.status_code)

httpx.__version__='0.25.0'
Traceback (most recent call last):
File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.1.3\plugins\python-ce\helpers\pydev\pydevd.py", line 1500, in _exec
pydev_imports.execfile(file, globals, locals) # execute the script
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.1.3\plugins\python-ce\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "C:\Users\anonymous\AppData\Roaming\JetBrains\PyCharmCE2023.2\scratches\scratch_2.py", line 4, in <module>
response = httpx.get('https://api.ipify.org', proxies='https://proxyname.com')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_api.py", line 189, in get
return request(
^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_api.py", line 100, in request
return client.request(
^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 814, in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 901, in send
response = self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 929, in _send_handling_auth
response = self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 966, in _send_handling_redirects
response = self._send_single_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 1002, in _send_single_request
response = transport.handle_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_transports\default.py", line 228, in handle_request
resp = self._pool.handle_request(req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpcore\_sync\connection_pool.py", line 262, in handle_request
raise exc
File "C:\Program Files\Python\Python312\Lib\site-packages\httpcore\_sync\connection_pool.py", line 245, in handle_request
response = connection.handle_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpcore\_sync\http_proxy.py", line 317, in handle_request
stream = stream.start_tls(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpcore\_backends\sync.py", line 149, in start_tls
raise RuntimeError(
RuntimeError: Attempted to add a TLS layer on top of the existing TLS stream, which is not supported by httpcore package

From my tests:

1. HTTPX allows an http:// (unencrypted) request to be proxied through a https:// (encrypted) proxy
2. HTTPX raises an httpcoreRuntimeError when an https:// (encrypted) request is proxied through an https:// (encrypted) proxy

requests supports use case 2, as per code in my initial post.

Can HTTPX/httpcore be fixed to achieve parity?

[karpetrosyan]

Thank you so much @mborsetti! This is a bug that will be fixed in the next release.
See: #840

[karpetrosyan]

Resolved in #840

[mborsetti]

@karpetrosyan Thanks for the quick action!

[lovelydinosaur]

Would you like to retry now?

[mborsetti]

Thanks for the fast turnaround, but ... hmmm?

Same code as above:

httpx.__version__='0.25.1'
Traceback (most recent call last):
File "C:\Users\anonymous\AppData\Roaming\JetBrains\PyCharmCE2023.2\scratches\scratch_2.py", line 7, in <module>
response = httpx.get('https://api.ipify.org', proxies='https://propxyname.com')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_api.py", line 189, in get
return request(
^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_api.py", line 100, in request
return client.request(
^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 814, in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 901, in send
response = self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 929, in _send_handling_auth
response = self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 966, in _send_handling_redirects
response = self._send_single_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 1002, in _send_single_request
response = transport.handle_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_transports\default.py", line 228, in handle_request
resp = self._pool.handle_request(req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpcore\_sync\connection_pool.py", line 262, in handle_request
raise exc
File "C:\Program Files\Python\Python312\Lib\site-packages\httpcore\_sync\connection_pool.py", line 245, in handle_request
response = connection.handle_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpcore\_sync\http_proxy.py", line 317, in handle_request
stream = stream.start_tls(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python\Python312\Lib\site-packages\httpcore\_backends\sync.py", line 149, in start_tls
raise RuntimeError(
RuntimeError: Attempted to add a TLS layer on top of the existing TLS stream, which is not supported by httpcore package

I reran the version using requests to ensure there wasn't anything else going on, and that one still works.

[mborsetti]

Update: I thought of checking the version of httpcore, and so much for pip install --upgrade httpx (or maybe just bad timing), it was still the old version.

I've manually updated httpcore, and it's all working fine now!

Thanks for the quick turnaround!!

httpx.__version__='0.25.1'
httpcore.__version__='1.0.1'
<my response here>