Implement SOCKS v4, v5 Proxy

[Kludex]

Originally opened by @sethmlarson on 2019-08-13 17:45:48 in encode/httpx

Related: #36

[Kludex]

Originally posted by @yeraydiazdiaz on 2019-09-02T15:05:29Z

I'm happy to lend a hand on this @sethmlarson. Does it depend on #259?

[Kludex]

Originally posted by @sethmlarson on 2019-09-02T15:14:42Z

It'll definitely intersect on the configuration stage on the client but the dispatcher implementation is separate, let's start by getting a sans-I/O implementation of SOCKSv4 and v5 w/o dependencies and go from there.

I actually think that the sans-I/O implementation should be it's own library, maybe on the python-http org, but it can start on one of our personal accounts. Would you like to be the originator or should I create a repo and add you and you can take it from there?

[Kludex]

Originally posted by @yeraydiazdiaz on 2019-09-02T15:40:14Z

I'll definitely need some help so it might be easier if it's all setup in non-personal repo from the start 🙂

[Kludex]

Originally posted by @sethmlarson on 2019-09-05T18:22:31Z

I pushed the initial commit: https://github.com/sethmlarson/socks
Feel free to make massive changes as nothing currently works E2E, it's just a result of me programming a few hours.
I've sent collaborator requests to everyone interested. :)

The repo will live under python-http once we release for the first time!

[Kludex]

Originally posted by @mIcHyAmRaNe on 2020-04-10T07:24:45Z

while waiting for the implementation, this is a temporary alternative for people who want to use socks4/5 with httpx by using pysocks :

# pip install PySocks

import httpx
import trio
import socks
import socket

socks.set_default_proxy(socks.SOCKS5, "127.0.0.1", 9050)
socket.socket = socks.socksocket

URL = 'http://ifconfig.me/ip'

with httpx.Client() as client:
 resp = client.get(URL)
 print(resp.text)

[Kludex]

Originally posted by @romis2012 on 2020-07-10T05:38:05Z

If it's relevant, there is 3rd party SOCKS implementation: httpx-socks

[Kludex]

Originally posted by @bbkane on 2020-07-10T16:44:44Z

It looks like that one doesnt support trio (my favorite async backend).

[Kludex]

Originally posted by @romis2012 on 2020-07-11T08:05:30Z

It looks like that one doesnt support trio (my favorite async backend).

Trio support added in version 0.2.0

[Kludex]

Originally posted by @cdeler on 2020-09-16T14:05:32Z

@tomchristie

I wrote a small PoC with httpcore + PySocks with requesting example.com through the local socks5 proxy.

Working with the PoC, I added new method to AsyncioBackend named open_socks_stream (I decided that socks4/5 is a transport for us like tcp, ssl or uds).

If this idea (PySocks and a new method in the backends) works for you, I can start adding socks proxies support to httpcore

[Kludex]

Originally posted by @florimondmanca on 2020-09-16T19:09:17Z

@cdeler From what I understand, SOCKS is an application-level protocol that sits on top of TCP (well, there's UDP in SOCKS5, but that's not something we should be thinking about for now), so in theory we shouldn't really need a new type of open_* method on concurrency backends.

Also, since I don't think it's been linked to here yet — @yeraydiazdiaz had started a lovely piece of work on HTTPCore already a few months back, based on the socksio library: encode/httpcore#51. Benefits of socksio is that it's sans-I/O, meaning that we can use it either with sync or async, just like h11 and h2 for HTTP/1.1 and HTTP/2.

So perhaps, if anyone's interested, there'd be room for getting that work up to date. I personally think socksio and the sans-I/O approach is our safest bet there if we want to have an as-simple-and-straightforward-as-possible implementation. :-)

[Kludex]

Originally posted by @cdeler on 2020-09-16T19:36:04Z

@florimondmanca
Whoops, I lost that there is a PR in progress...

On one hand you are right, proxy works on L7, but it is a transport for us...

[Kludex]

Originally posted by @cdeler on 2020-09-17T14:53:45Z

@florimondmanca you are right, socksio is really better, since it allows us implement the code on connection-level. I closed encode/httpcore#186 (with PySock) and opened encode/httpcore#187 (socksio) draft

[Kludex]

Originally posted by @cdeler on 2020-09-21T09:25:37Z

I wonder what we should do with socks4. It enforces us to process nslookup on our side (as socks4 connect do not accept domain names).

Lets imagine that someone wants to access "google.com" through socks4 proxy. What should we do there? To raise a ProxyError with "SOCKS4 protocol do not support domain names as a host address" or make nslookup on our side (does anyone know good nslookup libraries for async?) ?

[Kludex]

Originally posted by @romis2012 on 2020-09-21T09:49:54Z

SOCKS4 protocol do not support domain names as a host address

Some socks5 servers also don't support DNS resolving

does anyone know good nslookup libraries for async

See how it is implemented in python-socks which httpx-socks is based on. For asyncio backend you can also use aiodns.

[Kludex]

Originally posted by @KOLANICH on 2021-01-18T16:22:38Z

python-hyper/hyper#441 may be related (though it is for hyper)