Middleware API

[florimondmanca]

And here I am pitching the idea of a middleware API again… :-)

Definitely not for 1.0, and not pressing at all, but I wanted to give this a new shot to see what this looks like now that we're closer to 1.0 (compared to eg #345).

---

As I mentioned in https://github.com/encode/httpx/issues/984#issuecomment-633234552, and more recently in #1110 (comment), there is a variety of use cases that don't fall in the "make a custom transport" domain, the "subclass the client without using private API" domain. In general this is transport-agnostic feature that intercepts the request/response cycle, for example...

• Caching: look up the request key in a cache, and optionally return early with a response w/o hitting the transport.
• Throttling: defer the sending of the request until a throttle policy would be satisfied.
• HSTS (?): modify the request to use HTTPS if the requested website is on the HSTS Preload List.
• Retries: re-send a request until it succeeds, or fails too many times.

After playing with ideas, this PR is all we would need to support a middleware API that would support the above use cases and, I believe, many more.

Core ideas are…

• Requests are passed through a stack of middleware. The processing stack is:

# Process request...middleware_1
... middleware_n# The client's own hidden middleware...# Ultimately we dispatch to the transport.send_handling_redirects() # Process the response...
... middleware_nmiddleware_1

• The API is inspired by Starlette's BaseHTTPMiddleware: a base class with "dispatch hooks" that are given the Request and a call_next function for calling into the next middleware in the stack.

For example, a ThrottleMiddleware could be implemented as follows:

importtimefromtypingimportCallable, List, IteratorimportanyioimporthttpxclassBaseThrottleMiddleware:
def__init__(self, throttle: str) ->None:
self._history: List[float] = []
# Parse the thottle, which should be a string, like '100/minute'.count, _, duration=throttle.partition("/")
self._max_in_history=int(count)
self._cutoff= {"second": 1.0, "minute": 60.0, "hour": 3600.0}[duration]
def_iter_throttled(self) ->Iterator[None]:
now=time.time()
whilelen(self._history) >=self._max_in_history:
expiry=now-self._cutoff# Expire old entries in the history.self._history= [
timestampfortimestampinself._historyiftimestamp>expiry
]
# Sleep for a bit if we've exceeded the throttle rate.iflen(self._history) >=self._max_in_history:
yieldnow=time.time()
self._history.append(now)
classThrottleMiddleware(BaseThrottleMiddleware, httpx.Middleware):
""" An HTTPX middleware that adds some basic rate-throttling functionality. client = httpx.Client(middleware=[ThrottleMiddleware('100/minute')]) """defsend(self, request: httpx.Request, call_next: Callable) ->httpx.Response:
for_inself._iter_throttled():
time.sleep(0.1)
returncall_next(request)
classAsyncThrottleMiddleware(BaseThrottleMiddleware, httpx.AsyncMiddleware):
""" An HTTPX middleware that adds some basic rate-throttling functionality. client = httpx.AsyncClient(middleware=[AsyncThrottleMiddleware('100/minute')]) """asyncdefasend(
self, request: httpx.Request, call_next: Callable
) ->httpx.Response:
for_inself._iter_throttled():
awaitanyio.sleep(0.1)
returnawaitcall_next(request)
asyncdefmain() ->None:
fromstarlette.applicationsimportStarlettefromstarlette.responsesimportPlainTextResponsefromstarlette.routingimportRouteasyncdefhome(request): # type: ignorereturnPlainTextResponse("OK")
app=Starlette(routes=[Route("/", home)])
asyncwithhttpx.AsyncClient(
app=app, middleware=[AsyncThrottleMiddleware("1/second")]
) asclient:
awaitclient.get("http://testserver/") # Not throttledawaitclient.get("http://testserver/") # Throttled, waits for about 1s.if__name__=="__main__":
importasyncioasyncio.run(main())

Likewise, a caching middleware could look like this:

importtimefromtypingimportCallable, Dict, Tuple, OptionalimporthttpxclassCacheMiddleware(httpx.Middleware, httpx.AsyncMiddleware):
""" An HTTPX middleware that caches responses for a fixed amount of time. client = httpx.Client(middleware=[CacheMiddleware(ttl=3600)]) """def__init__(self, ttl: float) ->None:
self._cache=MemoryCache(ttl=ttl)
defsend(self, request: httpx.Request, call_next: Callable) ->httpx.Response:
response=self._cache.get(request)
ifresponseisnotNone:
returnresponseresponse=call_next(request)
self._cache.set(request, response)
returnresponseasyncdefasend(
self, request: httpx.Request, call_next: Callable
) ->httpx.Response:
response=self._cache.get(request)
ifresponseisnotNone:
returnresponseresponse=awaitcall_next(request)
self._cache.set(request, response)
returnresponseRequestKey=Tuple[str, httpx.URL, Tuple[Tuple[bytes, bytes], ...]]
classMemoryCache:
def__init__(self, ttl: float) ->None:
self._ttl=ttlself._data: Dict[RequestKey, Tuple[httpx.Response, float]] = {}
def_build_key(self, request: httpx.Request) ->RequestKey:
return (request.method, request.url, tuple(request.headers.raw))
defget(self, request: httpx.Request) ->Optional[httpx.Response]:
now=time.time()
key=self._build_key(request)
ifkeynotinself._data:
returnNoneresponse, expiry_date=self._data[key]
ifnow>expiry_date:
delself._data[key]
returnNonereturnresponsedefset(self, request: httpx.Request, response: httpx.Response) ->None:
now=time.time()
expiry_date=now+self._ttlkey=self._build_key(request)
self._data[key] = (response, expiry_date)
asyncdefmain() ->None:
fromstarlette.applicationsimportStarlettefromstarlette.responsesimportPlainTextResponsefromstarlette.routingimportRouteasyncdefhome(request): # type: ignorereturnPlainTextResponse("OK")
app=Starlette(routes=[Route("/", home)])
asyncwithhttpx.AsyncClient(
app=app, middleware=[CacheMiddleware(ttl=60)]
) asclient:
cached_response=awaitclient.get("http://testserver/")
response=awaitclient.get("http://testserver/")
assertresponseiscached_responseother_response=awaitclient.get("http://testserver/", params={"foo": "bar"})
assertother_responseisnotcached_responseif__name__=="__main__":
importasyncioasyncio.run(main())

Lastly, here's hstspreload back in the game:

fromtypingimportCallableimporthstspreloadimporthttpxclassHSTSMiddleware(httpx.Middleware, httpx.AsyncMiddleware):
""" An HTTPX middleware that enforces HTTPS on websites that are on the Chromium HSTS Preload list, mimicking the behavior of web browsers. client = httpx.Client(middleware=[HSTSMiddleware()]) """def_get_url(self, url: httpx.URL) ->httpx.URL:
if (
url.scheme=="http"andhstspreload.in_hsts_preload(url.host)
andlen(url.host.split(".")) >1
):
port=Noneifurl.port==80elseurl.porturl=url.copy_with(scheme="https", port=port)
returnurldefsend(self, request: httpx.Request, call_next: Callable) ->httpx.Response:
request.url=self._get_url(request.url)
returncall_next(request)
asyncdefasend(
self, request: httpx.Request, call_next: Callable
) ->httpx.Response:
request.url=self._get_url(request.url)
returnawaitcall_next(request)
asyncdefmain() ->None:
asyncwithhttpx.AsyncClient(middleware=[HSTSMiddleware()]) asclient:
response=awaitclient.get("http://paypal.org")
assertresponse.request.url.scheme=="https"if__name__=="__main__":
importasyncioasyncio.run(main())

---

Things I'm not sure about:

• Right now we pass middleware instances on client init: middleware=[SomeMiddleware(arg=1, ...)]. I think this is okay. (We don't need an equivalent of Starlette's Middleware wrapping helper, since HTTPX middleware aren't given any "parent app" on init.)
• Not super pleased with the naming of .send()/.asend().
  • One thing I'm pretty convinced of though is that we need the same property than for HTTPCore byte streams, i.e. "allow to implement both sync and async on the same class", so the method names must be different.
• Not sure if I need to do for m in middleware or for m in reverse(middleware).
• We're building the call_next function on each request, which could be a performance burden. OTOH Starlette is able to define .call_next() statically as a method of BaseHTTPMiddleware, and the middleware stack is built once and for all on app init. But HTTPX gets parameters that may differ on each request (auth, allow_redirects) so it seems hard to do differently. But not impossible I guess - needs some more thinking.

---

TODO:

• Validate this idea for a multi-request middleware, such as a RetryMiddleware.
• Needs tests.
• Needs docs.

[johnanthonyowens]

Another use case for middleware is tracing. We’re using Datadog APM tracing in our services and in order to get trace spans for all HTTP requests I’m currently monkeypatching send_single_request() to wrap a trace context manager around the request. This wouldn’t be possible with the proposal as I understand it because the middleware wouldn’t be able to observe redirects. One could argue that the right way to handle this is by wrapping httpcore in a custom transport that adds the tracing and then supplying that to my HTTPX clients - fair enough, I just haven’t tried that yet to see it works out in practice, and the monkeypatching approach is pretty trivial and wasn’t explicitly unsupported until the 0.14 release added the underscore. 😀

[lovelydinosaur]

@johnanthonyowens It might be worth opening a separate issue to discuss that in more detail. Things that'd be useful reference points here would be...

• What exactly where you tracking previously, and how?
• How does datadog handle this with requests or how would you be handling it if you were working against the requests API?
• Would request,redirect, and response event hooks be sufficient here for your use case?

[ionelmc]

Is there anything I can use right now to implement a response cache?

[florimondmanca]

@ionelmc Most likely a Client / AsyncClient subclass that overrides .send()?

[lovelydinosaur]

@ionelmc A sensible first thing to do with any question like that is to start with "how would I do this with requests" - have a look around in their ecosystem, and see if there's any implementations that do the same thing there, and then think about which part of the API it's plugged into.

The main override points for httpx for stuff like that are either:

• As @florimondmanca says, override .send() on the client instance, to wrap up some additional behaviour.
• Create a custom transport implementation that wraps up some additional behaviour at that layer, calling into the connection pool as needed.

[florimondmanca]

Just described a lighter form of this "middleware API" idea in the form of "interceptors", here… https://github.com/encode/httpx/issues/790#issuecomment-687823915

It's basically the same than the API proposed in this draft PR, except it's callable-based (sync functions for Client, async functions for AsyncClient). We get the drawback of having the sync/async schism exposed to developers and users (two kinds of everything), but it's also more lightweight in that there's no requirement to deal with classes.

[florimondmanca]

Okay, going to close this off again. For "middleware" that just wraps a request between the client and the final transport, it's already perfectly doable using the transport API, even though the instantiation pattern is a bit quirky for now (though not terrible).

importhstspreloadimporthttpcoreimporthttpxclassHSTSTransport(httpcore.SyncHTTPTransport, httpcore.AsyncHTTPTransport):
""" A transport wrapper that enforces HTTPS on websites that are on the Chromium HSTS Preload list, mimicking the behavior of web browsers. """def__init__(self, transport: Union[httpcore.SyncHTTPTransport, httpcore.AsyncHTTPTransport]) ->None:
self._transport=transportdef_maybe_https_url(self, url: tuple) ->tuple:
scheme, host, port, path=urlif (
scheme==b"http"andhstspreload.in_hsts_preload(host.decode())
andlen(host.decode().split(".")) >1
):
port=Noneifport==80elseportreturnb"https", host, port, pathreturnurldefrequest(self, method, url, headers, stream, ext):
url=self._maybe_https_url(url)
returnself._transport.request(method, url, headers, stream, ext)
asyncdefarequest(self, method, url, headers, stream, ext):
url=self._maybe_https_url(url)
returnawaitself._transport.arequest(method, url, headers, stream, ext)
transport=httpx.HTTPTransport() # Soontransport=HSTSTransport(transport)
withhttpx.Client(transport=transport):
...

[johtso]

@ionelmc also, regarding response caching.. this should be usable https://github.com/johtso/httpx-caching