Streaming multipart uploads (httpcore)

[florimondmanca]

Variant of #857 rebased against #804

[florimondmanca]

When dropping the Content-Length I was now getting an h11 error, but adding Transfer-Encoding: chunked resolved it.

Anyway, here's an update on server compatibility:

App	Compatible
HTTPBin	✅
Django microserver (on uvicorn WSGI)	❌
Starlette	✅

(I tried a full-blown Django project instead, but still no luck. I get a message like code 400, message Bad request syntax ('24'), and request.body is b"". What's happening?)

Script

>>>importhttpx>>>defupload(url): ... lorem=open("debug/lorem.txt") ... files= {"lorem": lorem} ... data= {"hello": "world"} ... returnhttpx.post(url, data=data, files=files) ...

HTTPBin

>>>url="https://httpbin.org/post">>>r=upload(url)
>>>r.json() # Contains correct "files" and "data"

Django microserver

# app.pyimportjsonfromdjango.confimportsettingsfromdjango.core.handlers.wsgiimportWSGIHandlerfromdjango.httpimportHttpResponsefromdjango.urlsimportpathsettings.configure(ROOT_URLCONF=__name__,)
defupload_file(request):
print(request.body) # XXX: this is b""... not sure why.content= {
"method": request.method,
"headers": dict(request.headers),
"data": dict(request.POST),
"files": dict(request.FILES),
}
returnHttpResponse(json.dumps(content), content_type="application/json")
urlpatterns= [path("", upload_file)]
app=WSGIHandler()

uvicorn app:app --interface wsgi

>>>url="http://localhost:8000">>>r=upload(url)
>>>r.json() # XXX: "data" and "files" and empty

Starlette

# app.pyfromstarlette.applicationsimportStarlettefromstarlette.routingimportRoutefromstarlette.responsesimportJSONResponsefromstarlette.requestsimportRequestfromstarlette.datastructuresimportUploadFileasyncdefupload(request: Request):
form=dict(awaitrequest.form())
hello: str=form["hello"]
lorem: UploadFile=form["lorem"]
content= {
"data": {"hello": hello},
"files": {"lorem": (awaitlorem.read()).decode()},
}
returnJSONResponse(content)
routes= [Route("/", upload, methods=["POST"])]
app=Starlette(routes=routes)

uvicorn app:app

>>>url="http://localhost:8000">>>r=upload(url)
>>>r.json() # Contains correct "files" and "data"

[florimondmanca]

OK so, a small improvement for Django is to use the ASGIHandler. Not sure why, but at least it allows request.body to contain the right data.

Still, request.FILES remains empty. Turns out it's because Django's multipart parser checks for Content-Type, defaults it to 0…

https://github.com/django/django/blob/40a64dd1e24d45f8e00a55b22a5174b8f1359b5c/django/http/multipartparser.py#L78-L83

… then stops as there's no body to read:

https://github.com/django/django/blob/40a64dd1e24d45f8e00a55b22a5174b8f1359b5c/django/http/multipartparser.py#L116-L119

Not sure if this is a bug in Django, but it's a hint that support for multipart uploads w/o Content-Length isn't universally supported out there.

---

So, cutting to the chase, I think we should:

• Include Content-Length when possible.
• When not possible, default to Transfer-Encoding: chunked, as some servers seem to support it (e.g. HTTPBin, Starlette do).

The cases when a file-like object doesn't have a measurable length are so small I think we're good enough with this approach for now. i.e. we almost always will include a Content-Length anyway.

[florimondmanca]

Closing as it's not strictly required for us to have the httpcore interface to move forward with streaming multipart, based on the conclusion above.