Implement streaming multipart uploads

[florimondmanca]

Fixes #114. ✨

Ensures we don't ever load any files or buffers in full into memory when uploading form data.

I've also refined tests to make sure we have full test coverage on the multipart implementation.

[florimondmanca]

Most interestingly, this shields us from issues people have been having in Requests/urllib3 due to urllib3 not supporting streaming in this case, eg:

• Response.content is wasteful in time and memory for large inputs psf/requests#4687: High resource usage of multipart in Requests
• The reason that uploading file larger than 2GB will be truncated. psf/requests#5024: Python bug when calling .sendall() with 2GB+ of content

Would be interesting to validate against a local but realistic setup though, eg streaming a 500MB+ file over to a local server, and see how memory usage behaves.

[florimondmanca]

I ran a bunch of experiments against HTTPBin and a Django microserver to see how they'd behave to the presence/absence of Content-Length for multipart POST requests.

They basically treat it as Content-Length: 0 and don't even try to read the body. As a result, from their POV the client hasn't sent any files/data.

So it looks like in practice Content-Lengthis required for multipart form data anyway…

Details below:

Click to expand

First, issuing requests against https://httpbin.org/post…

Including the Content-Length results in correct behavior:

>>>importhttpx>>>url="https://httpbin.org/post">>>lorem=open("debug/lorem.txt")
>>>files= {"lorem": lorem}
>>>data= {"hello": "world"}
>>>r=httpx.post(url, data=data, files=files)
>>>r.status_code200>>>r.json()
{'args': {},
'data': '',
'files': {'lorem': '\n\nLorem ipsum <...Redacted for brievety...>'},
'form': {'hello': 'world'},
'headers': {'Accept': '*/*',
'Accept-Encoding': 'gzip, deflate, br',
'Content-Length': '3261',
'Content-Type': 'multipart/form-data; boundary=e7fa8325d92b926b35d877268b7b8435',
'Host': 'httpbin.org',
'User-Agent': 'python-httpx/0.12.0',
'X-Amzn-Trace-Id': 'Root=1-5e69ebc0-2cde1d1a6d866d38fbb79c57'},
'json': None,
'origin': '92.169.139.224',
'url': 'https://httpbin.org/post'}

If Content-Length isn't included, then HTTPBin infers it as 0 (although the request content wasn't empty: it contained data and files) and doesn't even read the request:

>>>importhttpx>>>url="https://httpbin.org/post">>>lorem=open("debug/lorem.txt")
>>>files= {"lorem": lorem}
>>>data= {"hello": "world"}
>>>client=httpx.Client()
>>>request=client.build_request("POST", url, data=data, files=files)
>>>request.headers.pop("content-length")
'3261'>>>r=client.send(request)
>>>r.status_code200>>>r.request.headers# NOTE: no 'content-length' here.Headers({'host': 'httpbin.org', 'user-agent': 'python-httpx/0.12.0', 'accept': '*/*', 'accept-encoding': 'gzip, deflate, br', 'connection': 'keep-alive', 'content-type': 'multipart/form-data; boundary=405abb1f7a6217c5c1bfbfbf8b96010e'})
>>>r.json() # See how this is mostly empty?
{'args': {},
'data': '',
'files': {},
'form': {},
'headers': {'Accept': '*/*',
'Accept-Encoding': 'gzip, deflate, br',
'Content-Length': '0',
'Content-Type': 'multipart/form-data; boundary=405abb1f7a6217c5c1bfbfbf8b96010e',
'Host': 'httpbin.org',
'User-Agent': 'python-httpx/0.12.0',
'X-Amzn-Trace-Id': 'Root=1-5e69eb31-ef22d77230959bd05da2aac4'},
'json': None,
'origin': '92.169.139.224',
'url': 'https://httpbin.org/post'}

Now with requests-toolbelt…

First with Content-Length — works fine:

>>>importrequests>>>fromrequests_toolbelt.multipart.encoderimportMultipartEncoder>>>lorem=open("debug/lorem.txt", "rb")
>>>fields= {"lorem": ("lorem.txt", lorem, "text/plain"), "hello": "world"}
>>>m=MultipartEncoder(fields=fields)
>>>url="https://httpbin.org/post">>>r=requests.post(url, data=m, headers={"Content-Type": m.content_type})
>>>r.status_code200>>>r.request.headers
{'User-Agent': 'python-requests/2.23.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'multipart/form-data; boundary=f34a54a283644e4da0f6cc94c9f11548', 'Content-Length': '3261'}

Now, dropping Content-Length, I actually had to go through hoops to even be able to send the request due to weird edge cases in requests-toolbelt and Requests, but eventually we get the same behavior as in HTTPX: HTTPBin will interpret as 'Content-Length: 0' and won't read the body.

>>>importrequests>>>fromrequests_toolbelt.multipart.encoderimportMultipartEncoder>>>lorem=open("debug/lorem.txt", "rb")
>>>fields= {"lorem": ("lorem.txt", lorem, "text/plain"), "hello": "world"}
>>>m=MultipartEncoder(fields=fields)
>>>url="https://httpbin.org/post">>>s=requests.Session()
>>># BUG: 'MultipartEncoder' is not iterable>>>data=m.to_string()
>>># NOTE: request won't be streamed now, but not the point here.>>>req=requests.Request("POST", url, data=data, headers={"Content-Type": m.content_type})
>>>prepped=req.prepare()
>>># BUG: 'int has no len()' (because iterating over bytes yields ints)>>>prepped.body=prepped.body.splitlines()
>>>prepped.headers.pop("Content-Length")
'3261'>>>r=s.send(prepped)
>>>r.status_code200>>>r.json()
{'args': {},
'data': '',
'files': {},
'form': {},
'headers': {'Content-Type': 'multipart/form-data; boundary=9515a8b9bea94739968c0976c03d8d63',
'Host': 'httpbin.org',
'X-Amzn-Trace-Id': 'Root=1-5e69eeeb-e2950740e34e2f3045592140'},
'json': None,
'origin': '92.169.139.224',
'url': 'https://httpbin.org/post'}

Now, is this specific to HTTPBin? Let's try with a Django microserver.

# app.pyimportjsonfromdjango.confimportsettingsfromdjango.core.handlers.wsgiimportWSGIHandlerfromdjango.httpimportHttpResponsefromdjango.urlsimportpathsettings.configure(ROOT_URLCONF=__name__,)
defupload_file(request):
content= {
"method": request.method,
"headers": dict(request.headers),
"data": dict(request.POST),
"files": dict(request.FILES),
}
returnHttpResponse(json.dumps(content), content_type="application/json")
urlpatterns= [path("", upload_file)]
app=WSGIHandler()

uvicorn app:app --interface wsgi

>>>importhttpx>>>url="http://localhost:8000/">>>lorem=open("debug/lorem.txt")
>>>files= {"lorem": lorem}
>>>data= {"hello": "world"}
>>>client=httpx.Client()
>>>request=client.build_request("POST", url, data=data, files=files)
>>>request.headers.pop("content-length")
'3261'>>>r=client.send(request)
>>>r.status_code200>>>r.json()
{'method': 'POST',
'headers': {'Content-Length': '0',
'Host': 'localhost:8000',
'User-Agent': 'python-httpx/0.12.0',
'Accept': '*/*',
'Accept-Encoding': 'gzip, deflate, br',
'Connection': 'keep-alive',
'Content-Type': 'multipart/form-data; boundary=65a4b7387ae71a5324d29bbaaa9375bd'},
'data': {},
'files': {}}

So, here again, Django interpreted the absence of Content-Length header as "there's no body, so I'm not reading it".

[florimondmanca]

Ah so, actually, it's us that default to setting Content-Length to 0 in the sync case currently…

httpx/httpx/_dispatch/urllib3.py

Line 94 in a82adcc

content_length=int(request.headers.get("Content-Length", "0"))

I'll try this against #804 and see if we get the same results.

[florimondmanca]

After experimenting some more with #863, the httpcore interface does solve some problems (it doesn't necessarily force Content-Length: 0).

But the conclusion I reached is:

• We should include Content-Length if possible, as some servers will require it.
• If not possible (i.e. filelike is not seekable(), which is an extra edgy case anyway), include Transfer-Encoding: chunked, as some servers may support it.
• If not, well, let's have users wrap their weird files around BytesIO. :-)

[florimondmanca]

@tomchristie I think this is ready for a re-review.

I've revised a bit my assumptions: it's probably okay to just load the entire contents into memory if we end up dealing with weird file-like objects that are readable but non-seekable. This means we can cover all sensible situations consistently (i.e. include Content-Length at all times), and have streaming effective in 99.99% of cases.

[yeraydiazdiaz]

This looks great, just a question for my own curiosity. 👏 👏 👏

[florimondmanca]

Hey @tomchristie, how's this looking? :-) Are we ready to merge yet?

[florimondmanca]

Alright, updated these on top of the HTTPCore usage that's now in master, should be ready for a new quick review before merging. 👍

[yeraydiazdiaz]

Still LGTM 👍

[florimondmanca]

Okay, thanks, merging now. :-)