Add progress to streaming download

[cdeler]

Having found #1208, I tried to track last downloaded raw chunk of the response.

The question is:

1. should we also track decoded response chunks? I don't see any reason since decoded stream length might be vary from the raw stream length, and it has no bearing with downloading process (but may be I missed something)

[florimondmanca]

Still need to ponder on this a bit, but very neat implementation (code, docs and tests)!

[florimondmanca]

should we also track decoded response chunks?

Don't think so, since that's directly available as len(chunk) already.

[lovelydinosaur]

Great, thanks!

I think this has helped me. understand why I'm a bit wary of response.last_raw_chunk_size as the interface here, rather than response.num_bytes_downloaded (Or something similarly named.)

It seems to me that it's more likely to lead to brittle code. Also it's not super clear what the behaviour is expected to be in different cases. Eg...

• I don't think it'd work as expected if a user happened to be using iter_lines().
• It's not clear how it'd be expected to interact with Added chunk_size to Response.iter_bytes() and Response.aiter_bytes() (#394) #1271

In contrast if we used response.num_bytes_downloaded then there's just less to get wrong.
Also examples like this work, where .last_raw_chunk_size wouldn't...

response=httpx.get(...)
response.num_bytes_downloaded# Determine the final download size.

It is true that num_bytes_downloaded happens to be marginally more work in the tqdm example, but it's not much more to deal with.

num_bytes_downloaded=0withhttpx.stream("...") asresponse:
total= ...
withtqdm(total=...) asprogress:
forchunkinresponse.iter_bytes():
download_file.write(chunk)
progress.update(response.num_bytes_downloaded-num_bytes_downloaded)
num_bytes_downloaded=response.num_bytes_downloaded

What do we think?

[lovelydinosaur]

Lovely bit of functionality to work towards yup!
Really liking the docs example here.
Possible that we ought to be going for a total downloaded bytes API rather than a chunk size API?

[cdeler]

@tomchristie

I changed last_raw_chunk_size to num_bytes_downloaded.

For now we can state that there is a correlation between the amount of data (lets say N) drained from iter_raw(chunk_size=123) and num_bytes_downloaded. I.e. until the end of the stream we can assert that N <= response.num_bytes_downloaded. And well, we cannot say the same about iter_bytes(chunk_size=123) due to compression.

[florimondmanca]

Looking good with num_bytes_downloaded too, yup 👍

[lovelydinosaur]

Have made a couple of minor docs tweaks.
Looking great, thank you!