Documentation for the "transfer_manager" module

[tqa236]

Hello,

Thank you very much for the "transfer_manager" module that will probably greatly benefit us. However, I can't find the relevant documentation for this new module in the new documentation site. Do you know where it is?

Or if the documentation is not ready yet, is there an example of the usage of the new module somewhere?

Kind regards,
Anh

[andrewsg]

Hi @tqa236, thanks very much for your interest in our new feature! The documentation isn't yet published, but examples with comments are available: https://github.com/googleapis/python-storage/blob/9998a5e1c9e9e8920c4d40e13e39095585de657a/samples/snippets/storage_transfer_manager.py

As we get ready to publish the documentation, this file will be broken up into smaller files due to a one-sample-per-file rule, but they will remain in the same directory.

Hope this helps. Please feel free to send us any feedback, positive or negative, about the feature here in issues or comments.

[tqa236]

Thank you very much for your prompt reply @andrewsg. I have successfully used the new module thanks to the example. I noticed that, unlike the original PR, the second PR only supports concurrent files download, and not concurrent chunked downloads of a single file anymore. Can I know why it's left out and will it be added back in the future?

[andrewsg]

Unfortunately due to contention on the Python GIL, we found that there were no performance benefits for using the feature for chunks of a single file, versus a simple single-threaded download. In fact, the more threads, the slower the download.

Barring the use of gthead or some other monkey-patch solution that is appropriate for an end developer use case but not for a client library, it looks like we will need to implement a multiprocess solution to handle large files appropriately. I'll be working on prototypes for that approach this quarter. It will be a bit challenging to have it play nice with all of the things developers do with the library (recommended or not) so it may take some time.

I'm open to ideas from anyone reading this on approaches that might work, short of multiprocessing. In the meantime, the feature works well for large numbers of files just by using threads.

[tqa236]

Really appreciate the explanation @andrewsg

[andrewsg]

@tqa236 FYI we're re-adding chunked downloading, with multiprocessing, in #1002

[tqa236]

thanks for the head ups @andrewsg, it seems like a fairly big PR. Do you have a rough estimate of when it'll be available?

[andrewsg]

That depends on how the review goes. If there are no major issues, I'd be happy if it's out by the end of next week. I'm eager to get feedback from people using it!

[andrewsg]

@tqa236 This is being released now as 2.8.0. Thank you for your patience.