Using generated apitools client for storage uploads

[dhermes]

See #848

@craigcitro As we discussed this is the "testing sucks" "progress" I've made so far.

Positive outlook: The regression test passes

[craigcitro]

which tests were failing (other than the linter)?

[dhermes]

I don't think we should worry about which tests are failing. We just need to write new tests with your replay/record framework.

The issue being that we'll want to have the same types of things covered, e.g. multipart and resumable and media and similar.

[dhermes]

@craigcitro What say you about the replay/record testing features?

[craigcitro]

just imported them into apitools last night; was doing some related cleanup (eg that PR you just LGTMd), and i'm going to cut a release.

planning on sending you a few follow-up commits on top of these.

[dhermes]

That's awesome! Can't wait.

[dhermes]

@craigcitro Updates here?

[craigcitro]

apitools release is cut, but then this has been back-burnered. hoping to get to it today/tomorrow while some longer jobs are running.

[dhermes]

Cool. Someday maybe apitools could have some docs and then Craig can be free 👯

[craigcitro]

actually, it turned out i didn't really need any of the mocking code i added: the existing mocks already had sufficient coverage, i just needed to do a little cleanup.

i've pushed an extra commit to a branch in my fork; i'm happy to do whatever's most convenient for getting you unblocked.

two thoughts:

1. i wouldn't put much faith in any mocking-based tests for media transfers. our experience shows there are too many corners, and subtle changes on the server side, to really trust that mocks remain valid. in particular, the tests here seem to mostly test the positive cases ("does the upload/download work"), whereas most of the interesting bugs we've run into are around the failure modes (of which there are far too many).
2. i'd like to come back through and do a little cleanup around the use of apitools (i.e. i think some of the code can now be simplified), but i'm obviously not sitting on a lot of spare time right now. 😏 i'm still optimistic that i'll get to it, though.

[dhermes]

@craigcitro The values you swapped in aren't the values we expected based on the inputs. This is why I wanted to ditch our existing tests and use something that you have used to test apitools (i.e. your record/replay framework).

[craigcitro]

as mentioned on the commit comments, no one's threading the desired api endpoint into apitools, so there's no way it could know what address the tests were expecting. i guess i don't see the point of the current tests -- is there a use case where you'd point gcloud-python at something other than a google API?

you're welcome to rewrite the tests, but i don't have the time to do that, i'm afraid.

the apitools mocking framework doesn't support much for media testing, unfortunately. it's designed to allow record/replay of API calls, but only has some as-yet-unused media hooks.

[dhermes]

Can you point me towards the record/replay?

[craigcitro]

https://github.com/google/apitools/blob/master/apitools/base/py/testing/mock.py

[dhermes]

This is very much dead in light of #935