GAX iter.next() pattern seems to be breaking down

[dhermes]

For example, sometimes the iterable isn't an iterable

Traceback (most recent call last):
 File "/home/travis/build/GoogleCloudPlatform/gcloud-python/system_tests/logging_.py", line 215, in test_update_metric
 after_metrics, _ = Config.CLIENT.list_metrics()
 File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/logging/client.py", line 298, in list_metrics
 self.project, page_size, page_token)
 File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/logging/_gax.py", line 306, in list_metrics
 for log_metric_pb in page_iter.next()]
TypeError: 'LogMetric' object is not iterable

or

Traceback (most recent call last):
 File "/home/travis/build/GoogleCloudPlatform/gcloud-python/system_tests/pubsub.py", line 273, in test_fetch_delete_subscription_w_deleted_topic
 all_subs = retry_until_found_orphan(_fetch)()
 File "/home/travis/build/GoogleCloudPlatform/gcloud-python/system_tests/retry.py", line 126, in wrapped_function
 result = to_wrap(*args, **kwargs)
 File "/home/travis/build/GoogleCloudPlatform/gcloud-python/system_tests/pubsub.py", line 266, in _fetch
 return list(MethodIterator(Config.CLIENT.list_subscriptions))
 File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/gcloud/iterator.py", line 178, in __iter__
 page_token=self._token, page_size=self._page_size, **self._kw)
 File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/gcloud/pubsub/client.py", line 164, in list_subscriptions
 self.project, page_size, page_token)
 File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/gcloud/pubsub/_gax.py", line 242, in list_subscriptions
 for sub_pb in page_iter.next()]
TypeError: 'Subscription' object is not iterable

and sometimes it just gives a StopIteration when a new page is expected

Traceback (most recent call last):
 File "/home/travis/build/GoogleCloudPlatform/gcloud-python/system_tests/pubsub.py", line 130, in test_list_subscriptions
 empty, _ = topic.list_subscriptions()
 File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/gcloud/pubsub/topic.py", line 314, in list_subscriptions
 self.full_name, page_size, page_token)
 File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/gcloud/pubsub/_gax.py", line 198, in topic_list_subscriptions
 subs = page_iter.next()
 File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/google/gax/__init__.py", line 472, in next
 return self.__next__()
 File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/google/gax/__init__.py", line 478, in __next__
 self._current = next(self._page_iterator)
 File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/google/gax/__init__.py", line 435, in next
 return self.__next__()
 File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/google/gax/__init__.py", line 440, in __next__
 raise StopIteration
StopIteration

I withhold the flaky label because the same 10 test cases fail every time and in both Python 2 and 3:

1. test_fetch_delete_subscription_w_deleted_topic (pubsub.TestPubsub)
2. test_list_subscriptions (pubsub.TestPubsub)
3. test_list_topics (pubsub.TestPubsub)
4. test_list_metrics (logging_.TestLogging)
5. test_list_sinks (logging_.TestLogging)
6. test_log_struct (logging_.TestLogging)
7. test_log_struct_w_metadata (logging_.TestLogging)
8. test_log_text (logging_.TestLogging)
9. test_log_text_w_metadata (logging_.TestLogging)
10. test_update_metric (logging_.TestLogging)

[dhermes]

@tseaver It seems like #2124 is the culprit

[dhermes]

@tseaver ISTM that lines like

 token = page_iter.page_token or None
 return sinks, token

end up making page_token=None override the default value of page_token=INITIAL_PAGE in lots of these methods.

[dhermes]

I'm gonna send a PR

[dhermes]

@tseaver Update on this. I modified this to raise during the system tests for explicit None and actually update the GAX dependencies in my local install.

Running them

$ nosetests system_tests/logging_.py
$ nosetests system_tests/pubsub.py

we find 13 test cases that end up with an explicit None.

---

FWIW I ran with dependencies:

$ pip freeze | grep '\(protobuf\|gax\|grpc\|httplib2\|googleapis\|oauth2\)'
gax-google-logging-v2==0.8.1
gax-google-pubsub-v1==0.8.1
google-gax==0.12.5
googleapis-common-protos==1.1.0
grpc-google-logging-v2==0.8.1
grpc-google-pubsub-v1==0.8.1
grpcio==1.0.0
httplib2==0.9.2
oauth2client==3.0.0
protobuf==3.0.0

[dhermes]

@tseaver OK I went through and verified the code paths from the stack traces and annotated with some diffs. The following methods do pass an explicit None (they don't have a choice really)

• pubsub.Client.list_topics()
• pubsub.Client.list_subscriptions()
• pubsub.Topic.list_subscriptions()
• logging.Client.list_entries()
• logging.Client.list_sinks()
• logging.Client.list_metrics()
• logging.Logger.list_entries()