[Logging] `page_size` appears to not work; no iterator(s)

[DazWilkin]

See #136

Code

requirements.txt:

google-auth==1.27.0
google-cloud-logging==2.2.0

And main.py (derived from the Google published sample)

Using client.list_entries:

import os

from google.cloud import logging
from google.cloud.logging import DESCENDING

client = logging.Client()

PROJECT = os.getenv("PROJECT")
resource = "projects/{project}".format(project=PROJECT)
log_name = "{resource}/logs/cloudaudit.googleapis.com%2Factivity".format(
 resource=resource)
filter = "logName=\"{log_name}\"".format(log_name=log_name)

entries = client.list_entries(filter_=filter,
 order_by=DESCENDING,
 page_size=3,
 resource_names=[resource])

print("Listing entries for logger {}:".format(log_name))
for entry in entries:
 timestamp = entry.timestamp.isoformat()
 print("{}".format(timestamp))

Using logger.list_entries:

import os

from google.cloud import logging
from google.cloud.logging import DESCENDING

client = logging.Client()

log_name = "cloudaudit.googleapis.com%2Factivity"
logger = client.logger(log_name)

entries = logger.list_entries(
 order_by=DESCENDING,
 page_size=3,
)

print("Listing entries for logger {}:".format(log_name))
for entry in entries:
 timestamp = entry.timestamp.isoformat()
 print("{}".format(timestamp))

Steps to reproduce

Running the Python:

python3 cloud/main.py 
Listing entries for logger projects/dazwilkin-210302-66371557/logs/cloudaudit.googleapis.com%2Factivity:
2021-03-02T17:39:09.421835+00:00
2021-03-02T17:39:12.121825+00:00
2021-03-02T17:39:12.933201+00:00
2021-03-02T17:39:16.011850+00:00
2021-03-02T17:39:17.114823+00:00
2021-03-02T17:39:18.699000+00:00
2021-03-02T17:41:29.759709+00:00
2021-03-02T17:41:30.937346+00:00
2021-03-02T17:41:32.653000+00:00

NOTE Neither DESCENDING nor limited

Running the equivalent (!) gcloud yields the expected result:

gcloud logging read "logName=\"projects/${PROJECT}/logs/cloudaudit.googleapis.com%2Factivity\"" \
--project=${PROJECT} \
--order=desc \
--limit=3 \
--format="value(timestamp)" \
--log-http

...
== body start ==
{
 "filter": "timestamp>=\"2021-03-01T19:14:21.929877Z\" AND logName=\"projects/${PROJECT{/logs/cloudaudit.googleapis.com%2Factivity\"",
 "orderBy": "timestamp desc",
 "pageSize": 3,
 "resourceNames": ["projects/${PROJECT}"]
}
== body end ==
...
2021-03-02T17:41:32.653Z
2021-03-02T17:41:30.937346481Z
2021-03-02T17:41:29.759709770Z

Additionally:

• The list_entries result should be an interator that can be paged; it is not
• The library 'encourages' repetition of resource|project (logging_client.logger, filter and resource_names) and it's unclear how I can only specify this once

[daniel-sanche]

I believe this issue is related to #154: We currently have two different implementations (grpc and http), which are implemented in inconsistent ways. This results in the gRPC layer ignoring some of the arguments. Unfortuantely a fix will likely have to wait until our next major relase, because it'll have to be a breaking change.

In the mean time, you can try to use HTTP mode instead of gRPC mode, which seems to work as you expect. You can turn it on with client = logging.Client(_use_grpc=0) or setting the DISABLE_GRPC environment variable to any value.

Note though that if your code iterates through all pages. If you just want a single page, you have to iterate differently:

for entry in next(entries.pages):
 timestamp = entry.timestamp.isoformat()
 print("{}".format(timestamp))

https://googleapis.dev/python/google-api-core/latest/page_iterator.html

[DazWilkin]

It does not work (nor as documented by #154).

env | grep GRPC
DISABLE_GRPC=true
python3 cloud/main.py
Listing entries for logger cloudaudit.googleapis.com%2Factivity:
2021-03-03T17:27:32.162000+00:00
2021-03-03T17:27:30.517190+00:00
2021-03-03T17:27:29.339759+00:00
2021-03-03T17:25:47.088000+00:00
2021-03-03T17:25:42.505007+00:00
2021-03-03T17:25:42.115940+00:00
2021-03-03T17:25:39.407426+00:00
2021-03-03T17:25:38.717723+00:00
2021-03-03T17:25:36.399174+00:00

NOTE I'm using a new project today so the entries are different but the issue remains (all results instead of 3)

Yields:

• All results (expect 3)
• method returns generator (suggesting DISABLE_GRPC is ineffective!?)

And (unset DISABLE_GRPC):

NOTE Elsewhere this variable is referenced as GOOGLE_CLOUD_DISBABLE_GRPC: True. I tried that and this then returns an google.api_core.page_iterator.HTTPIterator but all results

client = logging.Client(_use_grpc=0)

Yields:

• All results (expect 3)
• method returns an google.api_core.page_iterator.HTTPIterator

I used a slightly different mechanism on the iterator:

iterator = logger.list_entries(
 order_by=DESCENDING,
 page_size=3,
)
print(type(iterator))

print("Listing entries for logger {}:".format(log_name))
for page in iterator.pages:
 for entry in page:
 timestamp = entry.timestamp.isoformat()
 print("{}".format(timestamp))

[daniel-sanche]

NOTE Elsewhere this variable is referenced as GOOGLE_CLOUD_DISBABLE_GRPC: True. I tried that and this then returns an google.api_core.page_iterator.HTTPIterator but all results

Sorry, you're right. The variable is called DISABLE_GRPC in the code, but it looks like the real name to use is GOOGLE_CLOUD_DISBABLE_GRPC

I used a slightly different mechanism on the iterator:

The problem with your code is the double loop. You are through iterating all pages, along with all entries in each page. This will give you all entries, just loaded in batches of 3. If you just want all 3 entries from a single page, you'll have to do something like the code I posted above:

for entry in next(entries.pages):
 timestamp = entry.timestamp.isoformat()
 print("{}".format(timestamp))

This should give you all 3 results on the first page of results, which seems to be what you're looking for. But let me know if you're still having issues.

[DazWilkin]

Doh! Thanks. Yes, of course.

I was thinking I would get one page of three results but I'm actually getting n pages of 3 results.

Thank you!

[keenan-devrel]

This should be addressed in the upcoming 3.0 release planned for Q4 2021.