Crash with google.protobuf.json_format.ParseError: Value '2024-09-23' has unexpected type <class 'datetime.date'>

[daaain]

Environment details

• OS type and version: python:3.11.9-bookworm base image
• Python version: 3.11.9
• pip version: pip 24.0 from /usr/local/lib/python3.11/site-packages/pip (python 3.11)
• google-cloud-logging version: 3.11.2

Steps to reproduce

Trying to send down a dict that has a DateTime property blows the library up with a google.protobuf.json_format.ParseError

I guess I could work around it manually and convert it to a string myself, but the bigger issue is that I wouldn't expect a logging library to crash my application.

Ideally the library should handle anything thrown at it with a best effort string conversion (even if it's just a useless class instance identity or anything), but at the very least have a way to log its own error rather than throwing an unhandled exception? Or are end users expected to wrap it in try / except and handle certain errors? Or is there a way to give the library a custom JSON parser?

Code example

from datetime import datetime
from google.cloud import logging

logging_client = logging.Client()
logging_client.setup_logging()

my_dict = {"my_date": datetime.now()}

logging_client.logger("default").log_struct({"message": "message", "my_dict": my_dict}, severity="INFO")

I thought maybe using the logging std lib might make a difference but that also crashes:

import logging
from datetime import datetime
from google.cloud import logging as gcloud_logging

logging_client = gcloud_logging.Client()
logging_client.setup_logging()

data_dict = {"my_date": datetime.now()}
logging.info("message field", extra={"json_fields": data_dict})

>>> Failed to submit 1 logs.
Traceback (most recent call last):
 File "/usr/local/lib/python3.11/site-packages/google/protobuf/json_format.py", line 600, in _ConvertFieldValuePair
 self.ConvertMessage(value, sub_message, '{0}.{1}'.format(path, name))
 File "/usr/local/lib/python3.11/site-packages/google/protobuf/json_format.py", line 485, in ConvertMessage
 methodcaller(_WKTJSONMETHODS[full_name][1], value, message, path)(self)
 File "/usr/local/lib/python3.11/site-packages/google/protobuf/json_format.py", line 704, in _ConvertStructMessage
 self._ConvertValueMessage(value[key], message.fields[key],
 File "/usr/local/lib/python3.11/site-packages/google/protobuf/json_format.py", line 682, in _ConvertValueMessage
 raise ParseError('Value {0} has unexpected type {1} at {2}'.format(
google.protobuf.json_format.ParseError: Value 2024-09-23 17:55:34.185216 has unexpected type <class 'datetime.datetime'> at LogEntry.jsonPayload.my_date

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File "/usr/local/lib/python3.11/site-packages/google/cloud/logging_v2/handlers/transports/background_thread.py", line 115, in _safely_commit_batch
 batch.commit()
 File "/usr/local/lib/python3.11/site-packages/google/cloud/logging_v2/logger.py", line 468, in commit
 client.logging_api.write_entries(
 File "/usr/local/lib/python3.11/site-packages/google/cloud/logging_v2/_gapic.py", line 154, in write_entries
 log_entry_pbs = [_log_entry_mapping_to_pb(entry) for entry in entries]
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/google/cloud/logging_v2/_gapic.py", line 154, in <listcomp>
 log_entry_pbs = [_log_entry_mapping_to_pb(entry) for entry in entries]
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/google/cloud/logging_v2/_gapic.py", line 557, in _log_entry_mapping_to_pb
 ParseDict(mapping, entry_pb)
 File "/usr/local/lib/python3.11/site-packages/google/protobuf/json_format.py", line 446, in ParseDict
 parser.ConvertMessage(js_dict, message, '')
 File "/usr/local/lib/python3.11/site-packages/google/protobuf/json_format.py", line 487, in ConvertMessage
 self._ConvertFieldValuePair(value, message, path)
 File "/usr/local/lib/python3.11/site-packages/google/protobuf/json_format.py", line 616, in _ConvertFieldValuePair
 raise ParseError(str(e)) from e
google.protobuf.json_format.ParseError: Value 2024-09-23 17:55:34.185216 has unexpected type <class 'datetime.datetime'> at LogEntry.jsonPayload.my_date

Stack trace

Traceback (most recent call last):
 File "/usr/local/lib/python3.11/site-packages/google/protobuf/json_format.py", line 600, in _ConvertFieldValuePair
 self.ConvertMessage(value, sub_message, '{0}.{1}'.format(path, name))
 File "/usr/local/lib/python3.11/site-packages/google/protobuf/json_format.py", line 485, in ConvertMessage
 methodcaller(_WKTJSONMETHODS[full_name][1], value, message, path)(self)
 File "/usr/local/lib/python3.11/site-packages/google/protobuf/json_format.py", line 704, in _ConvertStructMessage
 self._ConvertValueMessage(value[key], message.fields[key],
 File "/usr/local/lib/python3.11/site-packages/google/protobuf/json_format.py", line 670, in _ConvertValueMessage
 self._ConvertStructMessage(value, message.struct_value, path)
 File "/usr/local/lib/python3.11/site-packages/google/protobuf/json_format.py", line 704, in _ConvertStructMessage
 self._ConvertValueMessage(value[key], message.fields[key],
 File "/usr/local/lib/python3.11/site-packages/google/protobuf/json_format.py", line 682, in _ConvertValueMessage
 raise ParseError('Value {0} has unexpected type {1} at {2}'.format(
google.protobuf.json_format.ParseError: Value 2024-09-23 17:24:50.706366 has unexpected type <class 'datetime.datetime'> at LogEntry.jsonPayload.my_dict.my_date

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/usr/local/lib/python3.11/site-packages/google/cloud/logging_v2/logger.py", line 220, in log_struct
 self._do_log(client, StructEntry, info, **kw)
 File "/usr/local/lib/python3.11/site-packages/google/cloud/logging_v2/logger.py", line 169, in _do_log
 client.logging_api.write_entries(entries, partial_success=True)
 File "/usr/local/lib/python3.11/site-packages/google/cloud/logging_v2/_gapic.py", line 154, in write_entries
 log_entry_pbs = [_log_entry_mapping_to_pb(entry) for entry in entries]
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/google/cloud/logging_v2/_gapic.py", line 154, in <listcomp>
 log_entry_pbs = [_log_entry_mapping_to_pb(entry) for entry in entries]
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/google/cloud/logging_v2/_gapic.py", line 557, in _log_entry_mapping_to_pb
 ParseDict(mapping, entry_pb)
 File "/usr/local/lib/python3.11/site-packages/google/protobuf/json_format.py", line 446, in ParseDict
 parser.ConvertMessage(js_dict, message, '')
 File "/usr/local/lib/python3.11/site-packages/google/protobuf/json_format.py", line 487, in ConvertMessage
 self._ConvertFieldValuePair(value, message, path)
 File "/usr/local/lib/python3.11/site-packages/google/protobuf/json_format.py", line 616, in _ConvertFieldValuePair
 raise ParseError(str(e)) from e
google.protobuf.json_format.ParseError: Value 2024-09-23 17:24:50.706366 has unexpected type <class 'datetime.datetime'> at LogEntry.jsonPayload.my_dict.my_date```

[gkevinzheng]

@daaain:

We decided to, in the case of ParseErrors like this, rethrow a ValueError with that error message, so you will have to catch this on your own. This is because logging internally might cause a breaking change. We also made more explicit what the format of the dictionary that gets passed into log_struct should me in #958 as well.

This is because log_struct tries to convert the Python dictionary that gets passed in into a Protobuf Struct, which doesn't accept Python datetime objects. To work around this, you're probably going to have to use strftime on each datetime object.

I could see something about using custom JSON parsing similar to json.dumps.

[daaain]

Had a look at the PR, makes sense!

One thing I was wondering, would it be possible to add type hints so that MyPy or other static analysis libraries can catch if an unserialisable value is being passed?