RuntimeError: dictionary changed size during iteration

[YLivay]

Hello.

I've been getting the following error randomly:

RuntimeError: dictionary changed size during iteration
 at .get_rule ( /app/.venv/lib/python3.12/site-packages/proto/marshal/marshal.py:172 )
 at .to_python ( /app/.venv/lib/python3.12/site-packages/proto/marshal/marshal.py:200 )
 at .__getattr__ ( /app/.venv/lib/python3.12/site-packages/proto/message.py:882 )
 at ._comparator ( /app/.venv/lib/python3.12/site-packages/google/cloud/firestore_v1/base_query.py:1143 )
 at .push ( /app/.venv/lib/python3.12/site-packages/google/cloud/firestore_v1/watch.py:565 )
 at ._on_snapshot_target_change_no_change ( /app/.venv/lib/python3.12/site-packages/google/cloud/firestore_v1/watch.py:387 )
 at .on_snapshot ( /app/.venv/lib/python3.12/site-packages/google/cloud/firestore_v1/watch.py:466 )
 at ._thread_main ( /app/.venv/lib/python3.12/site-packages/google/api_core/bidi.py:667 )

This happens when _instances is being mutated while being iterated on.

proto/marshal/marshal.py:BaseMarshal.get_rule:

class BaseMarshal:

 ....

 def get_rule(self, proto_type):
 # Rules are needed to convert values between proto-plus and pb.
 # Retrieve the rule for the specified proto type.
 # The NoopRule will be used when a rule is not found.
 rule = self._rules.get(proto_type, self._noop)

 # If we don't find a rule, also check under `_instances`
 # in case there is a rule in another package.
 # See https://github.com/googleapis/proto-plus-python/issues/349
 if rule == self._noop and hasattr(self, "_instances"):
 for _, instance in self._instances.items():
 rule = instance._rules.get(proto_type, self._noop)
 if rule != self._noop:
 break
 return rule

From a quick scan the most likely place _instances is being mutated is when registering a new Marshal instance.

proto/marshal/marshal.py:Marshal.__new__:

class Marshal(BaseMarshal):

 ....

 _instances = {}

 def __new__(cls, *, name: str):
 """Create a marshal instance.

 Args:
 name (str): The name of the marshal. Instantiating multiple
 marshals with the same ``name`` argument will provide the
 same marshal each time.
 """
 klass = cls._instances.get(name)
 if klass is None:
 klass = cls._instances[name] = super().__new__(cls)

 return klass

Environment details

• Programming language: Python
• OS: Linux
• Language runtime version: 3.12
• Package version: 1.26.0

Steps to reproduce

I can't say for sure how to reproduce this, but it seems to happen when a Marshal instance is registered while Marshal.get_rule is busy iterating over self._instances.items() to see which rule can handle a given proto_type.

I'm encountering this randomly when using Firestore (v2.20.1) on_snapshot function to watch for changes on a collection. This on_snapshot is invoked in a separate thread.

To fix

If you're fine with get_rule iterating over potentially slightly stale data, I suggest copying _instances aside in Marshal.new, add the new marshaller to the copy it and set it back to _instances.

If we have to have _instances perfectly in sync, just add a RW lock.

[parthea]

Transferring to google-cloud-python as the code for proto-plus-python will be migrated there in 1-3 weeks.

[daniel-sanche]

This issue is being bulk closed due to inactivity. Please re-open if you think this still requires attention, and we'll take another look. Thanks!

[jacobg]

@daniel-sanche We're seeing this sporadically in GAE near app warm up when fetching from datastore. Can we get this re-opened?

 File "/layers/google.python.pip/pip/lib/python3.12/site-packages/django/db/models/fields/related_descriptors.py", line 523, in __get__
 rel_obj = self.get_queryset(instance=instance).get(**filter_args)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/layers/google.python.pip/pip/lib/python3.12/site-packages/django/db/models/query.py", line 631, in get
 num = len(clone)
 ^^^^^^^^^^
 File "/layers/google.python.pip/pip/lib/python3.12/site-packages/django/db/models/query.py", line 368, in __len__
 self._fetch_all()
 ^^^^^^^^^^^^^^^^^
 File "/layers/google.python.pip/pip/lib/python3.12/site-packages/django/db/models/query.py", line 1954, in _fetch_all
 self._result_cache = list(self._iterable_class(self))
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/layers/google.python.pip/pip/lib/python3.12/site-packages/django/db/models/query.py", line 93, in __iter__
 results = compiler.execute_sql(

 File "/layers/google.python.pip/pip/lib/python3.12/site-packages/django/db/models/sql/compiler.py", line 1623, in execute_sql
 cursor.execute(sql, params)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/layers/google.python.pip/pip/lib/python3.12/site-packages/django/db/backends/utils.py", line 79, in execute
 return self._execute_with_wrappers(

 File "/layers/google.python.pip/pip/lib/python3.12/site-packages/django/db/backends/utils.py", line 92, in _execute_with_wrappers
 return executor(sql, params, many, context)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/layers/google.python.pip/pip/lib/python3.12/site-packages/django/db/backends/utils.py", line 105, in _execute
 return self.cursor.execute(sql, params)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/layers/google.python.pip/pip/lib/python3.12/site-packages/gcloudc/db/backends/common/base/connection.py", line 384, in execute
 self.rowcount = self.last_select_command.execute() or -1
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/layers/google.python.pip/pip/lib/python3.12/site-packages/gcloudc/db/backends/common/commands.py", line 209, in execute
 for entity in self.query.fetch(limit=limit, offset=offset):
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/layers/google.python.pip/pip/lib/python3.12/site-packages/gcloudc/db/backends/common/base/query.py", line 428, in _fetch
 for entity in qry.fetch(offset=offset, limit=limit):
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/layers/google.python.pip/pip/lib/python3.12/site-packages/gcloudc/db/backends/datastore/base.py", line 242, in fetch
 for result in self._query.fetch(offset=offset, limit=limit):
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/api_core/page_iterator.py", line 208, in _items_iter
 for page in self._page_iter(increment=False):
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/api_core/page_iterator.py", line 244, in _page_iter
 page = self._next_page()
 ^^^^^^^^^^^^^^^^^
 File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/datastore/query.py", line 850, in _next_page
 entity_pbs = self._process_query_results(response_pb)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/datastore/query.py", line 762, in _process_query_results
 self._skipped_results = response_pb.batch.skipped_results
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/layers/google.python.pip/pip/lib/python3.12/site-packages/proto/message.py", line 831, in __getattr__
 return marshal.to_python(pb_type, pb_value, absent=key not in self)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/layers/google.python.pip/pip/lib/python3.12/site-packages/proto/marshal/marshal.py", line 198, in to_python
 return self.get_rule(proto_type=proto_type).to_python(value, absent=absent)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/layers/google.python.pip/pip/lib/python3.12/site-packages/proto/marshal/marshal.py", line 170, in get_rule
 for _, instance in self._instances.items():
 ^^^^^^^^^^^^^^^^^^^^^^^

Exception Type: RuntimeError
Exception Value: dictionary changed size during iteration

[jacobg]

@parthea Please see me previous comment