`collections.deque` is not atomic during copy

[yilei]

[Edit: This bug was originally reported as "not thread safe", and updated to "not atomic"]

Bug report

Bug description:

In free-threaded mode, calling copy on collections.deque is not thread safe atomic and can raise RuntimeError: deque mutated during iteration errors.

Example code:

import sys
import threading
from collections import deque
from copy import copy

print(f"Python {sys.version}")
print(f"GIL enabled: {sys._is_gil_enabled()}")

d = deque(range(100))
stop = threading.Event()


def mutate():
 i = 0
 while not stop.is_set():
 d.append(i)
 if len(d) > 200:
 d.popleft()
 i += 1


def copy_loop():
 while not stop.is_set():
 copy(d)


barrier = threading.Barrier(5)


def run(fn):
 barrier.wait()
 fn()


threads = [
 threading.Thread(target=run, args=(f,), daemon=True)
 for f in [mutate, mutate, copy_loop, copy_loop]
]
for t in threads:
 t.start()

barrier.wait()

stop.wait(timeout=5)
stop.set()
for t in threads:
 t.join(timeout=5)

print("Done")

Running under free-threaded build:

❯ ./python /tmp/deque.py
Python 3.15.0a6+ free-threading build (heads/main:945bf8ce1bf, Feb 13 2026, 22:04:08) [GCC 11.4.0]
GIL enabled: False
Exception in thread Thread-4 (run):
Exception in thread Thread-3 (run):
Traceback (most recent call last):
Traceback (most recent call last):
 File "cpython/Lib/threading.py", line 1075, in _bootstrap_inner
 self._context.run(self.run)
 ~~~~~~~~~~~~~~~~~^^^^^^^^^^
 File "cpython/Lib/threading.py", line 1017, in run
 self._target(*self._args, **self._kwargs)
 ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/tmp/deque.py", line 32, in run
 fn()
 ~~^^
 File "/tmp/deque.py", line 24, in copy_loop
 copy(d)
 ~~~~^^^
 File "cpython/Lib/copy.py", line 82, in copy
 return copier(x)
RuntimeError: deque mutated during iteration
 File "cpython/Lib/threading.py", line 1075, in _bootstrap_inner
 self._context.run(self.run)
 ~~~~~~~~~~~~~~~~~^^^^^^^^^^
 File "cpython/Lib/threading.py", line 1017, in run
 self._target(*self._args, **self._kwargs)
 ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/tmp/deque.py", line 32, in run
 fn()
 ~~^^
 File "/tmp/deque.py", line 24, in copy_loop
 copy(d)
 ~~~~^^^
 File "cpython/Lib/copy.py", line 82, in copy
 return copier(x)
RuntimeError: deque mutated during iteration
Done

CPython versions tested on:

CPython main branch, 3.14

Operating systems tested on:

Linux

Linked PRs

• gh-144809: Make deque copy atomic in free-threaded build #144966
• [3.14] gh-144809: Make deque copy atomic in free-threaded build (gh-144966) #145053

[serhiy-storchaka]

I do not think it is not thread safe. Copying does not crash Python, does not corrupt memory, and does not left a deque in an inconsistent state. It is not atomic -- but you cannot in general expect that an operation with non-constant complexity on collection is atomic.

cc @rhettinger

[yilei]

Thanks for taking a look at this!

Hmm, my mental model of thread safety is that I'd expect copy(d) would reasonably give me a copy when there are other concurrent mutations in parallel, without using e.g. a lock to guard the copy and mutations. And perhaps that's a wrong interpretation? And non-atomic is the better term. My assumption was largely based on:

1. Seeing earlier free-threading work from gh-112050: Make collections.deque thread-safe in free-threaded builds #113830
2. Similar code for dict and list succeeds without errors

Btw, additional context for this bug report is getsentry/sentry-python#5465 where we encountered this issue.

I'm fine if we couldn't make copy(d) work here without users adding guards around calls to deque methods (though IMO it hurts usability), it would be great if we could document this expectation somewhere.

[colesbury]

I think we should make this behave atomically because:

1. it's written so that it behaves atomically in the GIL-enabled build.
2. it should be relatively easy (... famous last words)
3. the motivation is a real bug report in user code

The problem is a combination of:

1. We lock the to-be-copied deque when __copy__ is called (good!)
2. We iterate over the to-be-copied deque, which tries to lock both the iterator and the deque

That triggers the deadlock avoidance routine, which unlocks the deque before re-locking both the iterator and the deque. The time in between the unlock and re-locking allows another thread to modify the deque (which would not be allowed in the GIL-enabled build).

[Yhg1s]

Given that we already have an optimization where we skip locking (directly) nested critical sections of the same object, and we order CRITICAL_SECTION2 locks by mutex address to avoid deadlocks (so the order of it/deque locking is already opaque to the caller) -- couldn't we just skip locking whichever lock in a CRITICAL_SECTION2 is currently locked in the (previous) top-most critical section?

[yilei]

couldn't we just skip locking whichever lock in a CRITICAL_SECTION2 is currently locked in the (previous) top-most critical section?

I think this works, and it's a good optimization to have anyway. I'll clean up my draft code, add some tests, and send a PR.

[colesbury]

couldn't we just skip locking whichever lock in a CRITICAL_SECTION2 is currently locked in the (previous) top-most critical section?

That would work for the specific code in deque, but it's not safe in general because if the thread state is detached and then attached, only the (new) top-most critical section is locked. For example:

Py_BEGIN_CRITICAL_SECTION(a);
Py_BEGIN_CRITICAL_SECTION2(a, b); // only locks b?

_PyThreadState_Detach(tstate);
_PyThreadState_Attach(tstate);

// it's important that both a and b are locked here.
// if the Py_BEGIN_CRITICAL_SECTION2 only pushed
// a critical section containing b, then only b will be locked

Py_END_CRITICAL_SECTION2();
Py_END_CRITICAL_SECTION();

I think we could probably get something to work safely along the lines of "only lock b", but I think it'll be complicated.

[colesbury]

#144966 is a small change that I'd be comfortable backporting to 3.14.

It'd be great to add an optimization for CRITICAL_SECTION2, but I think that will be a lot more complicated.

[yilei]

Makes sense, thanks @colesbury!