Data race between `resize_buffer_lock_held` and `_Py_DecRefShared` on a shared `BytesIO` buffer

[Naserume]

Bug report

Bug description:

In the free-threaded build, a whole-buffer BytesIO.read() returns self->buf by reference (Py_NewRef) instead of a copy,

cpython/Modules/_io/bytesio.c

Lines 432 to 437 in 9e863fa

	if (size > 1 &&
	self->pos == 0 && size == PyBytes_GET_SIZE(self->buf) &&
	FT_ATOMIC_LOAD_SSIZE_RELAXED(self->exports) == 0) {
	self->pos += size;
	return Py_NewRef(self->buf);
	}

so resize_buffer_lock_held() can reallocate that same self->buf in place (_PyBytes_Resize)

cpython/Modules/_io/bytesio.c

Lines 171 to 178 in 9e863fa

	if (SHARED_BUF(self)) {
	if (unshare_buffer_lock_held(self, alloc) < 0)
	return -1;
	}
	else {
	if (_PyBytes_Resize(&self->buf, alloc) < 0)
	return -1;
	}

while another thread concurrently decrefs the read-returned reference (_Py_DecRefShared), racing on the same buffer object without synchronization.

Reproducer:

import io
from threading import Thread, Barrier

shared = io.BytesIO(b'x' * 64)
N_W, N_R = 2, 8
barrier = Barrier(N_W + N_R)

def writer():
 barrier.wait()
 for i in range(20000):
 try:
 shared.seek(0)
 shared.write(b'a' * (64 + (i & 63)))
 except Exception:
 pass

def reader():
 barrier.wait()
 for _ in range(20000):
 try:
 shared.seek(0)
 shared.read()
 except Exception:
 pass

if __name__ == "__main__":
 threads = [Thread(target=writer) for _ in range(N_W)]
 threads += [Thread(target=reader) for _ in range(N_R)]
 for t in threads: t.start()
 for t in threads: t.join()

TSAN Report:

==================
WARNING: ThreadSanitizer: data race (pid=2292869)
 Read of size 8 at 0x7fffbc080190 by thread T2:
 #0 _PyObject_MiRealloc /cpython/Objects/obmalloc.c:360:13 
 #1 PyObject_Realloc /cpython/Objects/obmalloc.c:1731:12 
 #2 _PyBytes_Resize /cpython/Objects/bytesobject.c:3389:9 
 #3 resize_buffer_lock_held /cpython/./Modules/_io/bytesio.c:176:13 
 #4 write_bytes_lock_held /cpython/./Modules/_io/bytesio.c:216:13 
 #5 _io_BytesIO_write_impl /cpython/./Modules/_io/bytesio.c:782:20 
 #6 bytesio_setstate_lock_held /cpython/./Modules/_io/bytesio.c:927:14 
 #7 bytesio_setstate /cpython/./Modules/_io/bytesio.c:980:11 
 #8 _PyEval_EvalFrameDefault /cpython/Python/generated_cases.c.h:4261:35 
 ...
 Previous atomic write of size 8 at 0x7fffbc080190 by thread T8:
 #0 _Py_atomic_compare_exchange_ssize /cpython/./Include/cpython/pyatomic_gcc.h:130:10 
 #1 _Py_DecRefSharedIsDead /cpython/Objects/object.c:406:15 
 #2 _Py_DecRefSharedDebug /cpython/Objects/object.c:425:9 
 #3 _Py_DecRefShared /cpython/Objects/object.c:433:5 
 #4 _PyEval_EvalFrameDefault /cpython/Python/generated_cases.c.h 
 ...
SUMMARY: ThreadSanitizer: data race /cpython/Objects/obmalloc.c:360:13 in _PyObject_MiRealloc

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-151640: Track sharing of BytesIO buffer in free-threaded builds #151651
• [3.15] gh-151640: Track sharing of BytesIO buffer in free-threaded builds (GH-151651) #153151
• [3.14] gh-151640: Track sharing of BytesIO buffer in free-threaded builds (GH-151651) #153153