Summary
ChildErrorHarness.test_main in cuda_core/tests/memory_ipc/test_errors.py is marked @pytest.mark.flaky(reruns=2), but a rerun can never pass. The test instance is reused across reruns, and state left over from the failed attempt makes every retry fail while pickling the test object for the child process. The retries therefore hide the first attempt's real failure instead of retrying it.
Mechanism
PARENT_ACTION stores the allocated buffer on the instance, e.g. self.buffer = mr2.allocate(...) in TestDanglingBuffer, with mr2 tracked in self._extra_mrs.- The
finally block closes every resource in self._extra_mrs. - On rerun,
test_main calls multiprocessing.Process(target=self.child_main, ...). With the spawn start method, process.start() pickles the bound method and therefore self, including the stale self.buffer whose memory resource is now closed. - Pickling the buffer calls
_deep_reduce_device_memory_resource, which reads mr.allocation_handle and raises:
RuntimeError: DeviceMemoryResource has been closed
when serializing cuda.core._memory._buffer.Buffer object
when serializing test_errors.TestDanglingBuffer state
...
when serializing multiprocessing.context.Process object
So the pytest report shows this error for both reruns, and the failure that triggered the first rerun is never printed.
Where it was seen
Free-threaded CI job (Python 3.14t, CUDA 13.0.2, single L4) on PR #2750, TestDanglingBuffer::test_main[DeviceMR] and [PinnedMR]: two RERUN lines followed by PARALLEL FAILED, with the pickling error above as the reported cause.
https://github.com/NVIDIA/cuda-python/actions/runs/33913892109/job/101159542389
The same RERUN, RERUN, PARALLEL FAILED signature for TestDanglingBuffer appears on a main run without that PR:
https://github.com/NVIDIA/cuda-python/actions/runs/33820049080
Suggested fix
Any one of these makes reruns meaningful again:
- Reset per-attempt state at the top of
test_main (self.buffer = None, or delete it in finally), so nothing from a previous attempt is pickled. - Make the child entry point a module-level function that receives only picklable arguments, so
self is never pickled. - Define
__getstate__ on the harness to exclude buffers and memory resources.
Separately, the first-attempt failure itself is still unknown and worth capturing once reruns stop masking it; -rR in the pytest summary flags would show rerun tracebacks.
Summary
ChildErrorHarness.test_mainincuda_core/tests/memory_ipc/test_errors.pyis marked@pytest.mark.flaky(reruns=2), but a rerun can never pass. The test instance is reused across reruns, and state left over from the failed attempt makes every retry fail while pickling the test object for the child process. The retries therefore hide the first attempt's real failure instead of retrying it.Mechanism
PARENT_ACTIONstores the allocated buffer on the instance, e.g.self.buffer = mr2.allocate(...)inTestDanglingBuffer, withmr2tracked inself._extra_mrs.finallyblock closes every resource inself._extra_mrs.test_maincallsmultiprocessing.Process(target=self.child_main, ...). With thespawnstart method,process.start()pickles the bound method and thereforeself, including the staleself.bufferwhose memory resource is now closed._deep_reduce_device_memory_resource, which readsmr.allocation_handleand raises:So the pytest report shows this error for both reruns, and the failure that triggered the first rerun is never printed.
Where it was seen
Free-threaded CI job (Python 3.14t, CUDA 13.0.2, single L4) on PR #2750,
TestDanglingBuffer::test_main[DeviceMR]and[PinnedMR]: twoRERUNlines followed byPARALLEL FAILED, with the pickling error above as the reported cause.https://github.com/NVIDIA/cuda-python/actions/runs/33913892109/job/101159542389
The same
RERUN, RERUN, PARALLEL FAILEDsignature forTestDanglingBufferappears on amainrun without that PR:https://github.com/NVIDIA/cuda-python/actions/runs/33820049080
Suggested fix
Any one of these makes reruns meaningful again:
test_main(self.buffer = None, or delete it infinally), so nothing from a previous attempt is pickled.selfis never pickled.__getstate__on the harness to exclude buffers and memory resources.Separately, the first-attempt failure itself is still unknown and worth capturing once reruns stop masking it;
-rRin the pytest summary flags would show rerun tracebacks.