Skip to content

cuda.core tests: ChildErrorHarness reruns always fail on stale pickled state, masking the real flake #2775

Description

@Andy-Jost

Summary

ChildErrorHarness.test_main in cuda_core/tests/memory_ipc/test_errors.py is marked @pytest.mark.flaky(reruns=2), but a rerun can never pass. The test instance is reused across reruns, and state left over from the failed attempt makes every retry fail while pickling the test object for the child process. The retries therefore hide the first attempt's real failure instead of retrying it.

Mechanism

  1. PARENT_ACTION stores the allocated buffer on the instance, e.g. self.buffer = mr2.allocate(...) in TestDanglingBuffer, with mr2 tracked in self._extra_mrs.
  2. The finally block closes every resource in self._extra_mrs.
  3. On rerun, test_main calls multiprocessing.Process(target=self.child_main, ...). With the spawn start method, process.start() pickles the bound method and therefore self, including the stale self.buffer whose memory resource is now closed.
  4. Pickling the buffer calls _deep_reduce_device_memory_resource, which reads mr.allocation_handle and raises:
RuntimeError: DeviceMemoryResource has been closed
when serializing cuda.core._memory._buffer.Buffer object
when serializing test_errors.TestDanglingBuffer state
...
when serializing multiprocessing.context.Process object

So the pytest report shows this error for both reruns, and the failure that triggered the first rerun is never printed.

Where it was seen

Free-threaded CI job (Python 3.14t, CUDA 13.0.2, single L4) on PR #2750, TestDanglingBuffer::test_main[DeviceMR] and [PinnedMR]: two RERUN lines followed by PARALLEL FAILED, with the pickling error above as the reported cause.
https://github.com/NVIDIA/cuda-python/actions/runs/33913892109/job/101159542389

The same RERUN, RERUN, PARALLEL FAILED signature for TestDanglingBuffer appears on a main run without that PR:
https://github.com/NVIDIA/cuda-python/actions/runs/33820049080

Suggested fix

Any one of these makes reruns meaningful again:

  • Reset per-attempt state at the top of test_main (self.buffer = None, or delete it in finally), so nothing from a previous attempt is pickled.
  • Make the child entry point a module-level function that receives only picklable arguments, so self is never pickled.
  • Define __getstate__ on the harness to exclude buffers and memory resources.

Separately, the first-attempt failure itself is still unknown and worth capturing once reruns stop masking it; -rR in the pytest summary flags would show rerun tracebacks.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    triageNeeds the team's attention

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions