`TemporaryDirectory.__exit__` sometimes raises bad PermissionError

[kfrydel]

Bug report

TemporaryDirectory.__exit__ usually raises PermissionError if another process keeping a handle to a file in the tmp directory has just finished its execution.

Bug description:

Please consider the following block of code:

import multiprocessing
import os
import tempfile
import time


def _open_func(file_path):
 with open(file_path, "w"):
 time.sleep(1000)


def test():
 with tempfile.TemporaryDirectory(suffix="used_by_another_process") as dir_path:
 file_path = os.path.join(dir_path, "file_being_used")
 proc = multiprocessing.Process(target=_open_func, args=(file_path,))
 proc.start()
 while not os.path.exists(file_path):
 time.sleep(0.1)
 proc.terminate()
 proc.join()


if __name__ == "__main__":
 test()

Despite the child process being terminated and joined, the __exit__ method of TemporaryDirectory sometimes raises PermissionError:

Traceback (most recent call last):
 File "C:\Python\python.3.12.1\tools\Lib\shutil.py", line 634, in _rmtree_unsafe
 os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\kamil\\AppData\\Local\\Temp\\tmpvaou_oibused_by_another_process\\file_being_used'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "C:\Users\kamil\tmp_test\tmpdir.py", line 24, in <module>
 test()
 File "C:\Users\kamil\tmp_test\tmpdir.py", line 13, in test
 with tempfile.TemporaryDirectory(suffix="used_by_another_process") as dir_path:
 File "C:\Python\python.3.12.1\tools\Lib\tempfile.py", line 946, in __exit__
 self.cleanup()
 File "C:\Python\python.3.12.1\tools\Lib\tempfile.py", line 950, in cleanup
 self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
 File "C:\Python\python.3.12.1\tools\Lib\tempfile.py", line 930, in _rmtree
 _shutil.rmtree(name, onexc=onexc)
 File "C:\Python\python.3.12.1\tools\Lib\shutil.py", line 808, in rmtree
 return _rmtree_unsafe(path, onexc)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "C:\Python\python.3.12.1\tools\Lib\shutil.py", line 636, in _rmtree_unsafe
 onexc(os.unlink, fullname, err)
 File "C:\Python\python.3.12.1\tools\Lib\tempfile.py", line 905, in onexc
 _os.unlink(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\kamil\\AppData\\Local\\Temp\\tmpvaou_oibused_by_another_process\\file_being_used'

It does not reproduce in 100% but most executions fail. I can reproduce it only using Python 3.12.1. It does not happen to me on 3.12.0 or 3.11. It seems to be a regression in the last release.
With some small sleep after the proc.join() it stops reproducing so it looks like a kind of race condition.

The Windows version I use is:
Edition: Windows 10 Enterprise
Version: 21H2
OS build: 19044.3693

CPython versions tested on:

3.12

Operating systems tested on:

Windows

Linked PRs

• gh-113009: Fix multiprocessing Process.terminate() on Windows #113128
• [3.12] gh-113009: Fix multiprocessing Process.terminate() on Windows (GH-113128) #113177
• [3.11] gh-113009: Fix multiprocessing Process.terminate() on Windows (GH-113128) #113178

[savannahostrowski]

I was also able to reproduce this on Windows 11 Pro (22H2) with 3.11.7 and 3.12.1. It reproduces reliably with the code snippet above.

I also tested this on Linux (Debian) with 3.11, 3.12.0 and 3.12.1 and was unable to repro.

[serhiy-storchaka]

The cause is the same as in #109608 and #66223. There is also similar issue with directories: #86962.

[kfrydel]

I have spent some time debugging another issue in the same piece of code that revealed this bug and probably I have found the cause of this as well. So consider the following block of code:

import multiprocessing
import os
import tempfile
import time

import psutil


def _open_func(file_path):
 with open(file_path, "w"):
 time.sleep(1000)


def test(i):
 with tempfile.TemporaryDirectory(suffix="used_by_another_process") as dir_path:
 file_path = os.path.join(dir_path, "file_being_used")
 proc = multiprocessing.Process(target=_open_func, args=(file_path,))
 proc.start()
 while not os.path.exists(file_path):
 time.sleep(0.1)
 proc.terminate()
 proc.join()
 try:
 process = psutil.Process(proc.pid)
 print(f'{process=} {i=}')
 assert False
 except psutil.Error:
 pass


if __name__ == "__main__":
 for i in range(1000):
 test(i)

Running psutil.Process, since it takes some time, prevents from falling into PermissionError issue but sometimes the above code prints the child process information and finally fails on the assert False.

So the real problem here is probably in fact that invoking proc.terminate(); proc.join() finishes its execution before the process is stopped.

It looks like the regression was introduced by this change: a2074911ba Please note that it is not present in 3.12.0 but in 3.12.1 it is. Previously, the terminate method was not setting self.returncode so the wait method was indeed waiting. But now, since returncode is always set by terminate, wait does nothing. Reverting the change fixes both issues.
CC: @vstinner as the author of the change.

[vstinner]

Previously, the terminate method was not setting self.returncode so the wait method was indeed waiting. But now, since returncode is always set by terminate, wait does nothing.

Do you mean that in terminate(), when TerminateProcess() raises PermissionError and GetExitCodeProcess() returns an exit code different than STILL_ACTIVE, the process is in fact still running?

[vstinner]

Please log the returncode. Example:

def test():
 with tempfile.TemporaryDirectory(suffix="used_by_another_process") as dir_path:
 file_path = os.path.join(dir_path, "file_being_used")
 proc = multiprocessing.Process(target=_open_func, args=(file_path,))
 print(repr(proc))
 proc.start()
 while not os.path.exists(file_path):
 time.sleep(0.1)
 proc.terminate()
 print(f"terminate: {proc._popen.returncode=}")
 proc.join()
 print(f"join: {proc._popen.returncode=}")

[kfrydel]

No, I mean that we now set returncode in the else clause:

cpython/Lib/multiprocessing/popen_spawn_win32.py

Line 134 in a207491

self.returncode = -signal.SIGTERM

Thus wait skips waiting:

cpython/Lib/multiprocessing/popen_spawn_win32.py

Line 104 in a207491

if self.returncode is None:

and we may end up with running process for some time after Process.join finishes.

The output of the program with additional debug logs:

terminate: proc._popen.returncode=-15
join: proc._popen.returncode=-15