gh-114271: Make _thread.ThreadHandle thread-safe in free-threaded builds

[mpage]

We protect the mutable state of ThreadHandle using a _PyOnceFlag. Concurrent operations (i.e. join) on ThreadHandle block until it is their turn to execute or an earlier operation succeeds. Once an operation has been applied successfully all future operations complete immediately.

The join() method is now idempotent. It may be called multiple times but the underlying OS thread will only be joined once. After join() succeeds, any future calls to join() will succeed immediately.

The detach() method has been removed; nothing was using it.

• Issue: Make _threadmodule.c thread-safe in --disable-gil builds #114271

[mpage]

@colesbury - Would you please have a look and add the "skip news" label to this? In order to limit the size of the changes I'd like to get this merged first and then stack the tstate_lock stuff on top.

[mpage]

@colesbury - This should be good to go. Would you please have another look?

[colesbury]

Oh, the thread id reuse is tricky.

I'd like to avoid adding _PyEventRc if possible. The event doesn't seem like a great fit for ThreadHandleObject because we are never waiting on the event -- we're just using it like an atomic flag.

What about something like:

1. Make thread_is_exiting a plain variable that we use atomic operations to access, like int thread_is_exiting (because we don't have atomic ops on bool currently)
2. Pass the ThreadHandleObject to do_start_new_thread instead of having another object with a separate lifetime.

[mpage]

I'd like to avoid adding _PyEventRc if possible.

Me too! I thought pretty hard about this and couldn't come up with a better solution. Plus, I think we're going to need it later anyway if we want to be able to implement join with a timeout, at which point we will be waiting on it.

What about something like: ...

I considered this, but I think we need to set the flag after the PyThreadState has been destroyed. Otherwise, you could end up deadlocking in the following situation:

1. T0 joins T1.
2. T1 finishes. It has a thread-local with a finalizer that joins T1. The flag is set at this point, so we skip the self-join check and deadlock.

Since we have to set the flag after the PyThreadState has been destroyed, it's no longer safe to call Py_DECREF, so we can't use a ThreadHandleObject (or any Python object). We could create a ref-counted flag (or re-use one if it exists), but since we're already going to need the _PyEventRc anyway, I figured we'd just use it here.

[colesbury]

This looks good to me.

@pitrou, would you please look this over? The motivation is to make thread joining thread-safe in the freethreaded build, as well as fix a race condition that can affect both the default and freethreaded build (described by @mpage in #114839)

[pitrou]

In general this looks ok but I think this can be simplified, see below.

[mpage]

@pitrou - I think I've addressed all of your comments except for the question about using bool. A quick grep of the codebase reveals quite a few uses of bool (in central places like obmalloc.c and dictobject.c, among others). Even if there isn't explicit guidance, there's certainly precedent. I've also removed the detach() method. Would you please have another look?

@colesbury - This has changed a bit since you reviewed it. Would you please have another look? In particular, I'd like a sanity check that the use of relaxed atomics for accessing handle state is OK.

[mpage]

The failing macos-13 builds appear unrelated to this PR. They're failing with errors like:

error: call to undeclared function 'is_pad'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
if (py_is_pad(self->win)) {

[colesbury]

A few minor comments below

[mpage]

@pitrou - I think I've addressed everything now. Would you please have a look?

[mpage]

@pitrou@gpshead - Did you want to take a look at this?

[gpshead]

(review in progress, more to come)

[gpshead]

This is interesting... our docs already claim that threads can be joined multiple times. So I wonder why this existing test logic was previously explicitly checking for an error here.

In this sense this change aligns with what our docs claim so a behavior change here could be seen as a bugfix. I do not expect anyone to be depending on subsequent join()s of a thread raising regardless.

[bedevere-app]

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

[mpage]

I have made the requested changes; please review again

[mpage]

@gpshead - Would you please have another look at this?