SIGSEGV with generators in free-threaded build

[rostan-t]

Crash report

What happened?

Trying to iterate on a generator from multiple threads under the free-threaded build results in a segmentation fault.

Here is a minimal repro:

import concurrent.futures


def gen():
 while True:
 yield


it = gen()
with concurrent.futures.ThreadPoolExecutor() as executor:
 while True:
 _ = executor.submit(lambda: next(it))

The issue seems to be specific to generators as other kinds of iterators work well in parallel.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.14.0a0 experimental free-threading build (heads/main:c3b6dbff2c, Jun 10 2024, 16:54:16) [GCC 11.4.0]

Linked PRs

• gh-120321: Lock gen_send #120327
• gh-120321: Make gi_frame_state transitions atomic in FT build #142599
• gh-120321: Fix TSan reported race in gen_clear_frame #142995
• gh-120321: Avoid -Wunreachable-code warning on Clang #143022
• gh-120321: Make gen.gi_frame.clear() thread-safe #143112
• gh-120321: Fix TSan reported races on gi_frame_state #143128
• gh-120321: Add missing "return false" in gen_try_set_executing #144291
• gh-120321: Make gi_yieldfrom thread-safe in free-threading build #144292
• gh-120321: Add gi_state, cr_state, and ag_state attributes #144409
• gh-120321: fix thread safety of concurrently iterating over async generators #155025

[Fidget-Spinner]

Thanks for the repro. Looking into this.

[Fidget-Spinner]

Confirmed, with pydebug and tsan.

[Zheaoli]

May I take a chance to dive into this issue?

[Fidget-Spinner]

@Zheaoli sorry I assigned myself on this one I have a fix already and will put it up shortly. Do you have any specific issues in mind you would like to work on? I could help mentor/guide you on stuff you're interested in if I can help!

[Zheaoli]

sorry I assigned myself on this one I have a fix already and will put it up shortly

No problem!

Do you have any specific issues in mind you would like to work on? I could help mentor/guide you on stuff you're interested in if I can help!

For now, I'm a fresh man try to be more familiar with the CPython codebase.

I'm now working on #104280 to make dtrace more functional to help debug the process. I'm also interested in JIT and Tier2(especially to enhance the observerability for the Python process). But I'm still trying to find the issue I can fix and the thing I can do(lol

Thanks for knowing that you will be willing to mentor/guide me lol!

[Fidget-Spinner]

@Zheaoli please send me an email I'll help give some pointers privately