gh-140544: Always assume that thread locals are available

[ZeroIntensity]

Python has required thread local support since 3.12. By assuming that thread locals are always supported, we can improve the performance of third-party extensions by allowing them to access the attached thread and interpreter states directly.

cc @cdce8p

• Issue: Store current interpreter as a thread local variable #140544

[cdce8p]

Hmm, I'm still getting the same error I mentioned earlier.

 In file included from /.../cpython/Include/internal/pycore_object.h:12: (diff)
In file included from /.../cpython/Include/internal/pycore_gc.h:12: (diff)
/.../cpython/Include/internal/pycore_pystate.h:93:8: error: unknown type name '_Py_thread_local' (diff)
extern _Py_thread_local PyThreadState *_Py_tss_tstate; (diff)
^ (diff)
/.../cpython/Include/internal/pycore_pystate.h:93:38: error: expected ';' after top level declarator (diff)
extern _Py_thread_local PyThreadState *_Py_tss_tstate; (diff)
^ (diff)
; (diff)
/.../cpython/Include/internal/pycore_pystate.h:94:8: error: unknown type name '_Py_thread_local' (diff)
extern _Py_thread_local PyInterpreterState *_Py_tss_interp; (diff)
^ (diff)
/.../cpython/Include/internal/pycore_pystate.h:94:43: error: expected ';' after top level declarator (diff)
extern _Py_thread_local PyInterpreterState *_Py_tss_interp; (diff)
^ (diff)
; (diff)
/.../cpython/Include/internal/pycore_pystate.h:119:12: error: use of undeclared identifier '_Py_tss_tstate' (diff)
return _Py_tss_tstate; (diff)
^ (diff)
/.../cpython/Include/internal/pycore_pystate.h:213:12: error: use of undeclared identifier '_Py_tss_interp' (diff)
return _Py_tss_interp; (diff)
^ (diff)
6 errors generated. (diff)

Somehow the # ifdef Py_BUILD_CORE check fails, even though mypyc defines it.
https://github.com/python/mypy/blob/v1.18.2/mypyc/lib-rt/pythonsupport.h#L16-L18

[cdce8p]

Do I need to pass Py_BUILD_CORE through to my local cpython build? This is my current build setup

make clean
./configure --with-openssl=$(brew --prefix openssl@3)
make -j4 -s

[ZeroIntensity]

Hm, you're defining Py_BUILD_COREafterPython.h has been included, so pyport.h isn't seeing it. I'm not sure if that's something we intended to support.

[cdce8p]

Hm, you're defining Py_BUILD_COREafterPython.h has been included, so pyport.h isn't seeing it. I'm not sure if that's something we intended to support.

That hasn't been an issue so far. The check here works fine / if I remove Py_BUILD_CORE from mypyc I get the error as expected.

cpython/Include/internal/pycore_mimalloc.h

Lines 4 to 6 in f5394c2

	#ifndefPy_BUILD_CORE
	# error "this header requires Py_BUILD_CORE define"
	#endif

[ZeroIntensity]

I just pushed a new commit, where _Py_thread_local is moved to pycore_pystate.h. Can you test that?

[cdce8p]

I just pushed a new commit, where _Py_thread_local is moved to pycore_pystate.h. Can you test that?

This works now.

--

Hm, you're defining Py_BUILD_COREafterPython.h has been included, so pyport.h isn't seeing it. I'm not sure if that's something we intended to support.

I missed that mypyc is defining Py_BUILD_CORE before including any internal/** headers. Though that's much simpler than doing the same for Python.h which is included nearly everywhere.

[kumaraditya303]

@cdce8p Would you please check that mypyc still works after the new changes?

[cdce8p]

@cdce8p Would you please check that mypyc still works after the new changes?

Just ran the whole test suite again. Looks good.