[BUG]: Creation of py::tuple or py::list causes a crash with policy py::call_guard<py::gil_scoped_release>() on Ubuntu 22

[ylegoc]

Required prerequisites

• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• Consider asking first in the Gitter chat room or in a Discussion.

Problem description

After some tests on my Ubuntu 22, I remarked that pybind11 was crashing at the creation of a py::tuple or py::list object in my project.

Calling the function test() defined below causes a segmentation fault.

Crashes on:

• Ubuntu 22 with pybind11 2.9.2 and also 2.4.3
• Ubuntu 22 with pybind11 2.9.1 from distrib package

No crash on:

• Ubuntu 20 with pybind11 2.9.2 or 2.4.3

By looking deeper into the problem, I found out:
Using the policy py::call_guard<py::gil_scoped_release>() causes the crash. Without the policy it works well.

I tried different compilation options that all resulted in a crash.

To summarize:
The creation of a py::tuple or py::list object causes a crash on my Ubuntu 22 (gcc 11.2.0, Intel(R) Core(TM) i5-8500) with the policy py::call_guard<py::gil_scoped_release>().

Reproducible example code

.def_static("test", [] {
 py::tuple tup = py::make_tuple(42, py::none(), "spam");
}, py::call_guard<py::gil_scoped_release>())

[Skylion007]

@ylegoc Are you able find the specific commit which broke things? Also does it still crash when using master?

[ylegoc]

I removed the word regression from the title because this is a form of regression.
The simple code compiles and executes well on Ubuntu 20 (gcc 9.4).
However compiled on Ubuntu 22 (gcc 11.2), it crashes on Ubuntu 22.

That is why it seems to come from the gcc version but not from a bad commit.
I will try on master.

[henryiii]

We are fine on GCC 11 from AlmaLinux 9. Are you sure you don't have a race condition that happens to not trigger on the "old" system?

I'd recommend trying our test suite on your system.

[ylegoc]

I run the test suite on my system and it succeeds. Then I modified test_pytypes.cpp:
m.def("get_tuple", []() { return py::make_tuple(42, py::none(), "spam"); });
m.def("get_tuple_gil", []() { return py::make_tuple(42, py::none(), "spam"); }, py::call_guard<py::gil_scoped_release>());

And test_pytypes.py:
assert m.get_tuple() == (42, None, "spam")
assert m.get_tuple_gil() == (42, None, "spam")

And it's crashing in the get_tuple_gil() call with a segmentation fault.

However maybe this is not a good practice and I should release the gil only for non pybind11 code?

[gst]

FWIW : I tried g++-11 + Ubuntu 20.04 (focal) and no segfault.

edit: but: I'm not sure it's safe to call py::make_tuple with a released GIL actually. that looks to be the issue eventually ?

[henryiii]

You can’t modify the Python memory model with the GIL released. That’s why there’s a GIL. Making a tuple very much modifies Python memory. The crash is random and could be compiler dependent, it’s UB.