Crash when importing torch

[pasbi]

Issue description

My application crashes when pybind11::scoped_interpreter is destructed after importing torch.

Reproducible example code

#include <pybind11/embed.h>

int main()
{
 pybind11::scoped_interpreter guard{};
 pybind11::exec("import torch");
 // pybind11::module::import("torch"); // same problem
}

Note that the import itself works like a charm.
In the real code, I am using torch and it does what it is supposed to do.
The crash occurs when guard goes out of scope, i.e., when it is destructed.

I guess it's a double-free issue, however, I can't narrow down the problem in torch, I believe the problem is somewhere deep in its C-implementation. Importing tensorflow, numpy, matplotlib and other modules does not make such trouble.
I already asked in the gitter lobby, but I didn't get an answer.

[pasbi]

Just for reference, it works without pybind11 (see code). However, I want to use pybind11...

#include <Python.h>

int main(int argc, char* argv[])
{
 wchar_t *program = Py_DecodeLocale(argv[0], nullptr);
 if (program == nullptr) exit(1);
 Py_SetProgramName(program);
 Py_Initialize();
 PyRun_SimpleString("import torch");
 if (Py_FinalizeEx() < 0) exit(120);
 PyMem_RawFree(program);
}

[YannickJadoul]

I can't reproduce with pybind11 2.5.0. If this is still an issue, please reopen!

[pasbi]

The issue seems to be fixed in 2.5.0.
I can't reproduce it anymore.

[YannickJadoul]

Thanks for confirming, @pasbi!

[whybeyoung]

we still have problem in 2.10.0

c++ code:

#include "pybind11/embed.h"
#include <iostream>
#include <thread>
#include <chrono>
#include <sstream>

namespace py = pybind11;
using namespace std::chrono_literals;

class Wrapper
{
public:
 Wrapper()
 {
 py::gil_scoped_acquire acquire;
 _obj = py::module::import("wrapper").attr("Wrapper")();
 _wrapperInit = _obj.attr("wrapperInit");
 _wrapperFini = _obj.attr("wrapperFini");

 }

 ~Wrapper()
 {
 _wrapperInit.release();
 _wrapperFini.release();
 }

 int wrapperInit()
 {
 py::gil_scoped_acquire acquire;
 return _wrapperInit(nullptr).cast<int>();
 }

 void wrapperFini(int x)
 {
 py::gil_scoped_acquire acquire;
 _wrapperFini(x);
 }

 private:
 py::object _obj;
 py::object _wrapperInit;
 py::object _wrapperFini;
};
void thread_func(int iteration)
{
 Wrapper w;

 for (int i = 0; i < 1; i++)
 {
 w.wrapperInit();
 std::stringstream msg;
 msg << "iteration: " << iteration << " thread: " << std::this_thread::get_id() << std::endl;
 std::cout << msg.str();
 std::this_thread::sleep_for(100ms);
 }
}

int main() {
 py::scoped_interpreter guard{};
 py::gil_scoped_release release; // add this to release the GIL

 std::vector<std::thread> threads;

 for (int i = 0; i < 1; ++i)
 threads.push_back(std::thread(thread_func, 1));

 for (auto& t : threads)
 t.join();

 return 0;
}

wrapper.py code is

class Wrapper():
 serviceId = "mmocr"
 version = "backup.0"


 '''
 服务初始化
 @param config:
 插件初始化需要的一些配置，字典类型
 key: 配置名
 value: 配置的值
 @return
 ret: 错误码。无错误时返回0
 '''

 def wrapperInit(cls, config: {}) -> int:
 import torch
 print(config)

 print("Initializing ..")
 return 0

 def wrapperFini(cls) -> int:
 return 0

[Davidnet]

Hi I still can replicate that one in here:

#4137 (comment)