Problem with constructor for a class with custom operator new

[dimar]

Hello, all!

I have a problem exposing a class with custom operator new (custom allocation in a heap book).
Here is an example:

#include <pybind11/pybind11.h>
namespace py = pybind11;

class Example {
public:
 static void* operator new(std::size_t size);
 static void operator delete(void* p, std::size_t size);
 Example() {}
};

PYBIND11_MODULE(test, m) {
 m.doc() = "test";
 py::class_<Example>(m, "Example")
 .def(py::init<>())
 ;
}

The problem arises due to the fact that this statement

 .def(py::init<>())

is a shorthand for

.def("__init__", [](Example &instance) {new (&instance) Example();}

which results in the next error:
pybind11/pybind11.h:1320:60: error: no matching function for call to ‘Example::operator new(sizetype, Base*&)’ cl.def("__init__", [](Base *self_, Args... args) { new (self_) Base(args...); }, extra...);

It worked in Boost.Python out-of-the-box, because they have, apparently, different way of creating a constructor's bindings.

How it is possible to resolve this problem?

[wjakob]

Does it work if you also add dummy placement new operator, i.e.

void *operator new(size_t, void *ptr) { return ptr; }
void *operator new[](size_t, void *ptr) { return ptr; }

[dimar]

Thank you for the answer, Wenzel!
This code compiles:

static void*		operator new(size_t, void *ptr) { return ptr; }
static void*		operator new[](size_t, void *ptr) { return ptr; }

And it works even without dummy operator, just the first line. The difference with code in the question is the presence of void* ptr.
But I won't be able to change code in the library I am exporting. I can write a simple wrapper class, in which I can put overload of new and delete, it compiles. But I don't think that it is a nice workaround. I just wonder may be there is another way, since it worked in boost.python without any changes.

[jagerman]

Edit for future readers: #805 also includes changes that makes the workaround described below unnecessary.

---

PR #805 can help with a class that doesn't support placement new: it can handle a pointer, so you could write:

cl.def(py::init([](Arg a, int b) { return new Class(a, b); }))

We could, in theory, automatically map a py::init<Arg, int>() to that when the class doesn't support placement new initialization, but it might also be simpler to just document the above as the approach when a class doesn't support placement new.

[dimar]

Thank you, Jason!
Sorry, haven't found this discussion.
I checked the branch you are referring to (with factory), and added the code you suggested:

.def(py::init([]() { return new Example(); }))

It compiles, but when I try to import this module into python (v.2.7), I get the next error
ImportError: ./test.so: undefined symbol: _ZN7ExamplenwEm
which is due to the placement new, the same will be for the delete.

[jagerman]

I think that error is happening because your operator new and operator delete in your original example are missing an actual implementation.

I've added a test to the PR #805 branch where this is working.

[dimar]

Thank you, Jason!
Yes, you are right - I forgot about the implementation.
If placement new and delete are implemented as in your test it works fine, but in the library for which I am writing bindings they are implemented differently - they allocate and de-allocate heap pages in a heap book. It compiles, but I have an error while trying to create an instance of the class in Python (v.2.7).
I do not understand what causes the problem and for now I do not have an access to the source code of the library, but only to headers.

So the example looks like this:

class Example {
public:
 static void* operator new(std::size_t size);
 static void operator delete(void* p, std::size_t size);
 Example() {}
private:
 static MyHeapBook heapBook;
};

#define PAGE_SIZE 4096
MyHeapBook Example::heapBook(sizeof(Example), PAGE_SIZE);

void *Example::operator new(std::size_t size) {
 py::print("operator new called");
 return heapBook.allocate();
}
void Example::operator delete(void *p, std::size_t size) {
 py::print("operator delete called");
 heapBook.free(p);
}

PYBIND11_MODULE(test, m) {
 m.doc() = "test";
 py::class_<Example>(m, "Example")
 .def(py::init([]() { return new Example(); }))
 ;
}

where MyHeapBook is defined like this (I am omitting a lot of stuff, because I do not have an access to the source code):

class MyHeapPage;

class MyHeapBook {
public:
 // Constructors and destructors
 MyHeapBook(std::size_t objectSize, unsigned int numberOfObjectsInPage);
 virtual ~MyHeapBook();
 // Memory management
 void* allocate(); // Allocates memory for one object
 void free(void* objectPointer); // Frees the memory at address objectPointer
 /*...*/
private:
 MyHeapPage* firstPage; // The first page
 void* firstFreeObject; // The address of the first free object
 std::size_t objectSize; // The number of bytes per object
 unsigned int numberOfObjectsInPage; // The number of objects in the page
 std::mutex *mutex;
 /*...*/
};
void* MyHeapBook::allocate() {
 mutex->lock();
 if (firstFreeObject == 0) {
 firstPage = new MyHeapPage(objectSize,
 numberOfObjectsInPage, firstPage); // add a new memory page
 firstFreeObject = firstPage->objectArray;
 }
 void* objectPointer = (char*)firstFreeObject + sizeof(void*); // skip the header
 firstFreeObject = (void*)(*((std::size_t*)firstFreeObject)); // move to the next free object
 mutex->unlock();
 return objectPointer;
}
void MyHeapBook::free(void* objectPointer) {
 mutex->lock();
 // the location of the header of the freed object
 char* headerLocation = (char*)objectPointer - sizeof(void*);
 // make the previous "first free object" the next object of the freed object
 *((std::size_t*)headerLocation) = (std::size_t)(firstFreeObject);
 // the freed object becomes the first free object
 firstFreeObject = (void*)headerLocation;
 mutex->unlock();
}

class MyHeapPage {
public:
 // Constructors and destructors
 MyHeapPage(std::size_t objectSize,
 unsigned int numberOfObjectsInPage, MyHeapPage* nextPage);
 virtual ~MyHeapPage();
 /*...*/
private:
 friend class MyHeapBook;
 void* objectArray; // The object array
 MyHeapPage* nextPage; // The next page in the heap book
 /*...*/
};

And the error in Python is:

In [1]: import test
In [2]: test.Example()
operator new called
operator new called
*** Error in `python': free(): invalid pointer: 0x000000000159bda8 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fe0e6fc97e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7fe0e6fd237a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7fe0e6fd653c]
./test.so(_ZN8pybind116detail20call_operator_deleteEPv+0x18)[0x7fe0e2f5c9af]
./test.so(_ZN8pybind116class_I7ExampleIEE7deallocERKNS_6detail16value_and_holderE+0x49)[0x7fe0e2f65334]
./test.so(_ZN8pybind116detail14clear_instanceEP7_object+0x118)[0x7fe0e2f5751f]
./test.so(_ZN8pybind116detail18init_factory_resetEPNS0_8instanceE+0xe1)[0x7fe0e2f5d7a3]
./test.so(+0x71983)[0x7fe0e2f4e983]
./test.so(+0x7169d)[0x7fe0e2f4e69d]
./test.so(+0x715d9)[0x7fe0e2f4e5d9]
./test.so(+0x726b3)[0x7fe0e2f4f6b3]
./test.so(+0x72642)[0x7fe0e2f4f642]
./test.so(+0x721c7)[0x7fe0e2f4f1c7]
./test.so(+0x7223d)[0x7fe0e2f4f23d]
./test.so(_ZN8pybind1112cpp_function10dispatcherEP7_objectS2_S2_+0xa5a)[0x7fe0e2f5ae55]
...
...

Any ideas on the source of the error or on a workaround are appreciated!
Again, it worked in Boost.Python with def(boost::python::init<>()), so the problem is due to the usage of placement new and with pointers, I guess.

[jagerman]

What pybind with the factory PR is essentially:

• When the new type is created, a new pointer is is allocated with T::operator new(sizeof(T)) and stored in the part of the python object that pybind manages.
• When the factory init is called, it gets a new pointer, calls T::operator delete(oldptr); on the existing pointer in the object, then stores the new pointer in the object instead.

Ah, I see the issue: your class doesn't have T::operator delete(oldptr), and we aren't handling that.

[dimar]

Thank you, Jason!
Yes, you are right, it is because of the second argument in operator delete(void* p, std::size_t size). If I use operator delete(void* p) instead, it works, no errors.
The second argument in operator delete is used for some internal checking and cannot be omitted. Is it possible to include support for, maybe, variadic operator delete in pybind11?

[jagerman]

#952 should solve the delete problem.

[dimar]

Thank you very much, Jason!
Checked on the example and on a real code - works like a charm!

[jagerman]

Thanks for the report. (I'm reopening for bookkeeping just until the PRs get merged).