Use 'raise from' in initialization

[virtuald]

We ran into an issue where a unicode decode exception occurred during initialization, and it was difficult to figure out what was causing the exception without looking directly at the object. Unfortunately, pybind11 currently swallows all exceptions by only retaining the text. This would change it to use the equivalent of raise ImportError('something') from e.

There are several things to be considered here:

• I think this mechanism was introduced in python 3.3?, need to add appropriate #ifdefs, not 100% sure the best way to go about this
• Needs tests I assume?
• Probably should apply this in other places where exceptions might be swallowed, but I think that's more appropriate for another PR?

Suggested changelog entry:

* Added ``py::raise_from`` to enable chaining exceptions

[virtuald]

When y'all are reviewing things, I'd love some initial feedback on this idea.

[YannickJadoul]

We'll get to things in due time. Give us some time, please.

[henryiii]

Can you rebase on master, to see what is failing?

[virtuald]

Fails on Python 2.7, but as noted in the original description:

I think this mechanism was introduced in python 3.3?, need to add appropriate #ifdefs, not 100% sure the best way to go about this

[virtuald]

I changed my mind about this implementation, I think it makes more sense to have a py::raise_from function instead of adding a causes function to error_already_set. Should be good to go now.

[virtuald]

Looks like all the tests will pass now except for the NVIDIA thing which failed because of a download issue.

[rwgk]

Awesome, thanks!

[rwgk]

For future reference it would be ideal to have a full link, even if it's long, e.g. (the code looks the same at first glance):

https://github.com/python/cpython/blob/8f010dc920e1f6dc6a357e7cc1460a7a567c05c6/Python/errors.c#L589

[rwgk]

I looked a bit a the history of errors.c, I think @virtuald's copy is most similar to the oldest version (467ab19 on Oct 21, 2016) that still has any blamed lines today:

https://github.com/python/cpython/blob/467ab194fc6189d9f7310c89937c51abeac56839/Python/errors.c#L405

The diff is:

--- commit_a_467ab194fc6 2021-08-23 16:31:10.128182694 -0700+++ copy_pr2112 2021-08-23 16:32:52.148562421 -0700@@ -1,26 +1,20 @@-static PyObject *-_PyErr_FormatVFromCause(PyObject *exception, const char *format, va_list vargs)-{- PyObject *exc, *val, *val2, *tb;-- assert(PyErr_Occurred());+inline void raise_from(PyObject *type, const char *message) {+ // from cpython/errors.c _PyErr_FormatVFromCause+ PyObject *exc = nullptr, *val = nullptr, *val2 = nullptr, *tb = nullptr;
PyErr_Fetch(&exc, &val, &tb);
+
PyErr_NormalizeException(&exc, &val, &tb);
- if (tb != NULL) {+ if (tb != nullptr) {
PyException_SetTraceback(val, tb);
Py_DECREF(tb);
}
Py_DECREF(exc);
- assert(!PyErr_Occurred());-- PyErr_FormatV(exception, format, vargs);+ PyErr_SetString(type, message);
PyErr_Fetch(&exc, &val2, &tb);
PyErr_NormalizeException(&exc, &val2, &tb);
Py_INCREF(val);
PyException_SetCause(val2, val);
PyException_SetContext(val2, val);
PyErr_Restore(exc, val2, tb);
-- return NULL;
}

[rwgk]

For completeness, the 3 commits I looked at are:

• https://github.com/python/cpython/commits/467ab194fc6/Python/errors.c
• https://github.com/python/cpython/commits/b4bdecd0fc9/Python/errors.c
• https://github.com/python/cpython/commits/438a12dd9d8/Python/errors.c

467ab194fc6 (Serhiy Storchaka 2016-10-21 17:09:17 +0300 602) static PyObject *
b4bdecd0fc9 (Victor Stinner 2019-05-24 13:44:24 +0200 603) _PyErr_FormatVFromCause(PyThreadState *tstate, PyObject *exception,
b4bdecd0fc9 (Victor Stinner 2019-05-24 13:44:24 +0200 604) const char *format, va_list vargs)
467ab194fc6 (Serhiy Storchaka 2016-10-21 17:09:17 +0300 605) {
467ab194fc6 (Serhiy Storchaka 2016-10-21 17:09:17 +0300 606) PyObject *exc, *val, *val2, *tb;
467ab194fc6 (Serhiy Storchaka 2016-10-21 17:09:17 +0300 607) b4bdecd0fc9 (Victor Stinner 2019-05-24 13:44:24 +0200 608) assert(_PyErr_Occurred(tstate));
b4bdecd0fc9 (Victor Stinner 2019-05-24 13:44:24 +0200 609) _PyErr_Fetch(tstate, &exc, &val, &tb);
b4bdecd0fc9 (Victor Stinner 2019-05-24 13:44:24 +0200 610) _PyErr_NormalizeException(tstate, &exc, &val, &tb);
467ab194fc6 (Serhiy Storchaka 2016-10-21 17:09:17 +0300 611) if (tb != NULL) {
467ab194fc6 (Serhiy Storchaka 2016-10-21 17:09:17 +0300 612) PyException_SetTraceback(val, tb);
467ab194fc6 (Serhiy Storchaka 2016-10-21 17:09:17 +0300 613) Py_DECREF(tb);
467ab194fc6 (Serhiy Storchaka 2016-10-21 17:09:17 +0300 614) }
467ab194fc6 (Serhiy Storchaka 2016-10-21 17:09:17 +0300 615) Py_DECREF(exc);
b4bdecd0fc9 (Victor Stinner 2019-05-24 13:44:24 +0200 616) assert(!_PyErr_Occurred(tstate));
467ab194fc6 (Serhiy Storchaka 2016-10-21 17:09:17 +0300 617) 438a12dd9d8 (Victor Stinner 2019-05-24 17:01:38 +0200 618) _PyErr_FormatV(tstate, exception, format, vargs);
467ab194fc6 (Serhiy Storchaka 2016-10-21 17:09:17 +0300 619) b4bdecd0fc9 (Victor Stinner 2019-05-24 13:44:24 +0200 620) _PyErr_Fetch(tstate, &exc, &val2, &tb);
b4bdecd0fc9 (Victor Stinner 2019-05-24 13:44:24 +0200 621) _PyErr_NormalizeException(tstate, &exc, &val2, &tb);
467ab194fc6 (Serhiy Storchaka 2016-10-21 17:09:17 +0300 622) Py_INCREF(val);
467ab194fc6 (Serhiy Storchaka 2016-10-21 17:09:17 +0300 623) PyException_SetCause(val2, val);
467ab194fc6 (Serhiy Storchaka 2016-10-21 17:09:17 +0300 624) PyException_SetContext(val2, val);
b4bdecd0fc9 (Victor Stinner 2019-05-24 13:44:24 +0200 625) _PyErr_Restore(tstate, exc, val2, tb);
467ab194fc6 (Serhiy Storchaka 2016-10-21 17:09:17 +0300 626) 467ab194fc6 (Serhiy Storchaka 2016-10-21 17:09:17 +0300 627) return NULL;

[rwgk]

For easy future reference, this is the merged PR #3215 diff relative to the base version:

--- commit_a_467ab194fc6 2021-08-23 16:31:10.128182694 -0700+++ copy_final 2021-08-23 17:33:04.173103658 -0700@@ -1,19 +1,20 @@-static PyObject *-_PyErr_FormatVFromCause(PyObject *exception, const char *format, va_list vargs)-{- PyObject *exc, *val, *val2, *tb;+inline void raise_from(PyObject *type, const char *message) {+ // Based on _PyErr_FormatVFromCause:+ // https://github.com/python/cpython/blob/467ab194fc6189d9f7310c89937c51abeac56839/Python/errors.c#L405+ // See https://github.com/pybind/pybind11/pull/2112 for details.+ PyObject *exc = nullptr, *val = nullptr, *val2 = nullptr, *tb = nullptr;
assert(PyErr_Occurred());
PyErr_Fetch(&exc, &val, &tb);
PyErr_NormalizeException(&exc, &val, &tb);
- if (tb != NULL) {+ if (tb != nullptr) {
PyException_SetTraceback(val, tb);
Py_DECREF(tb);
}
Py_DECREF(exc);
assert(!PyErr_Occurred());
- PyErr_FormatV(exception, format, vargs);+ PyErr_SetString(type, message);
PyErr_Fetch(&exc, &val2, &tb);
PyErr_NormalizeException(&exc, &val2, &tb);
@@ -21,6 +22,4 @@
PyException_SetCause(val2, val);
PyException_SetContext(val2, val);
PyErr_Restore(exc, val2, tb);
-- return NULL;
}

[rwgk]

Hi all, I'll try to fix up the comment now as per my own suggestion as of a couple weeks ago. Hang on ...

[rwgk]

Unfortunately I cannot push to this branch, therefore I created a helper PR #3215.

[rwgk]

Merged via #3215.