ARROW-5863: [Python] Use atexit module for extension type finalization to avoid segfault

[wesm]

As reported on JIRA, the following script provokes a segfault

#! /usr/bin/env python
import pyarrow
import sys
del sys.modules['pyarrow.lib']

For some reason this does not trigger the destruction of the private _ExtensionTypesInitializer object. Not sure why (Antoine may know). Using the atexit module instead seems to do the trick

[wesm]

+1

Travis CI build: https://travis-ci.org/wesm/arrow/builds/555965259
Appveyor: https://ci.appveyor.com/project/wesm/arrow/builds/25830311

[pitrou]

I'm not sure I understand the underlying cause either. It seems the module-wide _ExtensionTypesInitializer instance was never destroyed. Would be worth opening a Cython issue for advice, IMHO.

[wesm]

I opened cython/cython#3039

[scoder]

In order to prevent common cases for crashes or other weird behaviour at shutdown, module-level cleanup is not enabled by default. Does pyarrow use it?

https://github.com/cython/cython/blob/3c4e11ddcf5634cf15c786e22ac1835a419a3622/Cython/Compiler/Options.py#L63-L67

The comment in there is not entirely correct for Py3 anymore, where the cleanup is triggered by CPython's module object cleanup. That might not be at system exit but when actually unloading the module for whatever (rare) reason, even before exiting. It's still typically the system exit, though.

[scoder]

Updated the comment here.

[wesm]

As reported on JIRA, the following script provokes a segfault

#! /usr/bin/env python
import pyarrow
import sys
del sys.modules['pyarrow.lib']

For some reason this does not trigger the destruction of the private _ExtensionTypesInitializer object. Not sure why (Antoine may know). Using the atexit module instead seems to do the trick

[wesm]

+1

Travis CI build: https://travis-ci.org/wesm/arrow/builds/555965259
Appveyor: https://ci.appveyor.com/project/wesm/arrow/builds/25830311

[pitrou]

I'm not sure I understand the underlying cause either. It seems the module-wide _ExtensionTypesInitializer instance was never destroyed. Would be worth opening a Cython issue for advice, IMHO.

[wesm]

I opened cython/cython#3039

[scoder]

In order to prevent common cases for crashes or other weird behaviour at shutdown, module-level cleanup is not enabled by default. Does pyarrow use it?

https://github.com/cython/cython/blob/3c4e11ddcf5634cf15c786e22ac1835a419a3622/Cython/Compiler/Options.py#L63-L67

The comment in there is not entirely correct for Py3 anymore, where the cleanup is triggered by CPython's module object cleanup. That might not be at system exit but when actually unloading the module for whatever (rare) reason, even before exiting. It's still typically the system exit, though.

[scoder]

Updated the comment here.

[wesm]

As reported on JIRA, the following script provokes a segfault

#! /usr/bin/env python
import pyarrow
import sys
del sys.modules['pyarrow.lib']

For some reason this does not trigger the destruction of the private _ExtensionTypesInitializer object. Not sure why (Antoine may know). Using the atexit module instead seems to do the trick

[wesm]

+1

Travis CI build: https://travis-ci.org/wesm/arrow/builds/555965259
Appveyor: https://ci.appveyor.com/project/wesm/arrow/builds/25830311

[pitrou]

I'm not sure I understand the underlying cause either. It seems the module-wide _ExtensionTypesInitializer instance was never destroyed. Would be worth opening a Cython issue for advice, IMHO.

[wesm]

I opened cython/cython#3039

[scoder]

In order to prevent common cases for crashes or other weird behaviour at shutdown, module-level cleanup is not enabled by default. Does pyarrow use it?

https://github.com/cython/cython/blob/3c4e11ddcf5634cf15c786e22ac1835a419a3622/Cython/Compiler/Options.py#L63-L67

The comment in there is not entirely correct for Py3 anymore, where the cleanup is triggered by CPython's module object cleanup. That might not be at system exit but when actually unloading the module for whatever (rare) reason, even before exiting. It's still typically the system exit, though.

[scoder]

Updated the comment here.

[wesm]

As reported on JIRA, the following script provokes a segfault

#! /usr/bin/env python
import pyarrow
import sys
del sys.modules['pyarrow.lib']

For some reason this does not trigger the destruction of the private _ExtensionTypesInitializer object. Not sure why (Antoine may know). Using the atexit module instead seems to do the trick

[wesm]

+1

Travis CI build: https://travis-ci.org/wesm/arrow/builds/555965259
Appveyor: https://ci.appveyor.com/project/wesm/arrow/builds/25830311

[pitrou]

I'm not sure I understand the underlying cause either. It seems the module-wide _ExtensionTypesInitializer instance was never destroyed. Would be worth opening a Cython issue for advice, IMHO.

[wesm]

I opened cython/cython#3039

[scoder]

In order to prevent common cases for crashes or other weird behaviour at shutdown, module-level cleanup is not enabled by default. Does pyarrow use it?

https://github.com/cython/cython/blob/3c4e11ddcf5634cf15c786e22ac1835a419a3622/Cython/Compiler/Options.py#L63-L67

The comment in there is not entirely correct for Py3 anymore, where the cleanup is triggered by CPython's module object cleanup. That might not be at system exit but when actually unloading the module for whatever (rare) reason, even before exiting. It's still typically the system exit, though.

[scoder]

Updated the comment here.

[wesm]

As reported on JIRA, the following script provokes a segfault

#! /usr/bin/env python
import pyarrow
import sys
del sys.modules['pyarrow.lib']

For some reason this does not trigger the destruction of the private _ExtensionTypesInitializer object. Not sure why (Antoine may know). Using the atexit module instead seems to do the trick

[wesm]

+1

Travis CI build: https://travis-ci.org/wesm/arrow/builds/555965259
Appveyor: https://ci.appveyor.com/project/wesm/arrow/builds/25830311

[pitrou]

I'm not sure I understand the underlying cause either. It seems the module-wide _ExtensionTypesInitializer instance was never destroyed. Would be worth opening a Cython issue for advice, IMHO.

[wesm]

I opened cython/cython#3039

[scoder]

In order to prevent common cases for crashes or other weird behaviour at shutdown, module-level cleanup is not enabled by default. Does pyarrow use it?

https://github.com/cython/cython/blob/3c4e11ddcf5634cf15c786e22ac1835a419a3622/Cython/Compiler/Options.py#L63-L67

The comment in there is not entirely correct for Py3 anymore, where the cleanup is triggered by CPython's module object cleanup. That might not be at system exit but when actually unloading the module for whatever (rare) reason, even before exiting. It's still typically the system exit, though.

[scoder]

Updated the comment here.

[wesm]

As reported on JIRA, the following script provokes a segfault

#! /usr/bin/env python
import pyarrow
import sys
del sys.modules['pyarrow.lib']

For some reason this does not trigger the destruction of the private _ExtensionTypesInitializer object. Not sure why (Antoine may know). Using the atexit module instead seems to do the trick

[wesm]

+1

Travis CI build: https://travis-ci.org/wesm/arrow/builds/555965259
Appveyor: https://ci.appveyor.com/project/wesm/arrow/builds/25830311

[pitrou]

I'm not sure I understand the underlying cause either. It seems the module-wide _ExtensionTypesInitializer instance was never destroyed. Would be worth opening a Cython issue for advice, IMHO.

[wesm]

I opened cython/cython#3039

[scoder]

In order to prevent common cases for crashes or other weird behaviour at shutdown, module-level cleanup is not enabled by default. Does pyarrow use it?

https://github.com/cython/cython/blob/3c4e11ddcf5634cf15c786e22ac1835a419a3622/Cython/Compiler/Options.py#L63-L67

The comment in there is not entirely correct for Py3 anymore, where the cleanup is triggered by CPython's module object cleanup. That might not be at system exit but when actually unloading the module for whatever (rare) reason, even before exiting. It's still typically the system exit, though.

[scoder]

Updated the comment here.

[wesm]

As reported on JIRA, the following script provokes a segfault

#! /usr/bin/env python
import pyarrow
import sys
del sys.modules['pyarrow.lib']

For some reason this does not trigger the destruction of the private _ExtensionTypesInitializer object. Not sure why (Antoine may know). Using the atexit module instead seems to do the trick

[wesm]

+1

Travis CI build: https://travis-ci.org/wesm/arrow/builds/555965259
Appveyor: https://ci.appveyor.com/project/wesm/arrow/builds/25830311

[pitrou]

I'm not sure I understand the underlying cause either. It seems the module-wide _ExtensionTypesInitializer instance was never destroyed. Would be worth opening a Cython issue for advice, IMHO.

[wesm]

I opened cython/cython#3039

[scoder]

In order to prevent common cases for crashes or other weird behaviour at shutdown, module-level cleanup is not enabled by default. Does pyarrow use it?

https://github.com/cython/cython/blob/3c4e11ddcf5634cf15c786e22ac1835a419a3622/Cython/Compiler/Options.py#L63-L67

The comment in there is not entirely correct for Py3 anymore, where the cleanup is triggered by CPython's module object cleanup. That might not be at system exit but when actually unloading the module for whatever (rare) reason, even before exiting. It's still typically the system exit, though.

[scoder]

Updated the comment here.

[wesm]

As reported on JIRA, the following script provokes a segfault

#! /usr/bin/env python
import pyarrow
import sys
del sys.modules['pyarrow.lib']

For some reason this does not trigger the destruction of the private _ExtensionTypesInitializer object. Not sure why (Antoine may know). Using the atexit module instead seems to do the trick

[wesm]

+1

Travis CI build: https://travis-ci.org/wesm/arrow/builds/555965259
Appveyor: https://ci.appveyor.com/project/wesm/arrow/builds/25830311

[pitrou]

I'm not sure I understand the underlying cause either. It seems the module-wide _ExtensionTypesInitializer instance was never destroyed. Would be worth opening a Cython issue for advice, IMHO.

[wesm]

I opened cython/cython#3039

[scoder]

In order to prevent common cases for crashes or other weird behaviour at shutdown, module-level cleanup is not enabled by default. Does pyarrow use it?

https://github.com/cython/cython/blob/3c4e11ddcf5634cf15c786e22ac1835a419a3622/Cython/Compiler/Options.py#L63-L67

The comment in there is not entirely correct for Py3 anymore, where the cleanup is triggered by CPython's module object cleanup. That might not be at system exit but when actually unloading the module for whatever (rare) reason, even before exiting. It's still typically the system exit, though.

[scoder]

Updated the comment here.