double free in io.TextIOWrapper

[scufre]

BPO	28387
Nosy	@benjaminp, @serhiy-storchaka
PRs	[Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
Files	textiowrapper_crash.py: Proof of concept to demostrate the issue textio.c.patch: Proposed fix textiowrapper_clear-3.5.patch textiowrapper_clear-2.7.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2016-11-03.13:41:50.317>
created_at = <Date 2016-10-07.19:32:57.587>
labels = ['extension-modules', '3.7', 'expert-IO', 'type-crash']
title = 'double free in io.TextIOWrapper'
updated_at = <Date 2017-03-31.16:36:23.657>
user = 'https://bugs.python.org/scufre'

bugs.python.org fields:

activity = <Date 2017-03-31.16:36:23.657>
actor = 'dstufft'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2016-11-03.13:41:50.317>
closer = 'serhiy.storchaka'
components = ['Extension Modules', 'IO']
creation = <Date 2016-10-07.19:32:57.587>
creator = 'scufre'
dependencies = []
files = ['45004', '45005', '45274', '45275']
hgrepos = []
issue_num = 28387
keywords = ['patch']
message_count = 3.0
messages = ['278263', '279717', '279994']
nosy_count = 5.0
nosy_names = ['benjamin.peterson', 'stutzbach', 'python-dev', 'serhiy.storchaka', 'scufre']
pr_nums = ['552']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue28387'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7']

[scufre]

We have found that you can produce a crash when an instance of _io.TextIOWrapper is being deallocated while there's another thread invoking the garbage collector. I've attached a simple script that should reproduce the issue (textiowrapper_crash.py)

Looking at the code of the _io module, we found that on the dealloc method of the TextIOWrapper class, it first tries to invoke the close method, then releases its internal members, after that removes itself from the garbage collector tracking and finally frees deallocates the remaining stuff. What happens, is that while releasing it's internal members, it might end up calling code that releases the interpreter lock (because its doing an operating system call), letting other threads execute. If for example the thread that comes in, invokes the garbage collector, on debug will raise an assert on gcmodule.c on line 351, where I understand it is complaining that it is tracking an object with refcount 0. In a release build, I suppose this goes on and will end up releasing the object (given it has a refcount of 0), and when the interrupted dealloc thread continues will end up freeing again itself which at some point produces a crash.

Attached is a proposed fix for the issue in textio.c.patch, where the it the call to _PyObject_GC_UNTRACK is now done right after the call to the close method and before release its internal members.

As a reference we have been able to reproduce this with Python 2.7.12 on Windows (i386)

[serhiy-storchaka]

Thank you for your report and patch Sebastian.

In Python 3 the solution can be simpler, just move the line "_PyObject_GC_UNTRACK(self);" above the line "textiowrapper_clear(self);". But calling PyObject_ClearWeakRefs() also should be moved up. Otherwise half-destroyed TextIOWrapper instance can be accessed via weak references. Following patches do this (and small refactoring).

[python-dev]

New changeset 91f024fc9b3a by Serhiy Storchaka in branch '2.7':
Issue bpo-28387: Fixed possible crash in _io.TextIOWrapper deallocator when
https://hg.python.org/cpython/rev/91f024fc9b3a

New changeset 89f7386104e2 by Serhiy Storchaka in branch '3.5':
Issue bpo-28387: Fixed possible crash in _io.TextIOWrapper deallocator when
https://hg.python.org/cpython/rev/89f7386104e2

New changeset c4319c0d0131 by Serhiy Storchaka in branch '3.6':
Issue bpo-28387: Fixed possible crash in _io.TextIOWrapper deallocator when
https://hg.python.org/cpython/rev/c4319c0d0131

New changeset 36af3566b67a by Serhiy Storchaka in branch 'default':
Issue bpo-28387: Fixed possible crash in _io.TextIOWrapper deallocator when
https://hg.python.org/cpython/rev/36af3566b67a

[ziegenbalg]

`Nov 30 09:15:44 ob systemd-coredump[1408395]: Process 1407506 (python) of user 1001 dumped core.

Stack trace of thread 1407700:
#0 0x00007f18c511d240 textiowrapper_dealloc (libpython3.9.so.1.0 + 0x1b8240)
#1 0x00007f18c506ce38 meth_dealloc (libpython3.9.so.1.0 + 0x107e38)
#2 0x00007f18c511a42b PyFile_WriteObject (libpython3.9.so.1.0 + 0x1b542b)
#3 0x00007f18c5120f5d builtin_print (libpython3.9.so.1.0 + 0x1bbf5d)
#4 0x00007f18c507909f cfunction_vectorcall_FASTCALL_KEYWORDS (libpython3.9.so.1.0 + 0x11409f)
#5 0x00007f18c5072042 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x10d042)
#6 0x00007f18c50702e9 _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10b2e9)
#7 0x00007f18c507e396 _PyFunction_Vectorcall (libpython3.9.so.1.0 + 0x119396)
#8 0x00007f18c50710c9 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x10c0c9)
#9 0x00007f18c50702e9 _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10b2e9)
#10 0x00007f18c507e396 _PyFunction_Vectorcall (libpython3.9.so.1.0 + 0x119396)
#11 0x00007f18c5071329 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x10c329)
#12 0x00007f18c507e66b function_code_fastcall (libpython3.9.so.1.0 + 0x11966b)
#13 0x00007f18c5071329 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x10c329)
#14 0x00007f18c507e66b function_code_fastcall (libpython3.9.so.1.0 + 0x11966b)
#15 0x00007f18c5071329 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x10c329)
#16 0x00007f18c50702e9 _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10b2e9)
#17 0x00007f18c507e396 _PyFunction_Vectorcall (libpython3.9.so.1.0 + 0x119396)
#18 0x00007f18c5072042 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x10d042)
#19 0x00007f18c506fd31 _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10ad31)
#20 0x00007f18c507e396 _PyFunction_Vectorcall (libpython3.9.so.1.0 + 0x119396)
#21 0x00007f18c5072042 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x10d042)
#22 0x00007f18c506fd31 _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10ad31)
#23 0x00007f18c507e396 _PyFunction_Vectorcall (libpython3.9.so.1.0 + 0x119396)
#24 0x00007f18c5072042 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x10d042)
#25 0x00007f18c507e66b function_code_fastcall (libpython3.9.so.1.0 + 0x11966b)
#26 0x00007f18c5073d73 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x10ed73)
#27 0x00007f18c506fd31 _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10ad31)
#28 0x00007f18c507e396 _PyFunction_Vectorcall (libpython3.9.so.1.0 + 0x119396)
#29 0x00007f18c5073d73 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x10ed73)
#30 0x00007f18c507e66b function_code_fastcall (libpython3.9.so.1.0 + 0x11966b)
#31 0x00007f18c5071329 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x10c329)
#32 0x00007f18c507e66b function_code_fastcall (libpython3.9.so.1.0 + 0x11966b)
#33 0x00007f18c5073d73 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x10ed73)
#34 0x00007f18c507e66b function_code_fastcall (libpython3.9.so.1.0 + 0x11966b)
#35 0x00007f18c5071329 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x10c329)
#36 0x00007f18c507e66b function_code_fastcall (libpython3.9.so.1.0 + 0x11966b)
#37 0x00007f18c5071329 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x10c329)
#38 0x00007f18c507e66b function_code_fastcall (libpython3.9.so.1.0 + 0x11966b)
#39 0x00007f18c5087510 method_vectorcall (libpython3.9.so.1.0 + 0x122510)
#40 0x00007f18c513849a t_bootstrap (libpython3.9.so.1.0 + 0x1d349a)
#41 0x00007f18c51382b8 pythread_wrapper (libpython3.9.so.1.0 + 0x1d32b8)
#42 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#43 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407516:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407605:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f16e1ccf0cb blas_thread_server (libopenblasp-r0-085ca80a.3.9.so + 0x3150cb)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407509:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407544:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407536:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407550:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407548:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407551:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407524:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407530:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407552:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407506:
#0 0x00007f18c4d8a0d4 do_futex_wait.constprop.0 (libpthread.so.0 + 0x120d4)
#1 0x00007f18c4d8a1d8 __new_sem_wait_slow.constprop.0 (libpthread.so.0 + 0x121d8)
#2 0x00007f18c505ee16 PyThread_acquire_lock_timed (libpython3.9.so.1.0 + 0xf9e16)
#3 0x00007f18c50f4021 acquire_timed (libpython3.9.so.1.0 + 0x18f021)
#4 0x00007f18c50f3e2f lock_PyThread_acquire_lock (libpython3.9.so.1.0 + 0x18ee2f)
#5 0x00007f18c507f43d method_vectorcall_VARARGS_KEYWORDS (libpython3.9.so.1.0 + 0x11a43d)
#6 0x00007f18c5071329 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x10c329)
#7 0x00007f18c506fd31 _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10ad31)
#8 0x00007f18c507e396 _PyFunction_Vectorcall (libpython3.9.so.1.0 + 0x119396)
#9 0x00007f18c5071329 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x10c329)
#10 0x00007f18c506fd31 _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10ad31)
#11 0x00007f18c507e396 _PyFunction_Vectorcall (libpython3.9.so.1.0 + 0x119396)
#12 0x00007f18c5071329 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x10c329)
#13 0x00007f18c50702e9 _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10b2e9)
#14 0x00007f18c507e396 _PyFunction_Vectorcall (libpython3.9.so.1.0 + 0x119396)
#15 0x00007f18c5087314 method_vectorcall (libpython3.9.so.1.0 + 0x122314)
#16 0x00007f18c5072042 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x10d042)
#17 0x00007f18c50702e9 _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10b2e9)
#18 0x00007f18c507e396 _PyFunction_Vectorcall (libpython3.9.so.1.0 + 0x119396)
#19 0x00007f18c5072042 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x10d042)
#20 0x00007f18c506fd31 _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10ad31)
#21 0x00007f18c507e396 _PyFunction_Vectorcall (libpython3.9.so.1.0 + 0x119396)
#22 0x00007f18c5072042 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x10d042)
#23 0x00007f18c506fd31 _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10ad31)
#24 0x00007f18c50ee9b5 _PyEval_EvalCodeWithName (libpython3.9.so.1.0 + 0x1899b5)
#25 0x00007f18c50ee94d PyEval_EvalCodeEx (libpython3.9.so.1.0 + 0x18994d)
#26 0x00007f18c50ee8ff PyEval_EvalCode (libpython3.9.so.1.0 + 0x1898ff)
#27 0x00007f18c511b68d run_eval_code_obj (libpython3.9.so.1.0 + 0x1b668d)
#28 0x00007f18c5117736 run_mod (libpython3.9.so.1.0 + 0x1b2736)
#29 0x00007f18c4fea58c pyrun_file.cold (libpython3.9.so.1.0 + 0x8558c)
#30 0x00007f18c5111a01 PyRun_SimpleFileExFlags (libpython3.9.so.1.0 + 0x1aca01)
#31 0x00007f18c510ebc6 Py_RunMain (libpython3.9.so.1.0 + 0x1a9bc6)
#32 0x00007f18c50e155d Py_BytesMain (libpython3.9.so.1.0 + 0x17c55d)
#33 0x00007f18c4dc21e2 __libc_start_main (libc.so.6 + 0x281e2)
#34 0x000055ce23d5209e _start (python3.9 + 0x109e)

Stack trace of thread 1407512:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407507:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407514:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407511:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407519:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407510:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407526:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407508:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407533:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407518:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407515:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407539:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407513:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407521:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407532:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407531:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407517:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407525:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407522:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407523:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407527:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407535:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407534:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407537:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407545:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407540:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407592:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f16e1ccf0cb blas_thread_server (libopenblasp-r0-085ca80a.3.9.so + 0x3150cb)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407529:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407520:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407541:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407604:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f16e1ccf0cb blas_thread_server (libopenblasp-r0-085ca80a.3.9.so + 0x3150cb)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407538:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407528:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407549:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407542:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407547:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407612:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f16e1ccf0cb blas_thread_server (libopenblasp-r0-085ca80a.3.9.so + 0x3150cb)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407623:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f16e1ccf0cb blas_thread_server (libopenblasp-r0-085ca80a.3.9.so + 0x3150cb)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407629:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f16e1ccf0cb blas_thread_server (libopenblasp-r0-085ca80a.3.9.so + 0x3150cb)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407553:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f18b447cf8b blas_thread_server (libopenblasp-r0-09e95953.3.13.so + 0x323f8b)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407589:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f16e1ccf0cb blas_thread_server (libopenblasp-r0-085ca80a.3.9.so + 0x3150cb)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407614:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f16e1ccf0cb blas_thread_server (libopenblasp-r0-085ca80a.3.9.so + 0x3150cb)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407596:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f16e1ccf0cb blas_thread_server (libopenblasp-r0-085ca80a.3.9.so + 0x3150cb)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407630:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f16e1ccf0cb blas_thread_server (libopenblasp-r0-085ca80a.3.9.so + 0x3150cb)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407631:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f16e1ccf0cb blas_thread_server (libopenblasp-r0-085ca80a.3.9.so + 0x3150cb)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407602:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f16e1ccf0cb blas_thread_server (libopenblasp-r0-085ca80a.3.9.so + 0x3150cb)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407615:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f16e1ccf0cb blas_thread_server (libopenblasp-r0-085ca80a.3.9.so + 0x3150cb)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407586:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f16e1ccf0cb blas_thread_server (libopenblasp-r0-085ca80a.3.9.so + 0x3150cb)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407590:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f16e1ccf0cb blas_thread_server (libopenblasp-r0-085ca80a.3.9.so + 0x3150cb)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407613:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f16e1ccf0cb blas_thread_server (libopenblasp-r0-085ca80a.3.9.so + 0x3150cb)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)

Stack trace of thread 1407591:
#0 0x00007f18c4d87692 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf692)
#1 0x00007f16e1ccf0cb blas_thread_server (libopenblasp-r0-085ca80a.3.9.so + 0x3150cb)
#2 0x00007f18c4d813f9 start_thread (libpthread.so.0 + 0x93f9)
#3 0x00007f18c4e9b4c3 __clone (libc.so.6 + 0x1014c3)
`

Appears I may have just encountered this bug again? Really heavy print I/O with 12 threads. Really hard to debug, core often doesn't get dumped. I'm lucky I got this core dump at all. It randomly succeeded in dumping the core.

Runing on python 3.9.9