[Python] Apache Arrow minimal cpp build segfaults with pyarrow libs

[asfimport]

Refer to kratsg/awkward-arrow-cmake-pybind11#6 (comment) and code-base available here: https://github.com/kratsg/awkward-arrow-cmake-pybind11/tree/0e196983b2452c4c3dbbde40984db7031397d0a7

The minimal build example provided by arrow (https://github.com/apache/arrow/blob/master/cpp/examples/minimal_build/example.cc ) works fine using the libs shipped with yum on CentOS7. Using the libs shipped by pip install pyarrow instead provides a segfault at runtime. Here is the gdb dump with debugging flags included

[root@dab7bb04d93f examples]# gdb ./../build/example_arrow 
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /Users/kratsg/awkward-arrow-cmake-pybind11/build/example_arrow...done.
(gdb) run
Starting program: /Users/kratsg/awkward-arrow-cmake-pybind11/examples/./../build/example_arrow 
warning: Error disabling address space randomization: Operation not permitted
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7f9c58dff700 (LWP 1643)]
* Reading CSV file 'test.csv' into table

Program received signal SIGSEGV, Segmentation fault.
__destroy<std::basic_string<char>*> (__last=<optimized out>, __first=0x3200000000) at /usr/include/c++/4.8.2/bits/stl_construct.h:103
103		 std::_Destroy(std::__addressof(*__first));
Missing separate debuginfos, use: debuginfo-install glibc-2.17-324.el7_9.x86_64 libgcc-4.8.5-44.el7.x86_64 libstdc++-4.8.5-44.el7.x86_64
(gdb) bt
#0 __destroy<std::basic_string<char>*> (__last=<optimized out>, __first=0x3200000000) at /usr/include/c++/4.8.2/bits/stl_construct.h:103
#1 _Destroy<std::basic_string<char>*> (__last=<optimized out>, __first=<optimized out>) at /usr/include/c++/4.8.2/bits/stl_construct.h:126
#2 _Destroy<std::basic_string<char>*, std::basic_string<char> > (__last=0x0, __first=<optimized out>) at /usr/include/c++/4.8.2/bits/stl_construct.h:151
#3 ~vector (this=0x7ffff43602a8, __in_chrg=<optimized out>) at /usr/include/c++/4.8.2/bits/stl_vector.h:415
#4 arrow::csv::ConvertOptions::~ConvertOptions (this=0x7ffff4360220, __in_chrg=<optimized out>)
 at /usr/local/venv/lib/python3.8/site-packages/pyarrow/include/arrow/csv/options.h:64
#5 0x000000000040171a in (anonymous namespace)::RunMain (argv=<optimized out>, argc=<optimized out>, this=<optimized out>, this=<optimized out>)
 at /Users/kratsg/awkward-arrow-cmake-pybind11/src/example/arrow.cpp:39
#6 0x000000000040135f in main (argc=<optimized out>, argv=<optimized out>) at /Users/kratsg/awkward-arrow-cmake-pybind11/src/example/arrow.cpp:66 

Below is the full CMake log + build log

# ./build.sh 
-- The CXX compiler identification is GNU 4.8.5
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- pybind11 v2.6.2 
-- Found PythonInterp: /usr/local/bin/python3.8 (found version "3.8.10") 
-- Found PythonLibs: /usr/local/lib/libpython3.8.so
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- PyArrow includes : /usr/local/venv/lib/python3.8/site-packages/pyarrow/include
-- PyArrow libraries : arrow;arrow_python
-- PyArrow library dirs : /usr/local/venv/lib/python3.8/site-packages/pyarrow
-- Awkward includes : /usr/local/venv/lib/python3.8/site-packages/awkward/include
-- Awkward libraries: /usr/local/venv/lib/python3.8/site-packages/awkward
-- Libraries: /usr/local/venv/lib/python3.8/site-packages/awkward/libawkward-cpu-kernels.so /usr/local/venv/lib/python3.8/site-packages/awkward/libawkward.so /usr/lib64/libdl.so /usr/local/venv/lib/python3.8/site-packages/pyarrow/libarrow.so /usr/local/venv/lib/python3.8/site-packages/pyarrow/libarrow_python.so
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/kratsg/awkward-arrow-cmake-pybind11/build
-- pybind11 v2.6.2 
-- PyArrow includes : /usr/local/venv/lib/python3.8/site-packages/pyarrow/include
-- PyArrow libraries : arrow;arrow_python
-- PyArrow library dirs : /usr/local/venv/lib/python3.8/site-packages/pyarrow
-- Awkward includes : /usr/local/venv/lib/python3.8/site-packages/awkward/include
-- Awkward libraries: /usr/local/venv/lib/python3.8/site-packages/awkward
-- Libraries: /usr/local/venv/lib/python3.8/site-packages/awkward/libawkward-cpu-kernels.so /usr/local/venv/lib/python3.8/site-packages/awkward/libawkward.so /usr/lib64/libdl.so /usr/local/venv/lib/python3.8/site-packages/pyarrow/libarrow.so /usr/local/venv/lib/python3.8/site-packages/pyarrow/libarrow_python.so
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/kratsg/awkward-arrow-cmake-pybind11/build
-- Cache values
CMAKE_BUILD_TYPE:STRING=RelWithDebInfo
CMAKE_INSTALL_PREFIX:PATH=/usr/local
CPU-KERNELS:FILEPATH=/usr/local/venv/lib/python3.8/site-packages/awkward/libawkward-cpu-kernels.so
LIBARROW:FILEPATH=/usr/local/venv/lib/python3.8/site-packages/pyarrow/libarrow.so
LIBARROW_PYTHON:FILEPATH=/usr/local/venv/lib/python3.8/site-packages/pyarrow/libarrow_python.so
LIBAWKWARD:FILEPATH=/usr/local/venv/lib/python3.8/site-packages/awkward/libawkward.so
LIBDL:FILEPATH=/usr/lib64/libdl.so
PYBIND11_FINDPYTHON:BOOL=OFF
PYBIND11_INSTALL:BOOL=OFF
PYBIND11_NOPYTHON:BOOL=OFF
PYBIND11_PYTHON_VERSION:STRING=
PYBIND11_TEST:BOOL=OFF

###################################################################
######################### BUILD LOG BELOW #########################
###################################################################

/usr/local/bin/cmake -S/Users/kratsg/awkward-arrow-cmake-pybind11/src -B/Users/kratsg/awkward-arrow-cmake-pybind11/build --check-build-system CMakeFiles/Makefile.cmake 0
/usr/local/bin/cmake -E cmake_progress_start /Users/kratsg/awkward-arrow-cmake-pybind11/build/CMakeFiles /Users/kratsg/awkward-arrow-cmake-pybind11/build//CMakeFiles/progress.marks
/usr/bin/gmake -f CMakeFiles/Makefile2 all
gmake[1]: Entering directory `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
/usr/bin/gmake -f CMakeFiles/babel.dir/build.make CMakeFiles/babel.dir/depend
/usr/bin/gmake -f CMakeFiles/example_arrow.dir/build.make CMakeFiles/example_arrow.dir/depend
/usr/bin/gmake -f CMakeFiles/example.dir/build.make CMakeFiles/example.dir/depend
gmake[2]: Entering directory `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
cd /Users/kratsg/awkward-arrow-cmake-pybind11/build && /usr/local/bin/cmake -E cmake_depends "Unix Makefiles" /Users/kratsg/awkward-arrow-cmake-pybind11/src /Users/kratsg/awkward-arrow-cmake-pybind11/src /Users/kratsg/awkward-arrow-cmake-pybind11/build /Users/kratsg/awkward-arrow-cmake-pybind11/build /Users/kratsg/awkward-arrow-cmake-pybind11/build/CMakeFiles/babel.dir/DependInfo.cmake --color=
gmake[2]: Entering directory `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
cd /Users/kratsg/awkward-arrow-cmake-pybind11/build && /usr/local/bin/cmake -E cmake_depends "Unix Makefiles" /Users/kratsg/awkward-arrow-cmake-pybind11/src /Users/kratsg/awkward-arrow-cmake-pybind11/src /Users/kratsg/awkward-arrow-cmake-pybind11/build /Users/kratsg/awkward-arrow-cmake-pybind11/build /Users/kratsg/awkward-arrow-cmake-pybind11/build/CMakeFiles/example_arrow.dir/DependInfo.cmake --color=
gmake[2]: Entering directory `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
cd /Users/kratsg/awkward-arrow-cmake-pybind11/build && /usr/local/bin/cmake -E cmake_depends "Unix Makefiles" /Users/kratsg/awkward-arrow-cmake-pybind11/src /Users/kratsg/awkward-arrow-cmake-pybind11/src /Users/kratsg/awkward-arrow-cmake-pybind11/build /Users/kratsg/awkward-arrow-cmake-pybind11/build /Users/kratsg/awkward-arrow-cmake-pybind11/build/CMakeFiles/example.dir/DependInfo.cmake --color=
gmake[2]: Leaving directory `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
/usr/bin/gmake -f CMakeFiles/example.dir/build.make CMakeFiles/example.dir/build
gmake[2]: Entering directory `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
gmake[2]: Leaving directory `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
/usr/bin/gmake -f CMakeFiles/babel.dir/build.make CMakeFiles/babel.dir/build
gmake[2]: Leaving directory `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
/usr/bin/gmake -f CMakeFiles/example_arrow.dir/build.make CMakeFiles/example_arrow.dir/build
gmake[2]: Entering directory `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
gmake[2]: Entering directory `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
[ 14%] Building CXX object CMakeFiles/example.dir/example/python.cpp.o
/usr/bin/c++ -Dexample_EXPORTS -isystem /Users/kratsg/awkward-arrow-cmake-pybind11/src/pybind11/include -isystem /usr/local/venv/lib/python3.8/site-packages/awkward/include -O2 -g -DNDEBUG -fPIC -fvisibility=hidden -std=gnu++11 -MD -MT CMakeFiles/example.dir/example/python.cpp.o -MF CMakeFiles/example.dir/example/python.cpp.o.d -o CMakeFiles/example.dir/example/python.cpp.o -c /Users/kratsg/awkward-arrow-cmake-pybind11/src/example/python.cpp
[ 28%] Building CXX object CMakeFiles/babel.dir/babel/python.cpp.o
/usr/bin/c++ -Dbabel_EXPORTS -isystem /Users/kratsg/awkward-arrow-cmake-pybind11/src/pybind11/include -isystem /usr/local/venv/lib/python3.8/site-packages/pyarrow/include -isystem /usr/local/venv/lib/python3.8/site-packages/awkward/include -O2 -g -DNDEBUG -fPIC -fvisibility=hidden -std=gnu++11 -MD -MT CMakeFiles/babel.dir/babel/python.cpp.o -MF CMakeFiles/babel.dir/babel/python.cpp.o.d -o CMakeFiles/babel.dir/babel/python.cpp.o -c /Users/kratsg/awkward-arrow-cmake-pybind11/src/babel/python.cpp
[ 42%] Building CXX object CMakeFiles/example_arrow.dir/example/arrow.cpp.o
/usr/bin/c++ -isystem /usr/local/venv/lib/python3.8/site-packages/pyarrow/include -O2 -g -DNDEBUG -std=gnu++11 -MD -MT CMakeFiles/example_arrow.dir/example/arrow.cpp.o -MF CMakeFiles/example_arrow.dir/example/arrow.cpp.o.d -o CMakeFiles/example_arrow.dir/example/arrow.cpp.o -c /Users/kratsg/awkward-arrow-cmake-pybind11/src/example/arrow.cpp
[ 57%] Linking CXX executable example_arrow
/usr/local/bin/cmake -E cmake_link_script CMakeFiles/example_arrow.dir/link.txt --verbose=1
/usr/bin/c++ -O2 -g -DNDEBUG CMakeFiles/example_arrow.dir/example/arrow.cpp.o -o example_arrow -Wl,-rpath,/usr/local/venv/lib/python3.8/site-packages/pyarrow /usr/local/venv/lib/python3.8/site-packages/pyarrow/libarrow.so 
gmake[2]: Leaving directory `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
[ 57%] Built target example_arrow
[ 71%] Building CXX object CMakeFiles/example.dir/example/minimal.cpp.o
/usr/bin/c++ -Dexample_EXPORTS -isystem /Users/kratsg/awkward-arrow-cmake-pybind11/src/pybind11/include -isystem /usr/local/venv/lib/python3.8/site-packages/awkward/include -O2 -g -DNDEBUG -fPIC -fvisibility=hidden -std=gnu++11 -MD -MT CMakeFiles/example.dir/example/minimal.cpp.o -MF CMakeFiles/example.dir/example/minimal.cpp.o.d -o CMakeFiles/example.dir/example/minimal.cpp.o -c /Users/kratsg/awkward-arrow-cmake-pybind11/src/example/minimal.cpp
[ 85%] Linking CXX shared module example.cpython-38-x86_64-linux-gnu.so
/usr/local/bin/cmake -E cmake_link_script CMakeFiles/example.dir/link.txt --verbose=1
/usr/bin/c++ -fPIC -O2 -g -DNDEBUG -shared -o example.cpython-38-x86_64-linux-gnu.so CMakeFiles/example.dir/example/python.cpp.o CMakeFiles/example.dir/example/minimal.cpp.o -Wl,-rpath,/usr/local/venv/lib/python3.8/site-packages/awkward /usr/local/venv/lib/python3.8/site-packages/awkward/libawkward-cpu-kernels.so /usr/local/venv/lib/python3.8/site-packages/awkward/libawkward.so /usr/lib64/libdl.so 
[100%] Linking CXX shared module babel.cpython-38-x86_64-linux-gnu.so
/usr/local/bin/cmake -E cmake_link_script CMakeFiles/babel.dir/link.txt --verbose=1
/usr/bin/c++ -fPIC -O2 -g -DNDEBUG -shared -o babel.cpython-38-x86_64-linux-gnu.so CMakeFiles/babel.dir/babel/python.cpp.o -Wl,-rpath,/usr/local/venv/lib/python3.8/site-packages/pyarrow:/usr/local/venv/lib/python3.8/site-packages/awkward /usr/local/venv/lib/python3.8/site-packages/pyarrow/libarrow.so /usr/local/venv/lib/python3.8/site-packages/pyarrow/libarrow_python.so /usr/local/venv/lib/python3.8/site-packages/awkward/libawkward-cpu-kernels.so /usr/local/venv/lib/python3.8/site-packages/awkward/libawkward.so /usr/lib64/libdl.so 
gmake[2]: Leaving directory `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
[100%] Built target example
gmake[2]: Leaving directory `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
[100%] Built target babel
gmake[1]: Leaving directory `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
/usr/local/bin/cmake -E cmake_progress_start /Users/kratsg/awkward-arrow-cmake-pybind11/build/CMakeFiles 0 

One can reproduce this by cloning the corresponding repository recursively, then building the docker image, compiling the code, and then running it

git clone --recursive git@github.com:kratsg/awkward-arrow-cmake-pybind11.git
make build-image-debug
make debug # launches docker image
> ./build.sh
> ./run_example.sh

where > denotes commands run inside the docker image.

Environment: See dockerfile: https://github.com/kratsg/awkward-arrow-cmake-pybind11/blob/0e196983b2452c4c3dbbde40984db7031397d0a7/Dockerfile
Reporter: Giordon Stark / @kratsg
Assignee: Giordon Stark / @kratsg

Externally tracked issue: kratsg/awkward-arrow-cmake-pybind11#6

PRs and other links:

• GitHub Pull Request #10535

_{Note: This issue was originally created as ARROW-13085. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
This is probably a classic case of C++ toolchain incompatibility.
Our manylinux wheels use the toolchain provided by the upstream manylinux build images (see https://quay.io/organization/pypa ). This is probably devtoolset-8 or devtoolset-9, depending on which manylinux wheel version (2010 or 2014) you used.

[asfimport]

Giordon Stark / @kratsg:
@pitrou - thanks for the nice hint. Switching to devtoolset-8 does fix the segfault! (Maybe the docs need to have some hint about making sure a compatible toolchain is used if one is building extensions using the python libs?)

 

# ./run_example.sh 
/Users/kratsg/awkward-arrow-cmake-pybind11/examples /Users/kratsg/awkward-arrow-cmake-pybind11
* Reading CSV file 'test.csv' into table
* Read table:
Integers: int64
Strings: string
Timestamps: timestamp[s]
----
Integers:
 [
 [
 1,
 2
 ]
 ]
Strings:
 [
 [
 "Some",
 "data"
 ]
 ]
Timestamps:
 [
 [
 2018-11-13 17:11:10,
 null
 ]
 ]
* Writing table into Arrow IPC file 'test.arrow'
 

where the changes I made to the Dockerfile are

diff --git a/Dockerfile b/Dockerfile
index 1fc311e..87b9289 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -7,6 +7,11 @@ WORKDIR /
 SHELL [ "/bin/bash", "-c" ]
 
 COPY build-requirements.txt /tmp/build-requirements.txt
-RUN python -m pip --no-cache-dir install --upgrade pip setuptools wheel && \
+
+RUN yum update -y -q && \
+ yum install -y -q centos-release-scl && \
+ yum install -y -q devtoolset-8 && \
+ echo "source scl_source enable devtoolset-8" >> ~/.bash_profile && \
+ python -m pip --no-cache-dir install --upgrade pip setuptools wheel && \
 python -m pip --no-cache-dir install --requirement /tmp/build-requirements.txt && \
 python -m pip list 

which is not so bad. This certainly was not obvious at all to me 😀

[asfimport]

Antoine Pitrou / @pitrou:
@kratsg Do you want to open a PR for improving the docs? Or at least suggest where this could be added?

[asfimport]

Giordon Stark / @kratsg:
Sure, let me fork and then create a PR and link it here.

[asfimport]

Giordon Stark / @kratsg:
I've opened a PR here #10535

[asfimport]

Antoine Pitrou / @pitrou:
Issue resolved by pull request 10535
#10535