[Python] Error when executing the command to build pyarrow

[llama90]

Describe the bug, including details regarding any error messages, version, and platform.

I encounter an error when executing the command(python setup.py build_ext --inplace) to build pyarrow.

Based on examining the git commits, it appears to be related as follows.

I use M1 mackbook.

• /Users/lama/workspace/arrow-latest/python/build/temp.macosx-11.0-arm64-cpython-310/lib.cpp:75745:101: error: no member named 'MakeTensor' in 'arrow::extension::FixedShapeTensorType'
  • GH-37484: [Python] Add a FixedSizeTensorScalar class #37533
• /Users/lama/workspace/arrow-latest/python/build/temp.macosx-11.0-arm64-cpython-310/lib.cpp:118724:67: error: no member named 'ExportChunkedArray' in namespace 'arrow'
• /Users/lama/workspace/arrow-latest/python/build/temp.macosx-11.0-arm64-cpython-310/lib.cpp:118897:98: error: no member named 'ImportChunkedArray' in namespace 'arrow'
  • GH-39984: [Python] Add ChunkedArray import/export to/from C  #39985

error message

$ python setup.py build_ext --inplace 
/Users/lama/workspace/arrow-latest/python/setup.py:34: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
 import pkg_resources
/Users/lama/mambaforge/envs/pyarrow-dev/lib/python3.10/site-packages/setuptools/__init__.py:80: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
!!

 ********************************************************************************
 Requirements should be satisfied by a PEP 517 installer.
 If you are using pip, you can try `pip install --use-pep517`.
 ********************************************************************************

!!
 dist.fetch_build_eggs(dist.setup_requires)
running build_ext
-- Running cmake for PyArrow
cmake -DCMAKE_INSTALL_PREFIX=/Users/lama/workspace/arrow-latest/python/pyarrow -DPYTHON_EXECUTABLE=/Users/lama/mambaforge/envs/pyarrow-dev/bin/python -DPython3_EXECUTABLE=/Users/lama/mambaforge/envs/pyarrow-dev/bin/python -DPYARROW_CXXFLAGS= -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_SUBSTRAIT=off -DPYARROW_BUILD_FLIGHT=off -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_ACERO=off -DPYARROW_BUILD_DATASET=off -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=off -DPYARROW_BUILD_PARQUET_ENCRYPTION=off -DPYARROW_BUILD_GCS=off -DPYARROW_BUILD_S3=off -DPYARROW_BUILD_HDFS=off -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_CYTHON_CPP=off -DPYARROW_GENERATE_COVERAGE=off -DCMAKE_BUILD_TYPE=release /Users/lama/workspace/arrow-latest/python
CMake Warning at /Users/lama/mambaforge/envs/pyarrow-dev/share/cmake-3.27/Modules/Platform/Darwin-Initialize.cmake:306 (message):
 Ignoring CMAKE_OSX_SYSROOT value:

 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk

 because the directory does not exist.
Call Stack (most recent call first):
 /Users/lama/mambaforge/envs/pyarrow-dev/share/cmake-3.27/Modules/CMakeSystemSpecificInitialize.cmake:34 (include)
 CMakeLists.txt:22 (project)


-- Using ccache: /Users/lama/anaconda3/envs/pyarrow-dev/bin/ccache
-- System processor: arm64
-- Arrow build warning level: PRODUCTION
-- Build Type: RELEASE
-- CMAKE_C_FLAGS: -Wall -Wno-unknown-warning-option -Wno-pass-failed -march=armv8-a -ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem /Users/lama/anaconda3/envs/pyarrow-dev/include -Qunused-arguments -fcolor-diagnostics -fno-omit-frame-pointer -Wno-unused-variable -Wno-maybe-uninitialized -Wno-parentheses-equality -Wno-constant-logical-operand -Wno-missing-declarations -Wno-sometimes-uninitialized -Wno-return-type-c-linkage
-- CMAKE_CXX_FLAGS: -fno-aligned-new -Wall -Wno-unknown-warning-option -Wno-pass-failed -march=armv8-a -ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem /Users/lama/anaconda3/envs/pyarrow-dev/include -Qunused-arguments -fcolor-diagnostics -fno-omit-frame-pointer -Wno-unused-variable -Wno-maybe-uninitialized -Wno-parentheses-equality -Wno-constant-logical-operand -Wno-missing-declarations -Wno-sometimes-uninitialized -Wno-return-type-c-linkage
-- Generator: Unix Makefiles
-- Build output directory: /Users/lama/workspace/arrow-latest/python/build/temp.macosx-11.0-arm64-cpython-310/release
CMake Warning at /Users/lama/anaconda3/envs/pyarrow-dev/lib/cmake/Arrow/Findlz4Alt.cmake:29 (find_package):
 By not providing "Findlz4.cmake" in CMAKE_MODULE_PATH this project has
 asked CMake to find a package configuration file provided by "lz4", but
 CMake did not find one.

 Could not find a package configuration file provided by "lz4" with any of
 the following names:

 lz4Config.cmake
 lz4-config.cmake

 Add the installation prefix of "lz4" to CMAKE_PREFIX_PATH or set "lz4_DIR"
 to a directory containing one of the above files. If "lz4" provides a
 separate development package or SDK, be sure it has been installed.
Call Stack (most recent call first):
 /Users/lama/mambaforge/envs/pyarrow-dev/share/cmake-3.27/Modules/CMakeFindDependencyMacro.cmake:76 (find_package)
 /Users/lama/anaconda3/envs/pyarrow-dev/lib/cmake/Arrow/ArrowConfig.cmake:99 (find_dependency)
 /Users/lama/anaconda3/envs/pyarrow-dev/lib/cmake/Arrow/ArrowConfig.cmake:122 (arrow_find_dependencies)
 CMakeLists.txt:268 (find_package)


-- Arrow version: 16.0.0
-- Found the Arrow shared library: /Users/lama/anaconda3/envs/pyarrow-dev/lib/libarrow.1600.0.0.dylib
-- Found the Arrow import library: ARROW_IMPORT_LIB-NOTFOUND
-- Found the Arrow static library: /Users/lama/anaconda3/envs/pyarrow-dev/lib/libarrow.a
-- Parquet Encryption is NOT Enabled
-- Configuring done (1.4s)
-- Generating done (0.0s)
-- Build files have been written to: /Users/lama/workspace/arrow-latest/python/build/temp.macosx-11.0-arm64-cpython-310
-- Finished cmake for PyArrow
-- Running cmake --build for PyArrow
cmake --build . --config release --
[ 1%] Compiling Cython CXX source for lib...
[ 1%] Built target lib_pyx
[ 3%] Built target cython_api_headers
[ 45%] Built target arrow_python
[ 47%] Building CXX object CMakeFiles/lib.dir/lib.cpp.o
/Users/lama/workspace/arrow-latest/python/build/temp.macosx-11.0-arm64-cpython-310/lib.cpp:75751:101: error: no member named 'MakeTensor' in 'arrow::extension::FixedShapeTensorType'
 __pyx_t_1 = arrow::py::GetResultValue<std::shared_ptr< arrow::Tensor> >(__pyx_v_c_type->MakeTensor(__pyx_v_scalar)); if (unlikely(__Pyx_ErrOccurredWithGIL())) __PYX_ERR(4, 1084, __pyx_L4_error)
 ~~~~~~~~~~~~~~ ^
/Users/lama/workspace/arrow-latest/python/build/temp.macosx-11.0-arm64-cpython-310/lib.cpp:118724:67: error: no member named 'ExportChunkedArray' in namespace 'arrow'
 __pyx_t_6 = __pyx_f_7pyarrow_3lib_check_status(arrow::ExportChunkedArray(__pyx_v_self->sp_chunked_array, __pyx_v_c_stream)); if (unlikely(__pyx_t_6 == ((int)-1))) __PYX_ERR(15, 1357, __pyx_L6_error)
 ~~~~~~~^
/Users/lama/workspace/arrow-latest/python/build/temp.macosx-11.0-arm64-cpython-310/lib.cpp:118897:98: error: no member named 'ImportChunkedArray' in namespace 'arrow'
 __pyx_t_2 = arrow::py::GetResultValue<std::shared_ptr< arrow::ChunkedArray> >(arrow::ImportChunkedArray(__pyx_v_c_stream)); if (unlikely(__Pyx_ErrOccurredWithGIL())) __PYX_ERR(15, 1385, __pyx_L4_error)
 ~~~~~~~^
3 errors generated.
make[2]: *** [CMakeFiles/lib.dir/build.make:93: CMakeFiles/lib.dir/lib.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:161: CMakeFiles/lib.dir/all] Error 2
make: *** [Makefile:136: all] Error 2
error: command '/Users/lama/mambaforge/envs/pyarrow-dev/bin/cmake' failed with exit code 2

Component(s)

Python

[rok]

Could you try deleting your cpp arrow and pyarrow binaries and build them again?

[llama90]

Thank you. I am trying as you suggested, but it seems like it might take some time as I am encountering issues during the attempt.

[rok]

I usually find it helpful to follow steps here.

[llama90]

@rok

I was able to resolve it thanks to you. After cleaning the entire build environment, it worked. Thank you :)