GH-25025: [C++] Move non core compute kernels into separate shared library

[raulcd]

Rationale for this change

Arrow is quite a heavy dependency and some users don't need all the tools we provide bundled with libarrow. Moving Arrow Compute to its own shared library allows users installations that better suit their needs having smaller memory footprint if necessary.
It might also help some users adding new kernels into an existing Arrow without recompiling it.

What changes are included in this PR?

• Move all the Arrow Compute kernel functions to a new ArrowCompute shared library (libarrow_compute.so).
• Create a new API to Initialize arrow compute registering the existing Kernels into the FunctionRegistry
• Update Python/R/CGLib bindings to automatically register the Compute kernels transparently to the user.
• Update Linux Packaging to provide the new arrow-compute library.
• Update documentation with new requirements to call arrow::compute::Initialize()

Are these changes tested?

Yes on all CI jobs.

Are there any user-facing changes?

Yes. The Arrow compute functions will be provided as a different library. Any user using Arrow Compute from C++ directly will require a call to arrow::compute::Initialize() in order for the functions and kernels to be registered

This PR includes breaking changes to public APIs.

• GitHub Issue: [C++] Split non-cast compute kernels into a separate shared library #25025

[raulcd]

@github-actions crossbow submit -g cpp

[raulcd]

It seems I am down to one job failure to the Windows R release job (CGLib Ruby is failing due to an existing issue):
I am unsure why I am getting missing symbols on dataset and acero when building statically. I've tried several things, around our *.pc.in and configure.win files to try and fix linking to the new arrow_compute but I am unsure where the problem is exactly coming.
@kou@nealrichardson@assignUser any idea if what can I try?

$ g++ -shared -s -static-libgcc -o arrow.dll tmp.def RTasks.o altrep.o array.o array_to_vector.o arraydata.o arrowExports.o bridge.o buffer.o chunkedarray.o compression.o compute-exec.o compute.o config.o csv.o dataset.o datatype.o expression.o extension-impl.o feather.o field.o filesystem.o io.o json.o memorypool.o message.o parquet.o r_to_arrow.o recordbatch.o recordbatchreader.o recordbatchwriter.o safe-call-into-r-impl.o scalar.o schema.o symbols.o table.o threadpool.o type_infer.o -L../windows/arrow-20.0.0.9000/lib-14.2.0/x64 -L../windows/arrow-20.0.0.9000/lib/x64-ucrt -larrow_dataset -larrow_acero -lparquet -larrow_compute -larrow -larrow_bundled_dependencies -lutf8proc -lsnappy -lz -lzstd -llz4 -lbz2 -lbrotlienc -lbrotlidec -lbrotlicommon -lole32 -lbcrypt -lpsapi -lcrypto -lcrypt32 -lre2 -luserenv -lversion -lws2_32 -lbcrypt -lwininet -lwinhttp -lsecur32 -lshlwapi -lncrypt -lcurl -lnormaliz -lssh2 -lgdi32 -lssl -lcrypto -lcrypt32 -lwldap32 -lz -lws2_32 -lnghttp2 -ldbghelp -LC:/rtools45/x86_64-w64-mingw32
C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_dataset.a(partition.cc.obj):(.text+0x7d9a): undefined reference to `__imp__ZN5arrow7compute7Grouper4MakeERKSt6vectorINS_10TypeHolderESaIS3_EEPNS0_11ExecContextE'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_dataset.a(partition.cc.obj):(.text+0x7f16): undefined reference to `__imp__ZN5arrow7compute7Grouper13MakeGroupingsERKNS_12NumericArrayINS_10UInt32TypeEEEjPNS0_11ExecContextE'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x1f2e): undefined reference to `__imp__ZN5arrow4util15TempVectorStack5allocEjPPhPi'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x1fb5): undefined reference to `__imp__ZN5arrow7compute9Hashing329HashBatchERKNS0_9ExecBatchEPjRSt6vectorINS0_14KeyColumnArrayESaIS7_EExPNS_4util15TempVectorStackExx'
C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x20b5): undefined reference to `__imp__ZN5arrow4util15TempVectorStack7releaseEij'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x2256): undefined reference to `__imp__ZN5arrow4util15TempVectorStack7releaseEij'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x7c05): undefined reference to `__imp__ZN5arrow4util15TempVectorStack4InitEPNS_10MemoryPoolEx'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x7c6f): undefined reference to `__imp__ZN5arrow4util15TempVectorStackD1Ev'
C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text$_ZN5arrow5acero12HashJoinNodeD1Ev[_ZN5arrow5acero12HashJoinNodeD1Ev]+0x2f): undefined reference to `__imp__ZN5arrow4util15TempVectorStackD1Ev'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text$_ZN5arrow5acero26BloomFilterPushdownContext17FilterSingleBatchEyPNS_7compute9ExecBatchE[_ZN5arrow5acero26BloomFilterPushdownContext17FilterSingleBatchEyPNS_7compute9ExecBatchE]+0x64f): undefined reference to `__imp__ZN5arrow7compute9Hashing329HashBatchERKNS0_9ExecBatchEPjRSt6vectorINS0_14KeyColumnArrayESaIS7_EExPNS_4util15TempVectorStackExx'

Still other improvements to be done but I wanted to have a clean CI PR to move to the cleaning and improvement stage.

[kou]

Acero and Dataset static libraries weren't built with -DARROW_COMPUTE_STATIC:

https://github.com/apache/arrow/actions/runs/14751558013/job/41410004318?pr=46261#step:7:14525

 [ 92%] Building CXX object src/arrow/acero/CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj
cd /D/a/arrow/arrow/src/build-x86_64-cpp/src/arrow/acero && /C/rtools40/ucrt64/bin/ccache.exe /C/rtools40/ucrt64/bin/c++.exe -DARROW_ACERO_EXPORTING -DARROW_ACERO_STATIC -DARROW_FLIGHT_SQL_STATIC -DARROW_FLIGHT_STATIC -DARROW_HAVE_RUNTIME_AVX2 -DARROW_HAVE_RUNTIME_BMI2 -DARROW_HAVE_RUNTIME_SSE4_2 -DARROW_HAVE_SSE4_2 -DARROW_STATIC -DARROW_WITH_TIMING_TESTS -DBOOST_ALL_NO_LIB -DUTF8PROC_STATIC -D_CRT_SECURE_NO_WARNINGS @CMakeFiles/arrow_acero_static.dir/includes_CXX.rsp -Wno-noexcept-type -DCURL_STATICLIB -fdiagnostics-color=always -Wa,-mbig-obj -Wall -fno-semantic-interposition -mxsave -msse4.2 -DUTF8PROC_STATIC -O3 -DNDEBUG -O2 -ftree-vectorize -std=c++17 -MD -MT src/arrow/acero/CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj -MF CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj.d -o CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj -c /D/a/arrow/arrow/cpp/src/arrow/acero/groupby_aggregate_node.cc

Could you add the following

if(ARROW_BUILD_STATIC ANDWIN32)
target_compile_definitions(arrow_compute_staticPUBLICARROW_COMPUTE_STATIC)
endif()

like we did for Flight?

arrow/cpp/src/arrow/flight/CMakeLists.txt

Lines 222 to 224 in baf97fd

	if(ARROW_BUILD_STATIC ANDWIN32)
	target_compile_definitions(arrow_flight_staticPUBLICARROW_FLIGHT_STATIC)
	endif()

[raulcd]

@github-actions crossbow submit -g nightly-tests

[raulcd]

@github-actions crossbow submit -g nightly-tests

[raulcd]

@github-actions crossbow submit -g cpp

[github-actions]

Revision: 9001ec2

Submitted crossbow builds: ursacomputing/crossbow @ actions-74be4cb1e9

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-39-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

[pitrou]

LGTM in general, I posted two small suggestions.

[zanmato1984]

+1.

Excellent work @raulcd !

[rok]

I was only able to do a quick pass. Looks great, love the ARROW_COMPUTE_EXPORT macro.

[jonkeane]

Thank you for all of this, it's been a long journey!

[raulcd]

Thanks everyone for taking the time to review. If no more concerns are raised, it has already been approved by 4 committers, I plan to merge in a couple days.

[raulcd]

Thanks everyone, I am merging!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 0 benchmarking runs that have been run so far on merge-commit 6a5db61.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.

[raulcd]

Rationale for this change

Arrow is quite a heavy dependency and some users don't need all the tools we provide bundled with libarrow. Moving Arrow Compute to its own shared library allows users installations that better suit their needs having smaller memory footprint if necessary.
It might also help some users adding new kernels into an existing Arrow without recompiling it.

What changes are included in this PR?

• Move all the Arrow Compute kernel functions to a new ArrowCompute shared library (libarrow_compute.so).
• Create a new API to Initialize arrow compute registering the existing Kernels into the FunctionRegistry
• Update Python/R/CGLib bindings to automatically register the Compute kernels transparently to the user.
• Update Linux Packaging to provide the new arrow-compute library.
• Update documentation with new requirements to call arrow::compute::Initialize()

Are these changes tested?

Yes on all CI jobs.

Are there any user-facing changes?

Yes. The Arrow compute functions will be provided as a different library. Any user using Arrow Compute from C++ directly will require a call to arrow::compute::Initialize() in order for the functions and kernels to be registered

This PR includes breaking changes to public APIs.

• GitHub Issue: [C++] Split non-cast compute kernels into a separate shared library #25025

[raulcd]

@github-actions crossbow submit -g cpp

[raulcd]

It seems I am down to one job failure to the Windows R release job (CGLib Ruby is failing due to an existing issue):
I am unsure why I am getting missing symbols on dataset and acero when building statically. I've tried several things, around our *.pc.in and configure.win files to try and fix linking to the new arrow_compute but I am unsure where the problem is exactly coming.
@kou@nealrichardson@assignUser any idea if what can I try?

$ g++ -shared -s -static-libgcc -o arrow.dll tmp.def RTasks.o altrep.o array.o array_to_vector.o arraydata.o arrowExports.o bridge.o buffer.o chunkedarray.o compression.o compute-exec.o compute.o config.o csv.o dataset.o datatype.o expression.o extension-impl.o feather.o field.o filesystem.o io.o json.o memorypool.o message.o parquet.o r_to_arrow.o recordbatch.o recordbatchreader.o recordbatchwriter.o safe-call-into-r-impl.o scalar.o schema.o symbols.o table.o threadpool.o type_infer.o -L../windows/arrow-20.0.0.9000/lib-14.2.0/x64 -L../windows/arrow-20.0.0.9000/lib/x64-ucrt -larrow_dataset -larrow_acero -lparquet -larrow_compute -larrow -larrow_bundled_dependencies -lutf8proc -lsnappy -lz -lzstd -llz4 -lbz2 -lbrotlienc -lbrotlidec -lbrotlicommon -lole32 -lbcrypt -lpsapi -lcrypto -lcrypt32 -lre2 -luserenv -lversion -lws2_32 -lbcrypt -lwininet -lwinhttp -lsecur32 -lshlwapi -lncrypt -lcurl -lnormaliz -lssh2 -lgdi32 -lssl -lcrypto -lcrypt32 -lwldap32 -lz -lws2_32 -lnghttp2 -ldbghelp -LC:/rtools45/x86_64-w64-mingw32
C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_dataset.a(partition.cc.obj):(.text+0x7d9a): undefined reference to `__imp__ZN5arrow7compute7Grouper4MakeERKSt6vectorINS_10TypeHolderESaIS3_EEPNS0_11ExecContextE'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_dataset.a(partition.cc.obj):(.text+0x7f16): undefined reference to `__imp__ZN5arrow7compute7Grouper13MakeGroupingsERKNS_12NumericArrayINS_10UInt32TypeEEEjPNS0_11ExecContextE'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x1f2e): undefined reference to `__imp__ZN5arrow4util15TempVectorStack5allocEjPPhPi'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x1fb5): undefined reference to `__imp__ZN5arrow7compute9Hashing329HashBatchERKNS0_9ExecBatchEPjRSt6vectorINS0_14KeyColumnArrayESaIS7_EExPNS_4util15TempVectorStackExx'
C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x20b5): undefined reference to `__imp__ZN5arrow4util15TempVectorStack7releaseEij'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x2256): undefined reference to `__imp__ZN5arrow4util15TempVectorStack7releaseEij'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x7c05): undefined reference to `__imp__ZN5arrow4util15TempVectorStack4InitEPNS_10MemoryPoolEx'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x7c6f): undefined reference to `__imp__ZN5arrow4util15TempVectorStackD1Ev'
C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text$_ZN5arrow5acero12HashJoinNodeD1Ev[_ZN5arrow5acero12HashJoinNodeD1Ev]+0x2f): undefined reference to `__imp__ZN5arrow4util15TempVectorStackD1Ev'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text$_ZN5arrow5acero26BloomFilterPushdownContext17FilterSingleBatchEyPNS_7compute9ExecBatchE[_ZN5arrow5acero26BloomFilterPushdownContext17FilterSingleBatchEyPNS_7compute9ExecBatchE]+0x64f): undefined reference to `__imp__ZN5arrow7compute9Hashing329HashBatchERKNS0_9ExecBatchEPjRSt6vectorINS0_14KeyColumnArrayESaIS7_EExPNS_4util15TempVectorStackExx'

Still other improvements to be done but I wanted to have a clean CI PR to move to the cleaning and improvement stage.

[kou]

Acero and Dataset static libraries weren't built with -DARROW_COMPUTE_STATIC:

https://github.com/apache/arrow/actions/runs/14751558013/job/41410004318?pr=46261#step:7:14525

 [ 92%] Building CXX object src/arrow/acero/CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj
cd /D/a/arrow/arrow/src/build-x86_64-cpp/src/arrow/acero && /C/rtools40/ucrt64/bin/ccache.exe /C/rtools40/ucrt64/bin/c++.exe -DARROW_ACERO_EXPORTING -DARROW_ACERO_STATIC -DARROW_FLIGHT_SQL_STATIC -DARROW_FLIGHT_STATIC -DARROW_HAVE_RUNTIME_AVX2 -DARROW_HAVE_RUNTIME_BMI2 -DARROW_HAVE_RUNTIME_SSE4_2 -DARROW_HAVE_SSE4_2 -DARROW_STATIC -DARROW_WITH_TIMING_TESTS -DBOOST_ALL_NO_LIB -DUTF8PROC_STATIC -D_CRT_SECURE_NO_WARNINGS @CMakeFiles/arrow_acero_static.dir/includes_CXX.rsp -Wno-noexcept-type -DCURL_STATICLIB -fdiagnostics-color=always -Wa,-mbig-obj -Wall -fno-semantic-interposition -mxsave -msse4.2 -DUTF8PROC_STATIC -O3 -DNDEBUG -O2 -ftree-vectorize -std=c++17 -MD -MT src/arrow/acero/CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj -MF CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj.d -o CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj -c /D/a/arrow/arrow/cpp/src/arrow/acero/groupby_aggregate_node.cc

Could you add the following

if(ARROW_BUILD_STATIC ANDWIN32)
target_compile_definitions(arrow_compute_staticPUBLICARROW_COMPUTE_STATIC)
endif()

like we did for Flight?

arrow/cpp/src/arrow/flight/CMakeLists.txt

Lines 222 to 224 in baf97fd

	if(ARROW_BUILD_STATIC ANDWIN32)
	target_compile_definitions(arrow_flight_staticPUBLICARROW_FLIGHT_STATIC)
	endif()

[raulcd]

@github-actions crossbow submit -g nightly-tests

[raulcd]

@github-actions crossbow submit -g nightly-tests

[raulcd]

@github-actions crossbow submit -g cpp

[github-actions]

Revision: 9001ec2

Submitted crossbow builds: ursacomputing/crossbow @ actions-74be4cb1e9

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-39-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

[pitrou]

LGTM in general, I posted two small suggestions.

[zanmato1984]

+1.

Excellent work @raulcd !

[rok]

I was only able to do a quick pass. Looks great, love the ARROW_COMPUTE_EXPORT macro.

[jonkeane]

Thank you for all of this, it's been a long journey!

[raulcd]

Thanks everyone for taking the time to review. If no more concerns are raised, it has already been approved by 4 committers, I plan to merge in a couple days.

[raulcd]

Thanks everyone, I am merging!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 0 benchmarking runs that have been run so far on merge-commit 6a5db61.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.

[raulcd]

Rationale for this change

Arrow is quite a heavy dependency and some users don't need all the tools we provide bundled with libarrow. Moving Arrow Compute to its own shared library allows users installations that better suit their needs having smaller memory footprint if necessary.
It might also help some users adding new kernels into an existing Arrow without recompiling it.

What changes are included in this PR?

• Move all the Arrow Compute kernel functions to a new ArrowCompute shared library (libarrow_compute.so).
• Create a new API to Initialize arrow compute registering the existing Kernels into the FunctionRegistry
• Update Python/R/CGLib bindings to automatically register the Compute kernels transparently to the user.
• Update Linux Packaging to provide the new arrow-compute library.
• Update documentation with new requirements to call arrow::compute::Initialize()

Are these changes tested?

Yes on all CI jobs.

Are there any user-facing changes?

Yes. The Arrow compute functions will be provided as a different library. Any user using Arrow Compute from C++ directly will require a call to arrow::compute::Initialize() in order for the functions and kernels to be registered

This PR includes breaking changes to public APIs.

• GitHub Issue: [C++] Split non-cast compute kernels into a separate shared library #25025

[raulcd]

@github-actions crossbow submit -g cpp

[raulcd]

It seems I am down to one job failure to the Windows R release job (CGLib Ruby is failing due to an existing issue):
I am unsure why I am getting missing symbols on dataset and acero when building statically. I've tried several things, around our *.pc.in and configure.win files to try and fix linking to the new arrow_compute but I am unsure where the problem is exactly coming.
@kou@nealrichardson@assignUser any idea if what can I try?

$ g++ -shared -s -static-libgcc -o arrow.dll tmp.def RTasks.o altrep.o array.o array_to_vector.o arraydata.o arrowExports.o bridge.o buffer.o chunkedarray.o compression.o compute-exec.o compute.o config.o csv.o dataset.o datatype.o expression.o extension-impl.o feather.o field.o filesystem.o io.o json.o memorypool.o message.o parquet.o r_to_arrow.o recordbatch.o recordbatchreader.o recordbatchwriter.o safe-call-into-r-impl.o scalar.o schema.o symbols.o table.o threadpool.o type_infer.o -L../windows/arrow-20.0.0.9000/lib-14.2.0/x64 -L../windows/arrow-20.0.0.9000/lib/x64-ucrt -larrow_dataset -larrow_acero -lparquet -larrow_compute -larrow -larrow_bundled_dependencies -lutf8proc -lsnappy -lz -lzstd -llz4 -lbz2 -lbrotlienc -lbrotlidec -lbrotlicommon -lole32 -lbcrypt -lpsapi -lcrypto -lcrypt32 -lre2 -luserenv -lversion -lws2_32 -lbcrypt -lwininet -lwinhttp -lsecur32 -lshlwapi -lncrypt -lcurl -lnormaliz -lssh2 -lgdi32 -lssl -lcrypto -lcrypt32 -lwldap32 -lz -lws2_32 -lnghttp2 -ldbghelp -LC:/rtools45/x86_64-w64-mingw32
C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_dataset.a(partition.cc.obj):(.text+0x7d9a): undefined reference to `__imp__ZN5arrow7compute7Grouper4MakeERKSt6vectorINS_10TypeHolderESaIS3_EEPNS0_11ExecContextE'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_dataset.a(partition.cc.obj):(.text+0x7f16): undefined reference to `__imp__ZN5arrow7compute7Grouper13MakeGroupingsERKNS_12NumericArrayINS_10UInt32TypeEEEjPNS0_11ExecContextE'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x1f2e): undefined reference to `__imp__ZN5arrow4util15TempVectorStack5allocEjPPhPi'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x1fb5): undefined reference to `__imp__ZN5arrow7compute9Hashing329HashBatchERKNS0_9ExecBatchEPjRSt6vectorINS0_14KeyColumnArrayESaIS7_EExPNS_4util15TempVectorStackExx'
C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x20b5): undefined reference to `__imp__ZN5arrow4util15TempVectorStack7releaseEij'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x2256): undefined reference to `__imp__ZN5arrow4util15TempVectorStack7releaseEij'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x7c05): undefined reference to `__imp__ZN5arrow4util15TempVectorStack4InitEPNS_10MemoryPoolEx'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x7c6f): undefined reference to `__imp__ZN5arrow4util15TempVectorStackD1Ev'
C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text$_ZN5arrow5acero12HashJoinNodeD1Ev[_ZN5arrow5acero12HashJoinNodeD1Ev]+0x2f): undefined reference to `__imp__ZN5arrow4util15TempVectorStackD1Ev'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text$_ZN5arrow5acero26BloomFilterPushdownContext17FilterSingleBatchEyPNS_7compute9ExecBatchE[_ZN5arrow5acero26BloomFilterPushdownContext17FilterSingleBatchEyPNS_7compute9ExecBatchE]+0x64f): undefined reference to `__imp__ZN5arrow7compute9Hashing329HashBatchERKNS0_9ExecBatchEPjRSt6vectorINS0_14KeyColumnArrayESaIS7_EExPNS_4util15TempVectorStackExx'

Still other improvements to be done but I wanted to have a clean CI PR to move to the cleaning and improvement stage.

[kou]

Acero and Dataset static libraries weren't built with -DARROW_COMPUTE_STATIC:

https://github.com/apache/arrow/actions/runs/14751558013/job/41410004318?pr=46261#step:7:14525

 [ 92%] Building CXX object src/arrow/acero/CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj
cd /D/a/arrow/arrow/src/build-x86_64-cpp/src/arrow/acero && /C/rtools40/ucrt64/bin/ccache.exe /C/rtools40/ucrt64/bin/c++.exe -DARROW_ACERO_EXPORTING -DARROW_ACERO_STATIC -DARROW_FLIGHT_SQL_STATIC -DARROW_FLIGHT_STATIC -DARROW_HAVE_RUNTIME_AVX2 -DARROW_HAVE_RUNTIME_BMI2 -DARROW_HAVE_RUNTIME_SSE4_2 -DARROW_HAVE_SSE4_2 -DARROW_STATIC -DARROW_WITH_TIMING_TESTS -DBOOST_ALL_NO_LIB -DUTF8PROC_STATIC -D_CRT_SECURE_NO_WARNINGS @CMakeFiles/arrow_acero_static.dir/includes_CXX.rsp -Wno-noexcept-type -DCURL_STATICLIB -fdiagnostics-color=always -Wa,-mbig-obj -Wall -fno-semantic-interposition -mxsave -msse4.2 -DUTF8PROC_STATIC -O3 -DNDEBUG -O2 -ftree-vectorize -std=c++17 -MD -MT src/arrow/acero/CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj -MF CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj.d -o CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj -c /D/a/arrow/arrow/cpp/src/arrow/acero/groupby_aggregate_node.cc

Could you add the following

if(ARROW_BUILD_STATIC ANDWIN32)
target_compile_definitions(arrow_compute_staticPUBLICARROW_COMPUTE_STATIC)
endif()

like we did for Flight?

arrow/cpp/src/arrow/flight/CMakeLists.txt

Lines 222 to 224 in baf97fd

	if(ARROW_BUILD_STATIC ANDWIN32)
	target_compile_definitions(arrow_flight_staticPUBLICARROW_FLIGHT_STATIC)
	endif()

[raulcd]

@github-actions crossbow submit -g nightly-tests

[raulcd]

@github-actions crossbow submit -g nightly-tests

[raulcd]

@github-actions crossbow submit -g cpp

[github-actions]

Revision: 9001ec2

Submitted crossbow builds: ursacomputing/crossbow @ actions-74be4cb1e9

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-39-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

[pitrou]

LGTM in general, I posted two small suggestions.

[zanmato1984]

+1.

Excellent work @raulcd !

[rok]

I was only able to do a quick pass. Looks great, love the ARROW_COMPUTE_EXPORT macro.

[jonkeane]

Thank you for all of this, it's been a long journey!

[raulcd]

Thanks everyone for taking the time to review. If no more concerns are raised, it has already been approved by 4 committers, I plan to merge in a couple days.

[raulcd]

Thanks everyone, I am merging!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 0 benchmarking runs that have been run so far on merge-commit 6a5db61.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.

[raulcd]

Rationale for this change

Arrow is quite a heavy dependency and some users don't need all the tools we provide bundled with libarrow. Moving Arrow Compute to its own shared library allows users installations that better suit their needs having smaller memory footprint if necessary.
It might also help some users adding new kernels into an existing Arrow without recompiling it.

What changes are included in this PR?

• Move all the Arrow Compute kernel functions to a new ArrowCompute shared library (libarrow_compute.so).
• Create a new API to Initialize arrow compute registering the existing Kernels into the FunctionRegistry
• Update Python/R/CGLib bindings to automatically register the Compute kernels transparently to the user.
• Update Linux Packaging to provide the new arrow-compute library.
• Update documentation with new requirements to call arrow::compute::Initialize()

Are these changes tested?

Yes on all CI jobs.

Are there any user-facing changes?

Yes. The Arrow compute functions will be provided as a different library. Any user using Arrow Compute from C++ directly will require a call to arrow::compute::Initialize() in order for the functions and kernels to be registered

This PR includes breaking changes to public APIs.

• GitHub Issue: [C++] Split non-cast compute kernels into a separate shared library #25025

[raulcd]

@github-actions crossbow submit -g cpp

[raulcd]

It seems I am down to one job failure to the Windows R release job (CGLib Ruby is failing due to an existing issue):
I am unsure why I am getting missing symbols on dataset and acero when building statically. I've tried several things, around our *.pc.in and configure.win files to try and fix linking to the new arrow_compute but I am unsure where the problem is exactly coming.
@kou@nealrichardson@assignUser any idea if what can I try?

$ g++ -shared -s -static-libgcc -o arrow.dll tmp.def RTasks.o altrep.o array.o array_to_vector.o arraydata.o arrowExports.o bridge.o buffer.o chunkedarray.o compression.o compute-exec.o compute.o config.o csv.o dataset.o datatype.o expression.o extension-impl.o feather.o field.o filesystem.o io.o json.o memorypool.o message.o parquet.o r_to_arrow.o recordbatch.o recordbatchreader.o recordbatchwriter.o safe-call-into-r-impl.o scalar.o schema.o symbols.o table.o threadpool.o type_infer.o -L../windows/arrow-20.0.0.9000/lib-14.2.0/x64 -L../windows/arrow-20.0.0.9000/lib/x64-ucrt -larrow_dataset -larrow_acero -lparquet -larrow_compute -larrow -larrow_bundled_dependencies -lutf8proc -lsnappy -lz -lzstd -llz4 -lbz2 -lbrotlienc -lbrotlidec -lbrotlicommon -lole32 -lbcrypt -lpsapi -lcrypto -lcrypt32 -lre2 -luserenv -lversion -lws2_32 -lbcrypt -lwininet -lwinhttp -lsecur32 -lshlwapi -lncrypt -lcurl -lnormaliz -lssh2 -lgdi32 -lssl -lcrypto -lcrypt32 -lwldap32 -lz -lws2_32 -lnghttp2 -ldbghelp -LC:/rtools45/x86_64-w64-mingw32
C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_dataset.a(partition.cc.obj):(.text+0x7d9a): undefined reference to `__imp__ZN5arrow7compute7Grouper4MakeERKSt6vectorINS_10TypeHolderESaIS3_EEPNS0_11ExecContextE'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_dataset.a(partition.cc.obj):(.text+0x7f16): undefined reference to `__imp__ZN5arrow7compute7Grouper13MakeGroupingsERKNS_12NumericArrayINS_10UInt32TypeEEEjPNS0_11ExecContextE'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x1f2e): undefined reference to `__imp__ZN5arrow4util15TempVectorStack5allocEjPPhPi'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x1fb5): undefined reference to `__imp__ZN5arrow7compute9Hashing329HashBatchERKNS0_9ExecBatchEPjRSt6vectorINS0_14KeyColumnArrayESaIS7_EExPNS_4util15TempVectorStackExx'
C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x20b5): undefined reference to `__imp__ZN5arrow4util15TempVectorStack7releaseEij'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x2256): undefined reference to `__imp__ZN5arrow4util15TempVectorStack7releaseEij'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x7c05): undefined reference to `__imp__ZN5arrow4util15TempVectorStack4InitEPNS_10MemoryPoolEx'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x7c6f): undefined reference to `__imp__ZN5arrow4util15TempVectorStackD1Ev'
C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text$_ZN5arrow5acero12HashJoinNodeD1Ev[_ZN5arrow5acero12HashJoinNodeD1Ev]+0x2f): undefined reference to `__imp__ZN5arrow4util15TempVectorStackD1Ev'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text$_ZN5arrow5acero26BloomFilterPushdownContext17FilterSingleBatchEyPNS_7compute9ExecBatchE[_ZN5arrow5acero26BloomFilterPushdownContext17FilterSingleBatchEyPNS_7compute9ExecBatchE]+0x64f): undefined reference to `__imp__ZN5arrow7compute9Hashing329HashBatchERKNS0_9ExecBatchEPjRSt6vectorINS0_14KeyColumnArrayESaIS7_EExPNS_4util15TempVectorStackExx'

Still other improvements to be done but I wanted to have a clean CI PR to move to the cleaning and improvement stage.

[kou]

Acero and Dataset static libraries weren't built with -DARROW_COMPUTE_STATIC:

https://github.com/apache/arrow/actions/runs/14751558013/job/41410004318?pr=46261#step:7:14525

 [ 92%] Building CXX object src/arrow/acero/CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj
cd /D/a/arrow/arrow/src/build-x86_64-cpp/src/arrow/acero && /C/rtools40/ucrt64/bin/ccache.exe /C/rtools40/ucrt64/bin/c++.exe -DARROW_ACERO_EXPORTING -DARROW_ACERO_STATIC -DARROW_FLIGHT_SQL_STATIC -DARROW_FLIGHT_STATIC -DARROW_HAVE_RUNTIME_AVX2 -DARROW_HAVE_RUNTIME_BMI2 -DARROW_HAVE_RUNTIME_SSE4_2 -DARROW_HAVE_SSE4_2 -DARROW_STATIC -DARROW_WITH_TIMING_TESTS -DBOOST_ALL_NO_LIB -DUTF8PROC_STATIC -D_CRT_SECURE_NO_WARNINGS @CMakeFiles/arrow_acero_static.dir/includes_CXX.rsp -Wno-noexcept-type -DCURL_STATICLIB -fdiagnostics-color=always -Wa,-mbig-obj -Wall -fno-semantic-interposition -mxsave -msse4.2 -DUTF8PROC_STATIC -O3 -DNDEBUG -O2 -ftree-vectorize -std=c++17 -MD -MT src/arrow/acero/CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj -MF CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj.d -o CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj -c /D/a/arrow/arrow/cpp/src/arrow/acero/groupby_aggregate_node.cc

Could you add the following

if(ARROW_BUILD_STATIC ANDWIN32)
target_compile_definitions(arrow_compute_staticPUBLICARROW_COMPUTE_STATIC)
endif()

like we did for Flight?

arrow/cpp/src/arrow/flight/CMakeLists.txt

Lines 222 to 224 in baf97fd

	if(ARROW_BUILD_STATIC ANDWIN32)
	target_compile_definitions(arrow_flight_staticPUBLICARROW_FLIGHT_STATIC)
	endif()

[raulcd]

@github-actions crossbow submit -g nightly-tests

[raulcd]

@github-actions crossbow submit -g nightly-tests

[raulcd]

@github-actions crossbow submit -g cpp

[github-actions]

Revision: 9001ec2

Submitted crossbow builds: ursacomputing/crossbow @ actions-74be4cb1e9

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-39-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

[pitrou]

LGTM in general, I posted two small suggestions.

[zanmato1984]

+1.

Excellent work @raulcd !

[rok]

I was only able to do a quick pass. Looks great, love the ARROW_COMPUTE_EXPORT macro.

[jonkeane]

Thank you for all of this, it's been a long journey!

[raulcd]

Thanks everyone for taking the time to review. If no more concerns are raised, it has already been approved by 4 committers, I plan to merge in a couple days.

[raulcd]

Thanks everyone, I am merging!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 0 benchmarking runs that have been run so far on merge-commit 6a5db61.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.

[raulcd]

Rationale for this change

Arrow is quite a heavy dependency and some users don't need all the tools we provide bundled with libarrow. Moving Arrow Compute to its own shared library allows users installations that better suit their needs having smaller memory footprint if necessary.
It might also help some users adding new kernels into an existing Arrow without recompiling it.

What changes are included in this PR?

• Move all the Arrow Compute kernel functions to a new ArrowCompute shared library (libarrow_compute.so).
• Create a new API to Initialize arrow compute registering the existing Kernels into the FunctionRegistry
• Update Python/R/CGLib bindings to automatically register the Compute kernels transparently to the user.
• Update Linux Packaging to provide the new arrow-compute library.
• Update documentation with new requirements to call arrow::compute::Initialize()

Are these changes tested?

Yes on all CI jobs.

Are there any user-facing changes?

Yes. The Arrow compute functions will be provided as a different library. Any user using Arrow Compute from C++ directly will require a call to arrow::compute::Initialize() in order for the functions and kernels to be registered

This PR includes breaking changes to public APIs.

• GitHub Issue: [C++] Split non-cast compute kernels into a separate shared library #25025

[raulcd]

@github-actions crossbow submit -g cpp

[raulcd]

It seems I am down to one job failure to the Windows R release job (CGLib Ruby is failing due to an existing issue):
I am unsure why I am getting missing symbols on dataset and acero when building statically. I've tried several things, around our *.pc.in and configure.win files to try and fix linking to the new arrow_compute but I am unsure where the problem is exactly coming.
@kou@nealrichardson@assignUser any idea if what can I try?

$ g++ -shared -s -static-libgcc -o arrow.dll tmp.def RTasks.o altrep.o array.o array_to_vector.o arraydata.o arrowExports.o bridge.o buffer.o chunkedarray.o compression.o compute-exec.o compute.o config.o csv.o dataset.o datatype.o expression.o extension-impl.o feather.o field.o filesystem.o io.o json.o memorypool.o message.o parquet.o r_to_arrow.o recordbatch.o recordbatchreader.o recordbatchwriter.o safe-call-into-r-impl.o scalar.o schema.o symbols.o table.o threadpool.o type_infer.o -L../windows/arrow-20.0.0.9000/lib-14.2.0/x64 -L../windows/arrow-20.0.0.9000/lib/x64-ucrt -larrow_dataset -larrow_acero -lparquet -larrow_compute -larrow -larrow_bundled_dependencies -lutf8proc -lsnappy -lz -lzstd -llz4 -lbz2 -lbrotlienc -lbrotlidec -lbrotlicommon -lole32 -lbcrypt -lpsapi -lcrypto -lcrypt32 -lre2 -luserenv -lversion -lws2_32 -lbcrypt -lwininet -lwinhttp -lsecur32 -lshlwapi -lncrypt -lcurl -lnormaliz -lssh2 -lgdi32 -lssl -lcrypto -lcrypt32 -lwldap32 -lz -lws2_32 -lnghttp2 -ldbghelp -LC:/rtools45/x86_64-w64-mingw32
C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_dataset.a(partition.cc.obj):(.text+0x7d9a): undefined reference to `__imp__ZN5arrow7compute7Grouper4MakeERKSt6vectorINS_10TypeHolderESaIS3_EEPNS0_11ExecContextE'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_dataset.a(partition.cc.obj):(.text+0x7f16): undefined reference to `__imp__ZN5arrow7compute7Grouper13MakeGroupingsERKNS_12NumericArrayINS_10UInt32TypeEEEjPNS0_11ExecContextE'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x1f2e): undefined reference to `__imp__ZN5arrow4util15TempVectorStack5allocEjPPhPi'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x1fb5): undefined reference to `__imp__ZN5arrow7compute9Hashing329HashBatchERKNS0_9ExecBatchEPjRSt6vectorINS0_14KeyColumnArrayESaIS7_EExPNS_4util15TempVectorStackExx'
C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x20b5): undefined reference to `__imp__ZN5arrow4util15TempVectorStack7releaseEij'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x2256): undefined reference to `__imp__ZN5arrow4util15TempVectorStack7releaseEij'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x7c05): undefined reference to `__imp__ZN5arrow4util15TempVectorStack4InitEPNS_10MemoryPoolEx'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x7c6f): undefined reference to `__imp__ZN5arrow4util15TempVectorStackD1Ev'
C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text$_ZN5arrow5acero12HashJoinNodeD1Ev[_ZN5arrow5acero12HashJoinNodeD1Ev]+0x2f): undefined reference to `__imp__ZN5arrow4util15TempVectorStackD1Ev'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text$_ZN5arrow5acero26BloomFilterPushdownContext17FilterSingleBatchEyPNS_7compute9ExecBatchE[_ZN5arrow5acero26BloomFilterPushdownContext17FilterSingleBatchEyPNS_7compute9ExecBatchE]+0x64f): undefined reference to `__imp__ZN5arrow7compute9Hashing329HashBatchERKNS0_9ExecBatchEPjRSt6vectorINS0_14KeyColumnArrayESaIS7_EExPNS_4util15TempVectorStackExx'

Still other improvements to be done but I wanted to have a clean CI PR to move to the cleaning and improvement stage.

[kou]

Acero and Dataset static libraries weren't built with -DARROW_COMPUTE_STATIC:

https://github.com/apache/arrow/actions/runs/14751558013/job/41410004318?pr=46261#step:7:14525

 [ 92%] Building CXX object src/arrow/acero/CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj
cd /D/a/arrow/arrow/src/build-x86_64-cpp/src/arrow/acero && /C/rtools40/ucrt64/bin/ccache.exe /C/rtools40/ucrt64/bin/c++.exe -DARROW_ACERO_EXPORTING -DARROW_ACERO_STATIC -DARROW_FLIGHT_SQL_STATIC -DARROW_FLIGHT_STATIC -DARROW_HAVE_RUNTIME_AVX2 -DARROW_HAVE_RUNTIME_BMI2 -DARROW_HAVE_RUNTIME_SSE4_2 -DARROW_HAVE_SSE4_2 -DARROW_STATIC -DARROW_WITH_TIMING_TESTS -DBOOST_ALL_NO_LIB -DUTF8PROC_STATIC -D_CRT_SECURE_NO_WARNINGS @CMakeFiles/arrow_acero_static.dir/includes_CXX.rsp -Wno-noexcept-type -DCURL_STATICLIB -fdiagnostics-color=always -Wa,-mbig-obj -Wall -fno-semantic-interposition -mxsave -msse4.2 -DUTF8PROC_STATIC -O3 -DNDEBUG -O2 -ftree-vectorize -std=c++17 -MD -MT src/arrow/acero/CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj -MF CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj.d -o CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj -c /D/a/arrow/arrow/cpp/src/arrow/acero/groupby_aggregate_node.cc

Could you add the following

if(ARROW_BUILD_STATIC ANDWIN32)
target_compile_definitions(arrow_compute_staticPUBLICARROW_COMPUTE_STATIC)
endif()

like we did for Flight?

arrow/cpp/src/arrow/flight/CMakeLists.txt

Lines 222 to 224 in baf97fd

	if(ARROW_BUILD_STATIC ANDWIN32)
	target_compile_definitions(arrow_flight_staticPUBLICARROW_FLIGHT_STATIC)
	endif()

[raulcd]

@github-actions crossbow submit -g nightly-tests

[raulcd]

@github-actions crossbow submit -g nightly-tests

[raulcd]

@github-actions crossbow submit -g cpp

[github-actions]

Revision: 9001ec2

Submitted crossbow builds: ursacomputing/crossbow @ actions-74be4cb1e9

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-39-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

[pitrou]

LGTM in general, I posted two small suggestions.

[zanmato1984]

+1.

Excellent work @raulcd !

[rok]

I was only able to do a quick pass. Looks great, love the ARROW_COMPUTE_EXPORT macro.

[jonkeane]

Thank you for all of this, it's been a long journey!

[raulcd]

Thanks everyone for taking the time to review. If no more concerns are raised, it has already been approved by 4 committers, I plan to merge in a couple days.

[raulcd]

Thanks everyone, I am merging!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 0 benchmarking runs that have been run so far on merge-commit 6a5db61.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.

[raulcd]

Rationale for this change

Arrow is quite a heavy dependency and some users don't need all the tools we provide bundled with libarrow. Moving Arrow Compute to its own shared library allows users installations that better suit their needs having smaller memory footprint if necessary.
It might also help some users adding new kernels into an existing Arrow without recompiling it.

What changes are included in this PR?

• Move all the Arrow Compute kernel functions to a new ArrowCompute shared library (libarrow_compute.so).
• Create a new API to Initialize arrow compute registering the existing Kernels into the FunctionRegistry
• Update Python/R/CGLib bindings to automatically register the Compute kernels transparently to the user.
• Update Linux Packaging to provide the new arrow-compute library.
• Update documentation with new requirements to call arrow::compute::Initialize()

Are these changes tested?

Yes on all CI jobs.

Are there any user-facing changes?

Yes. The Arrow compute functions will be provided as a different library. Any user using Arrow Compute from C++ directly will require a call to arrow::compute::Initialize() in order for the functions and kernels to be registered

This PR includes breaking changes to public APIs.

• GitHub Issue: [C++] Split non-cast compute kernels into a separate shared library #25025

[raulcd]

@github-actions crossbow submit -g cpp

[raulcd]

It seems I am down to one job failure to the Windows R release job (CGLib Ruby is failing due to an existing issue):
I am unsure why I am getting missing symbols on dataset and acero when building statically. I've tried several things, around our *.pc.in and configure.win files to try and fix linking to the new arrow_compute but I am unsure where the problem is exactly coming.
@kou@nealrichardson@assignUser any idea if what can I try?

$ g++ -shared -s -static-libgcc -o arrow.dll tmp.def RTasks.o altrep.o array.o array_to_vector.o arraydata.o arrowExports.o bridge.o buffer.o chunkedarray.o compression.o compute-exec.o compute.o config.o csv.o dataset.o datatype.o expression.o extension-impl.o feather.o field.o filesystem.o io.o json.o memorypool.o message.o parquet.o r_to_arrow.o recordbatch.o recordbatchreader.o recordbatchwriter.o safe-call-into-r-impl.o scalar.o schema.o symbols.o table.o threadpool.o type_infer.o -L../windows/arrow-20.0.0.9000/lib-14.2.0/x64 -L../windows/arrow-20.0.0.9000/lib/x64-ucrt -larrow_dataset -larrow_acero -lparquet -larrow_compute -larrow -larrow_bundled_dependencies -lutf8proc -lsnappy -lz -lzstd -llz4 -lbz2 -lbrotlienc -lbrotlidec -lbrotlicommon -lole32 -lbcrypt -lpsapi -lcrypto -lcrypt32 -lre2 -luserenv -lversion -lws2_32 -lbcrypt -lwininet -lwinhttp -lsecur32 -lshlwapi -lncrypt -lcurl -lnormaliz -lssh2 -lgdi32 -lssl -lcrypto -lcrypt32 -lwldap32 -lz -lws2_32 -lnghttp2 -ldbghelp -LC:/rtools45/x86_64-w64-mingw32
C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_dataset.a(partition.cc.obj):(.text+0x7d9a): undefined reference to `__imp__ZN5arrow7compute7Grouper4MakeERKSt6vectorINS_10TypeHolderESaIS3_EEPNS0_11ExecContextE'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_dataset.a(partition.cc.obj):(.text+0x7f16): undefined reference to `__imp__ZN5arrow7compute7Grouper13MakeGroupingsERKNS_12NumericArrayINS_10UInt32TypeEEEjPNS0_11ExecContextE'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x1f2e): undefined reference to `__imp__ZN5arrow4util15TempVectorStack5allocEjPPhPi'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x1fb5): undefined reference to `__imp__ZN5arrow7compute9Hashing329HashBatchERKNS0_9ExecBatchEPjRSt6vectorINS0_14KeyColumnArrayESaIS7_EExPNS_4util15TempVectorStackExx'
C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x20b5): undefined reference to `__imp__ZN5arrow4util15TempVectorStack7releaseEij'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x2256): undefined reference to `__imp__ZN5arrow4util15TempVectorStack7releaseEij'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x7c05): undefined reference to `__imp__ZN5arrow4util15TempVectorStack4InitEPNS_10MemoryPoolEx'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x7c6f): undefined reference to `__imp__ZN5arrow4util15TempVectorStackD1Ev'
C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text$_ZN5arrow5acero12HashJoinNodeD1Ev[_ZN5arrow5acero12HashJoinNodeD1Ev]+0x2f): undefined reference to `__imp__ZN5arrow4util15TempVectorStackD1Ev'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text$_ZN5arrow5acero26BloomFilterPushdownContext17FilterSingleBatchEyPNS_7compute9ExecBatchE[_ZN5arrow5acero26BloomFilterPushdownContext17FilterSingleBatchEyPNS_7compute9ExecBatchE]+0x64f): undefined reference to `__imp__ZN5arrow7compute9Hashing329HashBatchERKNS0_9ExecBatchEPjRSt6vectorINS0_14KeyColumnArrayESaIS7_EExPNS_4util15TempVectorStackExx'

Still other improvements to be done but I wanted to have a clean CI PR to move to the cleaning and improvement stage.

[kou]

Acero and Dataset static libraries weren't built with -DARROW_COMPUTE_STATIC:

https://github.com/apache/arrow/actions/runs/14751558013/job/41410004318?pr=46261#step:7:14525

 [ 92%] Building CXX object src/arrow/acero/CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj
cd /D/a/arrow/arrow/src/build-x86_64-cpp/src/arrow/acero && /C/rtools40/ucrt64/bin/ccache.exe /C/rtools40/ucrt64/bin/c++.exe -DARROW_ACERO_EXPORTING -DARROW_ACERO_STATIC -DARROW_FLIGHT_SQL_STATIC -DARROW_FLIGHT_STATIC -DARROW_HAVE_RUNTIME_AVX2 -DARROW_HAVE_RUNTIME_BMI2 -DARROW_HAVE_RUNTIME_SSE4_2 -DARROW_HAVE_SSE4_2 -DARROW_STATIC -DARROW_WITH_TIMING_TESTS -DBOOST_ALL_NO_LIB -DUTF8PROC_STATIC -D_CRT_SECURE_NO_WARNINGS @CMakeFiles/arrow_acero_static.dir/includes_CXX.rsp -Wno-noexcept-type -DCURL_STATICLIB -fdiagnostics-color=always -Wa,-mbig-obj -Wall -fno-semantic-interposition -mxsave -msse4.2 -DUTF8PROC_STATIC -O3 -DNDEBUG -O2 -ftree-vectorize -std=c++17 -MD -MT src/arrow/acero/CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj -MF CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj.d -o CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj -c /D/a/arrow/arrow/cpp/src/arrow/acero/groupby_aggregate_node.cc

Could you add the following

if(ARROW_BUILD_STATIC ANDWIN32)
target_compile_definitions(arrow_compute_staticPUBLICARROW_COMPUTE_STATIC)
endif()

like we did for Flight?

arrow/cpp/src/arrow/flight/CMakeLists.txt

Lines 222 to 224 in baf97fd

	if(ARROW_BUILD_STATIC ANDWIN32)
	target_compile_definitions(arrow_flight_staticPUBLICARROW_FLIGHT_STATIC)
	endif()

[raulcd]

@github-actions crossbow submit -g nightly-tests

[raulcd]

@github-actions crossbow submit -g nightly-tests

[raulcd]

@github-actions crossbow submit -g cpp

[github-actions]

Revision: 9001ec2

Submitted crossbow builds: ursacomputing/crossbow @ actions-74be4cb1e9

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-39-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

[pitrou]

LGTM in general, I posted two small suggestions.

[zanmato1984]

+1.

Excellent work @raulcd !

[rok]

I was only able to do a quick pass. Looks great, love the ARROW_COMPUTE_EXPORT macro.

[jonkeane]

Thank you for all of this, it's been a long journey!

[raulcd]

Thanks everyone for taking the time to review. If no more concerns are raised, it has already been approved by 4 committers, I plan to merge in a couple days.

[raulcd]

Thanks everyone, I am merging!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 0 benchmarking runs that have been run so far on merge-commit 6a5db61.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.

[raulcd]

Rationale for this change

Arrow is quite a heavy dependency and some users don't need all the tools we provide bundled with libarrow. Moving Arrow Compute to its own shared library allows users installations that better suit their needs having smaller memory footprint if necessary.
It might also help some users adding new kernels into an existing Arrow without recompiling it.

What changes are included in this PR?

• Move all the Arrow Compute kernel functions to a new ArrowCompute shared library (libarrow_compute.so).
• Create a new API to Initialize arrow compute registering the existing Kernels into the FunctionRegistry
• Update Python/R/CGLib bindings to automatically register the Compute kernels transparently to the user.
• Update Linux Packaging to provide the new arrow-compute library.
• Update documentation with new requirements to call arrow::compute::Initialize()

Are these changes tested?

Yes on all CI jobs.

Are there any user-facing changes?

Yes. The Arrow compute functions will be provided as a different library. Any user using Arrow Compute from C++ directly will require a call to arrow::compute::Initialize() in order for the functions and kernels to be registered

This PR includes breaking changes to public APIs.

• GitHub Issue: [C++] Split non-cast compute kernels into a separate shared library #25025

[raulcd]

@github-actions crossbow submit -g cpp

[raulcd]

It seems I am down to one job failure to the Windows R release job (CGLib Ruby is failing due to an existing issue):
I am unsure why I am getting missing symbols on dataset and acero when building statically. I've tried several things, around our *.pc.in and configure.win files to try and fix linking to the new arrow_compute but I am unsure where the problem is exactly coming.
@kou@nealrichardson@assignUser any idea if what can I try?

$ g++ -shared -s -static-libgcc -o arrow.dll tmp.def RTasks.o altrep.o array.o array_to_vector.o arraydata.o arrowExports.o bridge.o buffer.o chunkedarray.o compression.o compute-exec.o compute.o config.o csv.o dataset.o datatype.o expression.o extension-impl.o feather.o field.o filesystem.o io.o json.o memorypool.o message.o parquet.o r_to_arrow.o recordbatch.o recordbatchreader.o recordbatchwriter.o safe-call-into-r-impl.o scalar.o schema.o symbols.o table.o threadpool.o type_infer.o -L../windows/arrow-20.0.0.9000/lib-14.2.0/x64 -L../windows/arrow-20.0.0.9000/lib/x64-ucrt -larrow_dataset -larrow_acero -lparquet -larrow_compute -larrow -larrow_bundled_dependencies -lutf8proc -lsnappy -lz -lzstd -llz4 -lbz2 -lbrotlienc -lbrotlidec -lbrotlicommon -lole32 -lbcrypt -lpsapi -lcrypto -lcrypt32 -lre2 -luserenv -lversion -lws2_32 -lbcrypt -lwininet -lwinhttp -lsecur32 -lshlwapi -lncrypt -lcurl -lnormaliz -lssh2 -lgdi32 -lssl -lcrypto -lcrypt32 -lwldap32 -lz -lws2_32 -lnghttp2 -ldbghelp -LC:/rtools45/x86_64-w64-mingw32
C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_dataset.a(partition.cc.obj):(.text+0x7d9a): undefined reference to `__imp__ZN5arrow7compute7Grouper4MakeERKSt6vectorINS_10TypeHolderESaIS3_EEPNS0_11ExecContextE'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_dataset.a(partition.cc.obj):(.text+0x7f16): undefined reference to `__imp__ZN5arrow7compute7Grouper13MakeGroupingsERKNS_12NumericArrayINS_10UInt32TypeEEEjPNS0_11ExecContextE'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x1f2e): undefined reference to `__imp__ZN5arrow4util15TempVectorStack5allocEjPPhPi'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x1fb5): undefined reference to `__imp__ZN5arrow7compute9Hashing329HashBatchERKNS0_9ExecBatchEPjRSt6vectorINS0_14KeyColumnArrayESaIS7_EExPNS_4util15TempVectorStackExx'
C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x20b5): undefined reference to `__imp__ZN5arrow4util15TempVectorStack7releaseEij'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x2256): undefined reference to `__imp__ZN5arrow4util15TempVectorStack7releaseEij'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x7c05): undefined reference to `__imp__ZN5arrow4util15TempVectorStack4InitEPNS_10MemoryPoolEx'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x7c6f): undefined reference to `__imp__ZN5arrow4util15TempVectorStackD1Ev'
C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text$_ZN5arrow5acero12HashJoinNodeD1Ev[_ZN5arrow5acero12HashJoinNodeD1Ev]+0x2f): undefined reference to `__imp__ZN5arrow4util15TempVectorStackD1Ev'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text$_ZN5arrow5acero26BloomFilterPushdownContext17FilterSingleBatchEyPNS_7compute9ExecBatchE[_ZN5arrow5acero26BloomFilterPushdownContext17FilterSingleBatchEyPNS_7compute9ExecBatchE]+0x64f): undefined reference to `__imp__ZN5arrow7compute9Hashing329HashBatchERKNS0_9ExecBatchEPjRSt6vectorINS0_14KeyColumnArrayESaIS7_EExPNS_4util15TempVectorStackExx'

Still other improvements to be done but I wanted to have a clean CI PR to move to the cleaning and improvement stage.

[kou]

Acero and Dataset static libraries weren't built with -DARROW_COMPUTE_STATIC:

https://github.com/apache/arrow/actions/runs/14751558013/job/41410004318?pr=46261#step:7:14525

 [ 92%] Building CXX object src/arrow/acero/CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj
cd /D/a/arrow/arrow/src/build-x86_64-cpp/src/arrow/acero && /C/rtools40/ucrt64/bin/ccache.exe /C/rtools40/ucrt64/bin/c++.exe -DARROW_ACERO_EXPORTING -DARROW_ACERO_STATIC -DARROW_FLIGHT_SQL_STATIC -DARROW_FLIGHT_STATIC -DARROW_HAVE_RUNTIME_AVX2 -DARROW_HAVE_RUNTIME_BMI2 -DARROW_HAVE_RUNTIME_SSE4_2 -DARROW_HAVE_SSE4_2 -DARROW_STATIC -DARROW_WITH_TIMING_TESTS -DBOOST_ALL_NO_LIB -DUTF8PROC_STATIC -D_CRT_SECURE_NO_WARNINGS @CMakeFiles/arrow_acero_static.dir/includes_CXX.rsp -Wno-noexcept-type -DCURL_STATICLIB -fdiagnostics-color=always -Wa,-mbig-obj -Wall -fno-semantic-interposition -mxsave -msse4.2 -DUTF8PROC_STATIC -O3 -DNDEBUG -O2 -ftree-vectorize -std=c++17 -MD -MT src/arrow/acero/CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj -MF CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj.d -o CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj -c /D/a/arrow/arrow/cpp/src/arrow/acero/groupby_aggregate_node.cc

Could you add the following

if(ARROW_BUILD_STATIC ANDWIN32)
target_compile_definitions(arrow_compute_staticPUBLICARROW_COMPUTE_STATIC)
endif()

like we did for Flight?

arrow/cpp/src/arrow/flight/CMakeLists.txt

Lines 222 to 224 in baf97fd

	if(ARROW_BUILD_STATIC ANDWIN32)
	target_compile_definitions(arrow_flight_staticPUBLICARROW_FLIGHT_STATIC)
	endif()

[raulcd]

@github-actions crossbow submit -g nightly-tests

[raulcd]

@github-actions crossbow submit -g nightly-tests

[raulcd]

@github-actions crossbow submit -g cpp

[github-actions]

Revision: 9001ec2

Submitted crossbow builds: ursacomputing/crossbow @ actions-74be4cb1e9

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-39-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

[pitrou]

LGTM in general, I posted two small suggestions.

[zanmato1984]

+1.

Excellent work @raulcd !

[rok]

I was only able to do a quick pass. Looks great, love the ARROW_COMPUTE_EXPORT macro.

[jonkeane]

Thank you for all of this, it's been a long journey!

[raulcd]

Thanks everyone for taking the time to review. If no more concerns are raised, it has already been approved by 4 committers, I plan to merge in a couple days.

[raulcd]

Thanks everyone, I am merging!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 0 benchmarking runs that have been run so far on merge-commit 6a5db61.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.

[raulcd]

Rationale for this change

Arrow is quite a heavy dependency and some users don't need all the tools we provide bundled with libarrow. Moving Arrow Compute to its own shared library allows users installations that better suit their needs having smaller memory footprint if necessary.
It might also help some users adding new kernels into an existing Arrow without recompiling it.

What changes are included in this PR?

• Move all the Arrow Compute kernel functions to a new ArrowCompute shared library (libarrow_compute.so).
• Create a new API to Initialize arrow compute registering the existing Kernels into the FunctionRegistry
• Update Python/R/CGLib bindings to automatically register the Compute kernels transparently to the user.
• Update Linux Packaging to provide the new arrow-compute library.
• Update documentation with new requirements to call arrow::compute::Initialize()

Are these changes tested?

Yes on all CI jobs.

Are there any user-facing changes?

Yes. The Arrow compute functions will be provided as a different library. Any user using Arrow Compute from C++ directly will require a call to arrow::compute::Initialize() in order for the functions and kernels to be registered

This PR includes breaking changes to public APIs.

• GitHub Issue: [C++] Split non-cast compute kernels into a separate shared library #25025

[raulcd]

@github-actions crossbow submit -g cpp

[raulcd]

It seems I am down to one job failure to the Windows R release job (CGLib Ruby is failing due to an existing issue):
I am unsure why I am getting missing symbols on dataset and acero when building statically. I've tried several things, around our *.pc.in and configure.win files to try and fix linking to the new arrow_compute but I am unsure where the problem is exactly coming.
@kou@nealrichardson@assignUser any idea if what can I try?

$ g++ -shared -s -static-libgcc -o arrow.dll tmp.def RTasks.o altrep.o array.o array_to_vector.o arraydata.o arrowExports.o bridge.o buffer.o chunkedarray.o compression.o compute-exec.o compute.o config.o csv.o dataset.o datatype.o expression.o extension-impl.o feather.o field.o filesystem.o io.o json.o memorypool.o message.o parquet.o r_to_arrow.o recordbatch.o recordbatchreader.o recordbatchwriter.o safe-call-into-r-impl.o scalar.o schema.o symbols.o table.o threadpool.o type_infer.o -L../windows/arrow-20.0.0.9000/lib-14.2.0/x64 -L../windows/arrow-20.0.0.9000/lib/x64-ucrt -larrow_dataset -larrow_acero -lparquet -larrow_compute -larrow -larrow_bundled_dependencies -lutf8proc -lsnappy -lz -lzstd -llz4 -lbz2 -lbrotlienc -lbrotlidec -lbrotlicommon -lole32 -lbcrypt -lpsapi -lcrypto -lcrypt32 -lre2 -luserenv -lversion -lws2_32 -lbcrypt -lwininet -lwinhttp -lsecur32 -lshlwapi -lncrypt -lcurl -lnormaliz -lssh2 -lgdi32 -lssl -lcrypto -lcrypt32 -lwldap32 -lz -lws2_32 -lnghttp2 -ldbghelp -LC:/rtools45/x86_64-w64-mingw32
C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_dataset.a(partition.cc.obj):(.text+0x7d9a): undefined reference to `__imp__ZN5arrow7compute7Grouper4MakeERKSt6vectorINS_10TypeHolderESaIS3_EEPNS0_11ExecContextE'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_dataset.a(partition.cc.obj):(.text+0x7f16): undefined reference to `__imp__ZN5arrow7compute7Grouper13MakeGroupingsERKNS_12NumericArrayINS_10UInt32TypeEEEjPNS0_11ExecContextE'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x1f2e): undefined reference to `__imp__ZN5arrow4util15TempVectorStack5allocEjPPhPi'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x1fb5): undefined reference to `__imp__ZN5arrow7compute9Hashing329HashBatchERKNS0_9ExecBatchEPjRSt6vectorINS0_14KeyColumnArrayESaIS7_EExPNS_4util15TempVectorStackExx'
C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x20b5): undefined reference to `__imp__ZN5arrow4util15TempVectorStack7releaseEij'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x2256): undefined reference to `__imp__ZN5arrow4util15TempVectorStack7releaseEij'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x7c05): undefined reference to `__imp__ZN5arrow4util15TempVectorStack4InitEPNS_10MemoryPoolEx'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text+0x7c6f): undefined reference to `__imp__ZN5arrow4util15TempVectorStackD1Ev'
C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text$_ZN5arrow5acero12HashJoinNodeD1Ev[_ZN5arrow5acero12HashJoinNodeD1Ev]+0x2f): undefined reference to `__imp__ZN5arrow4util15TempVectorStackD1Ev'C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-20.0.0.9000/lib/x64-ucrt/libarrow_acero.a(hash_join_node.cc.obj):(.text$_ZN5arrow5acero26BloomFilterPushdownContext17FilterSingleBatchEyPNS_7compute9ExecBatchE[_ZN5arrow5acero26BloomFilterPushdownContext17FilterSingleBatchEyPNS_7compute9ExecBatchE]+0x64f): undefined reference to `__imp__ZN5arrow7compute9Hashing329HashBatchERKNS0_9ExecBatchEPjRSt6vectorINS0_14KeyColumnArrayESaIS7_EExPNS_4util15TempVectorStackExx'

Still other improvements to be done but I wanted to have a clean CI PR to move to the cleaning and improvement stage.

[kou]

Acero and Dataset static libraries weren't built with -DARROW_COMPUTE_STATIC:

https://github.com/apache/arrow/actions/runs/14751558013/job/41410004318?pr=46261#step:7:14525

 [ 92%] Building CXX object src/arrow/acero/CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj
cd /D/a/arrow/arrow/src/build-x86_64-cpp/src/arrow/acero && /C/rtools40/ucrt64/bin/ccache.exe /C/rtools40/ucrt64/bin/c++.exe -DARROW_ACERO_EXPORTING -DARROW_ACERO_STATIC -DARROW_FLIGHT_SQL_STATIC -DARROW_FLIGHT_STATIC -DARROW_HAVE_RUNTIME_AVX2 -DARROW_HAVE_RUNTIME_BMI2 -DARROW_HAVE_RUNTIME_SSE4_2 -DARROW_HAVE_SSE4_2 -DARROW_STATIC -DARROW_WITH_TIMING_TESTS -DBOOST_ALL_NO_LIB -DUTF8PROC_STATIC -D_CRT_SECURE_NO_WARNINGS @CMakeFiles/arrow_acero_static.dir/includes_CXX.rsp -Wno-noexcept-type -DCURL_STATICLIB -fdiagnostics-color=always -Wa,-mbig-obj -Wall -fno-semantic-interposition -mxsave -msse4.2 -DUTF8PROC_STATIC -O3 -DNDEBUG -O2 -ftree-vectorize -std=c++17 -MD -MT src/arrow/acero/CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj -MF CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj.d -o CMakeFiles/arrow_acero_static.dir/groupby_aggregate_node.cc.obj -c /D/a/arrow/arrow/cpp/src/arrow/acero/groupby_aggregate_node.cc

Could you add the following

if(ARROW_BUILD_STATIC ANDWIN32)
target_compile_definitions(arrow_compute_staticPUBLICARROW_COMPUTE_STATIC)
endif()

like we did for Flight?

arrow/cpp/src/arrow/flight/CMakeLists.txt

Lines 222 to 224 in baf97fd

	if(ARROW_BUILD_STATIC ANDWIN32)
	target_compile_definitions(arrow_flight_staticPUBLICARROW_FLIGHT_STATIC)
	endif()

[raulcd]

@github-actions crossbow submit -g nightly-tests

[raulcd]

@github-actions crossbow submit -g nightly-tests

[raulcd]

@github-actions crossbow submit -g cpp

[github-actions]

Revision: 9001ec2

Submitted crossbow builds: ursacomputing/crossbow @ actions-74be4cb1e9

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-39-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

[pitrou]

LGTM in general, I posted two small suggestions.

[zanmato1984]

+1.

Excellent work @raulcd !

[rok]

I was only able to do a quick pass. Looks great, love the ARROW_COMPUTE_EXPORT macro.

[jonkeane]

Thank you for all of this, it's been a long journey!

[raulcd]

Thanks everyone for taking the time to review. If no more concerns are raised, it has already been approved by 4 committers, I plan to merge in a couple days.

[raulcd]

Thanks everyone, I am merging!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 0 benchmarking runs that have been run so far on merge-commit 6a5db61.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.