Uh oh!
There was an error while loading. Please reload this page.
GH-47632: [CI][C++] Add a CI job for JNI on Linux - #47746
Conversation
| @@ -89,7 +89,9 @@ RUN --mount=type=secret,id=github_repository_owner \ | |||
| --x-install-root=${VCPKG_ROOT}/installed \ | |||
| --x-manifest-root=/arrow/ci/vcpkg \ | |||
| --x-feature=azure \ | |||
| --x-feature=dev \ | |||
There was a problem hiding this comment.
Do we really need to enable this for Python wheels?
There was a problem hiding this comment.
Yes, to reuse this image for build configuration for JNI.
I think that reusing this image reduces maintenance cost but you don't want to increase this image size, right?
I'll create a new image for build configuration for JNI.
There was a problem hiding this comment.
My main concern is that it makes vcpkg build times even larger when the cache isn't fresh or there is no cache.
| elif [ -n "${CMAKE_PRESET}" ]; then | ||
| cmake \ | ||
| --preset="${CMAKE_PRESET}" \ | ||
| ${ARROW_CMAKE_ARGS} \ | ||
| ${source_dir} |
There was a problem hiding this comment.
We want to provide preset for JNI build to share build options in apache/arrow and apache/arrow-java.
There was a problem hiding this comment.
Why not pass --preset to the existing cmake call below?
There was a problem hiding this comment.
If we use it, all variables by --preset are overridden because it specify most variables explicitly. Explicit variables are used instead of variables set by --preset.
| if [ -x "$(command -v sudo)" ]; then | ||
| SUDO=sudo | ||
| else | ||
| if [ "$(id --user)" -eq 0 ]; then |
There was a problem hiding this comment.
We don't need to use sudo with the root user.
manylinux image doesn't provide /usr/bin/sudo but /opt/rh/gcc-toolset-12/root/usr/bin/sudo exists.
| if ! type storage-testbench >/dev/null 2>&1; then | ||
| exclude_tests+=("arrow-gcsfs-test") | ||
| fi | ||
| if ! type minio >/dev/null 2>&1; then | ||
| exclude_tests+=("arrow-s3fs-test") | ||
| fi |
There was a problem hiding this comment.
They don't exist in manylinux image.
| if [ "${ARROW_USE_MESON:-OFF}" = "OFF" ] && [ "${ARROW_EMSCRIPTEN:-OFF}" = "OFF" ]; then | ||
| CMAKE_PREFIX_PATH="${CMAKE_INSTALL_PREFIX:-${ARROW_HOME}}" | ||
| if [ -n "${VCPKG_ROOT}" ] && [ -n "${VCPKG_TARGET_TRIPLET}" ]; then | ||
| CMAKE_PREFIX_PATH+=";${VCPKG_ROOT}/installed/${VCPKG_TARGET_TRIPLET}" | ||
| fi | ||
| cmake \ | ||
| -S ${source_dir}/examples/minimal_build \ | ||
| -B ${build_dir}/examples/minimal_build \ | ||
| -DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" | ||
| cmake --build ${build_dir}/examples/minimal_build | ||
| pushd ${source_dir}/examples/minimal_build | ||
| ${build_dir}/examples/minimal_build/arrow-example | ||
| popd | ||
| fi |
There was a problem hiding this comment.
This is a test for our CMake packages. If find_package(Arrow) doesn't work, this will detect it.
JNI build uses find_package(Arrow).
| @@ -19,6 +19,7 @@ | |||
| "re2", | |||
| "snappy", | |||
| "utf8proc", | |||
| "xsimd", | |||
| @@ -176,7 +176,7 @@ class Process::Impl { | |||
| for (const auto& kv : process::environment::current()) { | |||
| env[kv.key()] = process::environment::value(kv.value()); | |||
| } | |||
| env["PATH"] = process::environment::value(current_exe.parent_path()); | |||
| env["PATH"] = process::environment::value(current_exe.parent_path().string()); | |||
There was a problem hiding this comment.
Recent Boost or old g++ (12.2.1) don't accept std::filesystem::path:
/arrow/cpp/src/arrow/testing/process.cc: In member function ‘arrow::Status arrow::util::Process::Impl::SetExecutable(const std::string&)’:
/arrow/cpp/src/arrow/testing/process.cc:179:74: error: call of overloaded ‘value(std::filesystem::__cxx11::path)’ is ambiguous
179 | env["PATH"] = process::environment::value(current_exe.parent_path());
| ^
In file included from /arrow/cpp/src/arrow/testing/process.cc:44:
/opt/vcpkg/installed/amd64-linux-static-release/include/boost/process/v2/environment.hpp:727:14: note: candidate: ‘boost::process::v2::environment::value::value(boost::process::v2::environment::value_view)’
727 | explicit value(value_view kv) : value_(kv.c_str()) {}
| ^~~~~
/opt/vcpkg/installed/amd64-linux-static-release/include/boost/process/v2/environment.hpp:723:5: note: candidate: ‘boost::process::v2::environment::value::value(string_type&&)’
723 | value( string_type&& source ) : value_(std::move(source)) {}
| ^~~~~
/opt/vcpkg/installed/amd64-linux-static-release/include/boost/process/v2/environment.hpp:722:5: note: candidate: ‘boost::process::v2::environment::value::value(const string_type&)’
722 | value( const string_type& source ) : value_(source) {}
| ^~~~~
We need to convert std::filesystem::path to std::string explicitly.
| permissions: | ||
| contents: read | ||
| packages: write |
There was a problem hiding this comment.
This is for using GitHub Packages as vcpkg cache.
There was a problem hiding this comment.
Done and moved to steps.jni-linux.permissions from the top-level permissions to reduce scope.
kou
commented
Oct 10, 2025
This is ready. This is a blocker of new Apache Arrow Java release. So this is needed for 22.0.0 C++ release. |
kou
commented
Oct 11, 2025
@raulcd Can we merge this for 22.0.0? |
raulcd
commented
Oct 12, 2025
Yes, I'll cherry-pick it on the maintenance branch once we merge it. |
kou
commented
Oct 12, 2025
Thanks. Let's merge this. |
After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 0c8ce5f. There were no benchmark performance regressions. 🎉 The full Conbench report has more details. It also includes information about 2 possible false positives for unstable benchmarks that are known to sometimes produce them. |
### Rationale for this change This is for preventing to break Apache Arrow Java JNI use case on Linux. ### What changes are included in this PR? * Add a CI job that uses build options for JNI use case * Install more packages in manylinux image that is also used by JNI build ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: #47632 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
kou
commented
Oct 13, 2025
This is also a blocker of 22.0.0. |
raulcd
commented
Oct 13, 2025
Thanks for pointing that out! |
### Rationale for this change This is for preventing to break Apache Arrow Java JNI use case on Linux. ### What changes are included in this PR? * Add a CI job that uses build options for JNI use case * Install more packages in manylinux image that is also used by JNI build ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: apache#47632 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
| # This is for testing find_package(Arrow). | ||
| # | ||
| # Note that this is not a perfect solution. We should improve this | ||
| # later. |
### Rationale for this change This is for preventing to break Apache Arrow Java JNI use case on Linux. ### What changes are included in this PR? * Add a CI job that uses build options for JNI use case * Install more packages in manylinux image that is also used by JNI build ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: apache#47632 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Rationale for this change
This is for preventing to break Apache Arrow Java JNI use case on Linux.
What changes are included in this PR?
Are these changes tested?
Yes.
Are there any user-facing changes?
No.