ARROW-17451: [CI][Java] Use manylinux2014 image for JNI#13920

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kou merged 6 commits into from
Aug 23, 2022
Merged

ARROW-17451: [CI][Java] Use manylinux2014 image for JNI #13920

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b319661
ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
kou Aug 18, 2022
36b05d1
Add missing libplasma_java
kou Aug 19, 2022
31a65eb
Show ccache statistics
kou Aug 19, 2022
3e7930d
Add missing ccache related configurations
kou Aug 20, 2022
64cf4c7
Fix path
kou Aug 20, 2022
489f84e
Fix cache key
kou Aug 20, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions .github/workflows/java_jni.yml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -47,7 +47,7 @@ env:
jobs:

docker:
name: AMD64 Debian 9 Java JNI (Gandiva, Plasma, ORC, Dataset)
name: AMD64 manylinux2014 Java JNI
runs-on: ubuntu-latest
if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
timeout-minutes: 90
Expand All@@ -61,23 +61,23 @@ jobs:
uses: actions/cache@v2
with:
path: .docker
key: maven-${{ hashFiles('java/**') }}
restore-keys: maven-
key: java-jni-manylinux-2014-${{ hashFiles('cpp/**', 'java/**') }}
restore-keys: java-jni-manylinux-2014-
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: 3.8
- name: Setup Archery
run: pip install -e dev/archery[docker]
- name: Execute Docker Build
run: archery docker run debian-java-jni
run: archery docker run java-jni-manylinux-2014
- name: Docker Push
if: success() && github.event_name == 'push' && github.repository == 'apache/arrow'
continue-on-error: true
run: archery docker push debian-java-jni
run: archery docker push java-jni-manylinux-2014

docker_integration_python:
name: AMD64 Debian 9 Java C Data Interface Integration
name: AMD64 Conda Java C Data Interface Integration
runs-on: ubuntu-latest
if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
timeout-minutes: 90
Expand Down
10 changes: 9 additions & 1 deletion ci/docker/java-jni-manylinux-201x.dockerfile
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -33,5 +33,13 @@ RUN vcpkg install \

# Install Java
ARG java=1.8.0
RUN yum install -y java-$java-openjdk-devel && yum clean all
RUN yum install -y java-$java-openjdk-devel rh-maven35 && yum clean all
ENV JAVA_HOME=/usr/lib/jvm/java-$java-openjdk/

# For ci/scripts/java_*.sh
ENV ARROW_GANDIVA_JAVA=ON \
ARROW_HOME=/tmp/local \
ARROW_JAVA_CDATA=ON \
ARROW_JNI=ON \
ARROW_PLASMA=ON \
ARROW_USE_CCACHE=ON
7 changes: 5 additions & 2 deletions ci/scripts/java_build.sh
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -18,10 +18,13 @@

set -ex

if [[ "${ARROW_JAVA_BUILD:-ON}" != "ON" ]]; then
exit
fi

arrow_dir=${1}
source_dir=${1}/java
build_dir=${2}
cpp_build_dir=${build_dir}/cpp/${ARROW_BUILD_TYPE:-debug}
java_jni_dist_dir=${3}

: ${BUILD_DOCS_JAVA:=OFF}
Expand DownExpand Up@@ -80,7 +83,7 @@ if [ "${ARROW_JAVA_CDATA}" = "ON" ]; then
fi

if [ "${ARROW_GANDIVA_JAVA}" = "ON" ]; then
${mvn} -Darrow.cpp.build.dir=${cpp_build_dir} -Parrow-jni install
${mvn} -Darrow.cpp.build.dir=${java_jni_dist_dir} -Parrow-jni install
fi

if [ "${ARROW_PLASMA}" = "ON" ]; then
Expand Down
25 changes: 20 additions & 5 deletions ci/scripts/java_jni_macos_build.sh
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -29,6 +29,7 @@ echo "=== Clear output directories and leftovers ==="
rm -rf ${build_dir}

echo "=== Building Arrow C++ libraries ==="
install_dir=${build_dir}/cpp-install
: ${ARROW_BUILD_TESTS:=OFF}
: ${ARROW_DATASET:=ON}
: ${ARROW_FILESYSTEM:=ON}
Expand All@@ -40,9 +41,15 @@ echo "=== Building Arrow C++ libraries ==="
: ${ARROW_PLASMA:=ON}
: ${ARROW_PYTHON:=OFF}
: ${ARROW_S3:=ON}
: ${ARROW_USE_CCACHE:=OFF}

@pitroupitrouAug 19, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not, but you should perhaps also pass it to CMake below?

@koukouAug 20, 2022

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I changed wrong file... I should have changed java_jni_manylinux_build.sh...

I've passed it to CMake arguments. I've updated java_jni_manylinux_build.sh too.

I chose ARROW_USE_CCACHE:=OFF here because cpp_build.sh uses it. I set ARROW_USE_CCACHE=ON in caller like other jobs.

: ${CMAKE_BUILD_TYPE:=Release}
: ${CMAKE_UNITY_BUILD:=ON}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
echo "=== ccache statistics before build ==="
ccache -s
fi

export ARROW_TEST_DATA="${arrow_dir}/testing/data"
export PARQUET_TEST_DATA="${arrow_dir}/cpp/submodules/parquet-testing/data"
export AWS_EC2_METADATA_DISABLED=TRUE
Expand DownExpand Up@@ -75,12 +82,13 @@ cmake \
-DARROW_S3=${ARROW_S3} \
-DARROW_SNAPPY_USE_SHARED=OFF \
-DARROW_THRIFT_USE_SHARED=OFF \
-DARROW_USE_CCACHE=${ARROW_USE_CCACHE} \
-DARROW_UTF8PROC_USE_SHARED=OFF \
-DARROW_ZSTD_USE_SHARED=OFF \
-DAWSSDK_SOURCE=BUNDLED \
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_INSTALL_PREFIX=${build_dir}/cpp \
-DCMAKE_INSTALL_PREFIX=${install_dir} \
-DCMAKE_UNITY_BUILD=${CMAKE_UNITY_BUILD} \
-DPARQUET_BUILD_EXAMPLES=OFF \
-DPARQUET_BUILD_EXECUTABLES=OFF \
Expand All@@ -102,12 +110,17 @@ ${arrow_dir}/ci/scripts/java_jni_build.sh \
${build_dir} \
${dist_dir}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
echo "=== ccache statistics after build ==="
ccache -s
fi

echo "=== Copying libraries to the distribution folder ==="
mkdir -p "${dist_dir}"
cp -L ${build_dir}/cpp/lib/libgandiva_jni.dylib ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_dataset_jni.dylib ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_orc_jni.dylib ${dist_dir}
cp -L ${install_dir}/lib/libarrow_dataset_jni.dylib ${dist_dir}
cp -L ${install_dir}/lib/libarrow_orc_jni.dylib ${dist_dir}
cp -L ${install_dir}/lib/libgandiva_jni.dylib ${dist_dir}
cp -L ${build_dir}/cpp/*/libplasma_java.dylib ${dist_dir}

echo "=== Checking shared dependencies for libraries ==="

Expand All@@ -122,9 +135,11 @@ archery linking check-dependencies \
--allow libcurl \
--allow libgandiva_jni \
--allow libncurses \
--allow libplasma_java \
--allow libz \
libarrow_cdata_jni.dylib \
libarrow_dataset_jni.dylib \
libarrow_orc_jni.dylib \
libgandiva_jni.dylib
libgandiva_jni.dylib \
libplasma_java.dylib
popd
26 changes: 20 additions & 6 deletions ci/scripts/java_jni_manylinux_build.sh
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -32,6 +32,7 @@ echo "=== Building Arrow C++ libraries ==="
devtoolset_version=$(rpm -qa "devtoolset-*-gcc" --queryformat %{VERSION} | \
grep -o "^[0-9]*")
devtoolset_include_cpp="/opt/rh/devtoolset-${devtoolset_version}/root/usr/include/c++/${devtoolset_version}"
: ${ARROW_BUILD_TESTS:=OFF}
: ${ARROW_DATASET:=ON}
: ${ARROW_GANDIVA:=ON}
: ${ARROW_GANDIVA_JAVA:=ON}
Expand All@@ -44,14 +45,19 @@ devtoolset_include_cpp="/opt/rh/devtoolset-${devtoolset_version}/root/usr/includ
: ${ARROW_PLASMA_JAVA_CLIENT:=ON}
: ${ARROW_PYTHON:=OFF}
: ${ARROW_S3:=ON}
: ${ARROW_BUILD_TESTS:=OFF}
: ${ARROW_USE_CCACHE:=OFF}
: ${CMAKE_BUILD_TYPE:=Release}
: ${CMAKE_UNITY_BUILD:=ON}
: ${VCPKG_ROOT:=/opt/vcpkg}
: ${VCPKG_FEATURE_FLAGS:=-manifests}
: ${VCPKG_TARGET_TRIPLET:=${VCPKG_DEFAULT_TRIPLET:-x64-linux-static-${CMAKE_BUILD_TYPE}}}
: ${GANDIVA_CXX_FLAGS:=-isystem;${devtoolset_include_cpp};-isystem;${devtoolset_include_cpp}/x86_64-redhat-linux;-isystem;-lpthread}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
echo "=== ccache statistics before build ==="
ccache -s
fi

export ARROW_TEST_DATA="${arrow_dir}/testing/data"
export PARQUET_TEST_DATA="${arrow_dir}/cpp/submodules/parquet-testing/data"
export AWS_EC2_METADATA_DISABLED=TRUE
Expand DownExpand Up@@ -87,11 +93,12 @@ cmake \
-DARROW_S3=${ARROW_S3} \
-DARROW_SNAPPY_USE_SHARED=OFF \
-DARROW_THRIFT_USE_SHARED=OFF \
-DARROW_USE_CCACHE=${ARROW_USE_CCACHE} \
-DARROW_UTF8PROC_USE_SHARED=OFF \
-DARROW_ZSTD_USE_SHARED=OFF \
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_INSTALL_PREFIX=${build_dir}/cpp \
-DCMAKE_INSTALL_PREFIX=${ARROW_HOME} \
-DCMAKE_UNITY_BUILD=${CMAKE_UNITY_BUILD} \
-DORC_SOURCE=BUNDLED \
-DORC_PROTOBUF_EXECUTABLE=${VCPKG_ROOT}/installed/${VCPKG_TARGET_TRIPLET}/tools/protobuf/protoc \
Expand DownExpand Up@@ -126,11 +133,17 @@ ${arrow_dir}/ci/scripts/java_jni_build.sh \
${build_dir} \
${dist_dir}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
echo "=== ccache statistics after build ==="
ccache -s
fi


echo "=== Copying libraries to the distribution folder ==="
cp -L ${build_dir}/cpp/lib/libgandiva_jni.so ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_dataset_jni.so ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_orc_jni.so ${dist_dir}
cp -L ${ARROW_HOME}/lib/libarrow_dataset_jni.so ${dist_dir}
cp -L ${ARROW_HOME}/lib/libarrow_orc_jni.so ${dist_dir}
cp -L ${ARROW_HOME}/lib/libgandiva_jni.so ${dist_dir}
cp -L ${build_dir}/cpp/*/libplasma_java.so ${dist_dir}

echo "=== Checking shared dependencies for libraries ==="

Expand All@@ -149,5 +162,6 @@ archery linking check-dependencies \
libarrow_cdata_jni.so \
libarrow_dataset_jni.so \
libarrow_orc_jni.so \
libgandiva_jni.so
libgandiva_jni.so \
libplasma_java.so
popd
9 changes: 6 additions & 3 deletions ci/scripts/java_test.sh
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -18,9 +18,12 @@

set -ex

if [[ "${ARROW_JAVA_TEST:-ON}" != "ON" ]]; then
exit
fi

arrow_dir=${1}
source_dir=${1}/java
cpp_build_dir=${2}/cpp/${ARROW_BUILD_TYPE:-debug}
java_jni_dist_dir=${3}

# For JNI and Plasma tests
Expand All@@ -36,7 +39,7 @@ pushd ${source_dir}
${mvn} test

if [ "${ARROW_JNI}" = "ON" ]; then
${mvn} test -Parrow-jni -pl adapter/orc,gandiva,dataset -Darrow.cpp.build.dir=${cpp_build_dir}
${mvn} test -Parrow-jni -pl adapter/orc,gandiva,dataset -Darrow.cpp.build.dir=${java_jni_dist_dir}
fi

if [ "${ARROW_JAVA_CDATA}" = "ON" ]; then
Expand All@@ -46,7 +49,7 @@ fi
if [ "${ARROW_PLASMA}" = "ON" ]; then
pushd ${source_dir}/plasma
java -cp target/test-classes:target/classes \
-Djava.library.path=${cpp_build_dir} \
-Djava.library.path=${java_jni_dist_dir} \
org.apache.arrow.plasma.PlasmaClientTest
popd
fi
Expand Down
10 changes: 5 additions & 5 deletions cpp/src/plasma/CMakeLists.txt
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -185,14 +185,14 @@ if(ARROW_PLASMA_JAVA_CLIENT)

if(APPLE)
target_link_libraries(plasma_java
plasma_shared
${PLASMA_LINK_LIBS}
plasma_static
${PLASMA_STATIC_LINK_LIBS}
"-undefined dynamic_lookup"
${PTHREAD_LIBRARY})
else(APPLE)
target_link_libraries(plasma_java plasma_shared ${PLASMA_LINK_LIBS}
else()
target_link_libraries(plasma_java plasma_static ${PLASMA_STATIC_LINK_LIBS}
${PTHREAD_LIBRARY})
endif(APPLE)
endif()
endif()
#
# Unit tests
Expand Down
11 changes: 10 additions & 1 deletion dev/tasks/java-jars/github.yml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -28,7 +28,11 @@ jobs:
{{ macros.github_checkout_arrow()|indent }}
{{ macros.github_install_archery()|indent }}
- name: Build C++ Libs
run: archery docker run java-jni-manylinux-2014
run: |
archery docker run \
-e ARROW_JAVA_BUILD=OFF \
-e ARROW_JAVA_TEST=OFF \
java-jni-manylinux-2014
- name: Compress into single artifact to keep directory structure
run: tar -cvzf arrow-shared-libs-linux.tar.gz arrow/java-dist/
- name: Upload Artifacts
Expand DownExpand Up@@ -61,6 +65,9 @@ jobs:
# aws-sdk-cpp and bundled aws-sdk-cpp. We uninstall Homebrew's
# aws-sdk-cpp to ensure using only bundled aws-sdk-cpp.
brew uninstall aws-sdk-cpp
- name: Setup ccache
run: |
arrow/ci/scripts/ccache_setup.sh
- name: Build C++ Libs
run: |
set -e
Expand DownExpand Up@@ -100,11 +107,13 @@ jobs:
test -f arrow/java-dist/libarrow_cdata_jni.dylib
test -f arrow/java-dist/libarrow_dataset_jni.dylib
test -f arrow/java-dist/libgandiva_jni.dylib
test -f arrow/java-dist/libplasma_java.dylib
test -f arrow/java-dist/libarrow_orc_jni.dylib
test -f arrow/java-dist/libarrow_cdata_jni.so
test -f arrow/java-dist/libarrow_dataset_jni.so
test -f arrow/java-dist/libarrow_orc_jni.so
test -f arrow/java-dist/libgandiva_jni.so
test -f arrow/java-dist/libplasma_java.so
- name: Build Bundled Jar
run: |
set -e
Expand Down
40 changes: 6 additions & 34 deletions docker-compose.yml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -122,7 +122,6 @@ x-hierarchy:
- debian-go-cgo
- debian-go-cgo-python
- debian-java
- debian-java-jni
- debian-js
- fedora-cpp:
- fedora-python
Expand DownExpand Up@@ -999,10 +998,14 @@ services:
<<: *ccache
volumes:
- .:/arrow:delegated
- ${DOCKER_VOLUME_PREFIX}maven-cache:/root/.m2:delegated
- ${DOCKER_VOLUME_PREFIX}python-wheel-manylinux2014-ccache:/ccache:delegated
command:
["pip install -e /arrow/dev/archery &&
/arrow/ci/scripts/java_jni_manylinux_build.sh /arrow /build /arrow/java-dist"]
["pip install -e /arrow/dev/archery && \
/arrow/ci/scripts/java_jni_manylinux_build.sh /arrow /build /arrow/java-dist && \
source /opt/rh/rh-maven35/enable && \
/arrow/ci/scripts/java_build.sh /arrow /build /arrow/java-dist && \
/arrow/ci/scripts/java_test.sh /arrow /build /arrow/java-dist"]

############################## Integration #################################

Expand DownExpand Up@@ -1541,37 +1544,6 @@ services:
/arrow/ci/scripts/java_build.sh /arrow /build &&
/arrow/ci/scripts/java_test.sh /arrow /build"

debian-java-jni:
# Includes plasma test, jni for gandiva and orc, and C data interface.
# Usage:
# docker-compose build debian-java
# docker-compose build debian-java-jni
# docker-compose run debian-java-jni
image: ${REPO}:${ARCH}-debian-9-java-${JDK}-maven-${MAVEN}-jni
build:
context: .
dockerfile: ci/docker/linux-apt-jni.dockerfile
cache_from:
- ${REPO}:${ARCH}-debian-9-java-${JDK}-maven-${MAVEN}-jni
args:
llvm: ${LLVM}
shm_size: *shm-size
environment:
<<: *ccache
ARROW_BUILD_TESTS: "OFF"
ARROW_S3: "OFF"
ARROW_SUBSTRAIT: "OFF"

@pitroupitrouAug 19, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you reuse these environment variables in the java-manylinux build?

@koukouAug 19, 2022•
edited
Loading

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use ARROW_BUILD_TESTS=OFF and ARROW_SUBSTRAIT=OFF and they are already used.
(But I want to enable ARROW_BUILD_TESTS in ARROW-17081 / #13911 to run a Dataset JNI test that uses arrow_testing.)

We can't use ARROW_S3=OFF because our JNI packages enable S3 recently: ARROW-16584 / #13157

volumes:
- .:/arrow:delegated
- ${DOCKER_VOLUME_PREFIX}maven-cache:/root/.m2:delegated
- ${DOCKER_VOLUME_PREFIX}debian-ccache:/ccache:delegated
command:
/bin/bash -c "
/arrow/ci/scripts/cpp_build.sh /arrow /build &&
/arrow/ci/scripts/java_jni_build.sh /arrow /build /tmp/java_dist &&
/arrow/ci/scripts/java_build.sh /arrow /build /tmp/java_dist &&
/arrow/ci/scripts/java_test.sh /arrow /build /tmp/java_dist"

oracle-java:
# Usage:
# docker-compose build oracle-java
Expand Down
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
ARROW-17451: [CI][Java] Use manylinux2014 image for JNI#13920

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

kou
merged 6 commits into

from

Aug 23, 2022
 Merged


ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
#13920
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
b319661

ARROW-17451: [CI][Java] Use manylinux2014 image for JNI


kou Aug 18, 2022
36b05d1

Add missing libplasma_java


kou Aug 19, 2022
31a65eb

Show ccache statistics


kou Aug 19, 2022
3e7930d

Add missing ccache related configurations


kou Aug 20, 2022
64cf4c7

Fix path


kou Aug 20, 2022
489f84e

Fix cache key


kou Aug 20, 2022

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


12 changes: 6 additions & 6 deletions
.github/workflows/java_jni.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -47,7 +47,7 @@ env:
jobs:

 docker:
 name: AMD64 Debian 9 Java JNI (Gandiva, Plasma, ORC, Dataset)
 name: AMD64 manylinux2014 Java JNI
 runs-on: ubuntu-latest
 if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
 timeout-minutes: 90
Expand All@@ -61,23 +61,23 @@ jobs:
 uses: actions/cache@v2
 with:
 path: .docker
 key: maven-${{ hashFiles('java/**') }}
 restore-keys: maven-
 key: java-jni-manylinux-2014-${{ hashFiles('cpp/**', 'java/**') }}
 restore-keys: java-jni-manylinux-2014-
 - name: Setup Python
 uses: actions/setup-python@v4
 with:
 python-version: 3.8
 - name: Setup Archery
 run: pip install -e dev/archery[docker]
 - name: Execute Docker Build
 run: archery docker run debian-java-jni
 run: archery docker run java-jni-manylinux-2014
 - name: Docker Push
 if: success() && github.event_name == 'push' && github.repository == 'apache/arrow'
 continue-on-error: true
 run: archery docker push debian-java-jni
 run: archery docker push java-jni-manylinux-2014

 docker_integration_python:
 name: AMD64 Debian 9 Java C Data Interface Integration
 name: AMD64 Conda Java C Data Interface Integration
 runs-on: ubuntu-latest
 if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
 timeout-minutes: 90
Expand Down

10 changes: 9 additions & 1 deletion
ci/docker/java-jni-manylinux-201x.dockerfile

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -33,5 +33,13 @@ RUN vcpkg install \

# Install Java
ARG java=1.8.0
RUN yum install -y java-$java-openjdk-devel && yum clean all
RUN yum install -y java-$java-openjdk-devel rh-maven35 && yum clean all
ENV JAVA_HOME=/usr/lib/jvm/java-$java-openjdk/

# For ci/scripts/java_*.sh
ENV ARROW_GANDIVA_JAVA=ON \
 ARROW_HOME=/tmp/local \
 ARROW_JAVA_CDATA=ON \
 ARROW_JNI=ON \
 ARROW_PLASMA=ON \
 ARROW_USE_CCACHE=ON

7 changes: 5 additions & 2 deletions
ci/scripts/java_build.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -18,10 +18,13 @@

set -ex

if [[ "${ARROW_JAVA_BUILD:-ON}" != "ON" ]]; then
 exit
fi

arrow_dir=${1}
source_dir=${1}/java
build_dir=${2}
cpp_build_dir=${build_dir}/cpp/${ARROW_BUILD_TYPE:-debug}
java_jni_dist_dir=${3}

: ${BUILD_DOCS_JAVA:=OFF}
Expand DownExpand Up@@ -80,7 +83,7 @@ if [ "${ARROW_JAVA_CDATA}" = "ON" ]; then
fi

if [ "${ARROW_GANDIVA_JAVA}" = "ON" ]; then
 ${mvn} -Darrow.cpp.build.dir=${cpp_build_dir} -Parrow-jni install
 ${mvn} -Darrow.cpp.build.dir=${java_jni_dist_dir} -Parrow-jni install
fi

if [ "${ARROW_PLASMA}" = "ON" ]; then
Expand Down

25 changes: 20 additions & 5 deletions
ci/scripts/java_jni_macos_build.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -29,6 +29,7 @@ echo "=== Clear output directories and leftovers ==="
rm -rf ${build_dir}

echo "=== Building Arrow C++ libraries ==="
install_dir=${build_dir}/cpp-install
: ${ARROW_BUILD_TESTS:=OFF}
: ${ARROW_DATASET:=ON}
: ${ARROW_FILESYSTEM:=ON}
Expand All@@ -40,9 +41,15 @@ echo "=== Building Arrow C++ libraries ==="
: ${ARROW_PLASMA:=ON}
: ${ARROW_PYTHON:=OFF}
: ${ARROW_S3:=ON}
: ${ARROW_USE_CCACHE:=OFF}
@pitroupitrouAug 19, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not, but you should perhaps also pass it to CMake below?
@koukouAug 20, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I changed wrong file... I should have changed java_jni_manylinux_build.sh...
I've passed it to CMake arguments. I've updated java_jni_manylinux_build.sh too.
I chose ARROW_USE_CCACHE:=OFF here because cpp_build.sh uses it. I set ARROW_USE_CCACHE=ON in caller like other jobs.
: ${CMAKE_BUILD_TYPE:=Release}
: ${CMAKE_UNITY_BUILD:=ON}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics before build ==="
 ccache -s
fi

export ARROW_TEST_DATA="${arrow_dir}/testing/data"
export PARQUET_TEST_DATA="${arrow_dir}/cpp/submodules/parquet-testing/data"
export AWS_EC2_METADATA_DISABLED=TRUE
Expand DownExpand Up@@ -75,12 +82,13 @@ cmake \
 -DARROW_S3=${ARROW_S3} \
 -DARROW_SNAPPY_USE_SHARED=OFF \
 -DARROW_THRIFT_USE_SHARED=OFF \
 -DARROW_USE_CCACHE=${ARROW_USE_CCACHE} \
 -DARROW_UTF8PROC_USE_SHARED=OFF \
 -DARROW_ZSTD_USE_SHARED=OFF \
 -DAWSSDK_SOURCE=BUNDLED \
 -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
 -DCMAKE_INSTALL_LIBDIR=lib \
 -DCMAKE_INSTALL_PREFIX=${build_dir}/cpp \
 -DCMAKE_INSTALL_PREFIX=${install_dir} \
 -DCMAKE_UNITY_BUILD=${CMAKE_UNITY_BUILD} \
 -DPARQUET_BUILD_EXAMPLES=OFF \
 -DPARQUET_BUILD_EXECUTABLES=OFF \
Expand All@@ -102,12 +110,17 @@ ${arrow_dir}/ci/scripts/java_jni_build.sh \
 ${build_dir} \
 ${dist_dir}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics after build ==="
 ccache -s
fi

echo "=== Copying libraries to the distribution folder ==="
mkdir -p "${dist_dir}"
cp -L ${build_dir}/cpp/lib/libgandiva_jni.dylib ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_dataset_jni.dylib ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_orc_jni.dylib ${dist_dir}
cp -L ${install_dir}/lib/libarrow_dataset_jni.dylib ${dist_dir}
cp -L ${install_dir}/lib/libarrow_orc_jni.dylib ${dist_dir}
cp -L ${install_dir}/lib/libgandiva_jni.dylib ${dist_dir}
cp -L ${build_dir}/cpp/*/libplasma_java.dylib ${dist_dir}

echo "=== Checking shared dependencies for libraries ==="

Expand All@@ -122,9 +135,11 @@ archery linking check-dependencies \
 --allow libcurl \
 --allow libgandiva_jni \
 --allow libncurses \
 --allow libplasma_java \
 --allow libz \
 libarrow_cdata_jni.dylib \
 libarrow_dataset_jni.dylib \
 libarrow_orc_jni.dylib \
 libgandiva_jni.dylib
 libgandiva_jni.dylib \
 libplasma_java.dylib
popd

26 changes: 20 additions & 6 deletions
ci/scripts/java_jni_manylinux_build.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -32,6 +32,7 @@ echo "=== Building Arrow C++ libraries ==="
devtoolset_version=$(rpm -qa "devtoolset-*-gcc" --queryformat %{VERSION} | \
 grep -o "^[0-9]*")
devtoolset_include_cpp="/opt/rh/devtoolset-${devtoolset_version}/root/usr/include/c++/${devtoolset_version}"
: ${ARROW_BUILD_TESTS:=OFF}
: ${ARROW_DATASET:=ON}
: ${ARROW_GANDIVA:=ON}
: ${ARROW_GANDIVA_JAVA:=ON}
Expand All@@ -44,14 +45,19 @@ devtoolset_include_cpp="/opt/rh/devtoolset-${devtoolset_version}/root/usr/includ
: ${ARROW_PLASMA_JAVA_CLIENT:=ON}
: ${ARROW_PYTHON:=OFF}
: ${ARROW_S3:=ON}
: ${ARROW_BUILD_TESTS:=OFF}
: ${ARROW_USE_CCACHE:=OFF}
: ${CMAKE_BUILD_TYPE:=Release}
: ${CMAKE_UNITY_BUILD:=ON}
: ${VCPKG_ROOT:=/opt/vcpkg}
: ${VCPKG_FEATURE_FLAGS:=-manifests}
: ${VCPKG_TARGET_TRIPLET:=${VCPKG_DEFAULT_TRIPLET:-x64-linux-static-${CMAKE_BUILD_TYPE}}}
: ${GANDIVA_CXX_FLAGS:=-isystem;${devtoolset_include_cpp};-isystem;${devtoolset_include_cpp}/x86_64-redhat-linux;-isystem;-lpthread}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics before build ==="
 ccache -s
fi

export ARROW_TEST_DATA="${arrow_dir}/testing/data"
export PARQUET_TEST_DATA="${arrow_dir}/cpp/submodules/parquet-testing/data"
export AWS_EC2_METADATA_DISABLED=TRUE
Expand DownExpand Up@@ -87,11 +93,12 @@ cmake \
 -DARROW_S3=${ARROW_S3} \
 -DARROW_SNAPPY_USE_SHARED=OFF \
 -DARROW_THRIFT_USE_SHARED=OFF \
 -DARROW_USE_CCACHE=${ARROW_USE_CCACHE} \
 -DARROW_UTF8PROC_USE_SHARED=OFF \
 -DARROW_ZSTD_USE_SHARED=OFF \
 -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
 -DCMAKE_INSTALL_LIBDIR=lib \
 -DCMAKE_INSTALL_PREFIX=${build_dir}/cpp \
 -DCMAKE_INSTALL_PREFIX=${ARROW_HOME} \
 -DCMAKE_UNITY_BUILD=${CMAKE_UNITY_BUILD} \
 -DORC_SOURCE=BUNDLED \
 -DORC_PROTOBUF_EXECUTABLE=${VCPKG_ROOT}/installed/${VCPKG_TARGET_TRIPLET}/tools/protobuf/protoc \
Expand DownExpand Up@@ -126,11 +133,17 @@ ${arrow_dir}/ci/scripts/java_jni_build.sh \
 ${build_dir} \
 ${dist_dir}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics after build ==="
 ccache -s
fi


echo "=== Copying libraries to the distribution folder ==="
cp -L ${build_dir}/cpp/lib/libgandiva_jni.so ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_dataset_jni.so ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_orc_jni.so ${dist_dir}
cp -L ${ARROW_HOME}/lib/libarrow_dataset_jni.so ${dist_dir}
cp -L ${ARROW_HOME}/lib/libarrow_orc_jni.so ${dist_dir}
cp -L ${ARROW_HOME}/lib/libgandiva_jni.so ${dist_dir}
cp -L ${build_dir}/cpp/*/libplasma_java.so ${dist_dir}

echo "=== Checking shared dependencies for libraries ==="

Expand All@@ -149,5 +162,6 @@ archery linking check-dependencies \
 libarrow_cdata_jni.so \
 libarrow_dataset_jni.so \
 libarrow_orc_jni.so \
 libgandiva_jni.so
 libgandiva_jni.so \
 libplasma_java.so
popd

9 changes: 6 additions & 3 deletions
ci/scripts/java_test.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -18,9 +18,12 @@

set -ex

if [[ "${ARROW_JAVA_TEST:-ON}" != "ON" ]]; then
 exit
fi

arrow_dir=${1}
source_dir=${1}/java
cpp_build_dir=${2}/cpp/${ARROW_BUILD_TYPE:-debug}
java_jni_dist_dir=${3}

# For JNI and Plasma tests
Expand All@@ -36,7 +39,7 @@ pushd ${source_dir}
${mvn} test

if [ "${ARROW_JNI}" = "ON" ]; then
 ${mvn} test -Parrow-jni -pl adapter/orc,gandiva,dataset -Darrow.cpp.build.dir=${cpp_build_dir}
 ${mvn} test -Parrow-jni -pl adapter/orc,gandiva,dataset -Darrow.cpp.build.dir=${java_jni_dist_dir}
fi

if [ "${ARROW_JAVA_CDATA}" = "ON" ]; then
Expand All@@ -46,7 +49,7 @@ fi
if [ "${ARROW_PLASMA}" = "ON" ]; then
 pushd ${source_dir}/plasma
 java -cp target/test-classes:target/classes \
 -Djava.library.path=${cpp_build_dir} \
 -Djava.library.path=${java_jni_dist_dir} \
 org.apache.arrow.plasma.PlasmaClientTest
 popd
fi
Expand Down

10 changes: 5 additions & 5 deletions
cpp/src/plasma/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -185,14 +185,14 @@ if(ARROW_PLASMA_JAVA_CLIENT)

 if(APPLE)
 target_link_libraries(plasma_java
plasma_shared
 ${PLASMA_LINK_LIBS}
plasma_static
 ${PLASMA_STATIC_LINK_LIBS}
 "-undefined dynamic_lookup"
 ${PTHREAD_LIBRARY})
 else(APPLE)
 target_link_libraries(plasma_java plasma_shared ${PLASMA_LINK_LIBS}
 else()
 target_link_libraries(plasma_java plasma_static ${PLASMA_STATIC_LINK_LIBS}
 ${PTHREAD_LIBRARY})
 endif(APPLE)
 endif()
endif()
#
# Unit tests
Expand Down

11 changes: 10 additions & 1 deletion
dev/tasks/java-jars/github.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -28,7 +28,11 @@ jobs:
 {{ macros.github_checkout_arrow()|indent }}
 {{ macros.github_install_archery()|indent }}
 - name: Build C++ Libs
 run: archery docker run java-jni-manylinux-2014
 run: |
 archery docker run \
 -e ARROW_JAVA_BUILD=OFF \
 -e ARROW_JAVA_TEST=OFF \
 java-jni-manylinux-2014
 - name: Compress into single artifact to keep directory structure
 run: tar -cvzf arrow-shared-libs-linux.tar.gz arrow/java-dist/
 - name: Upload Artifacts
Expand DownExpand Up@@ -61,6 +65,9 @@ jobs:
 # aws-sdk-cpp and bundled aws-sdk-cpp. We uninstall Homebrew's
 # aws-sdk-cpp to ensure using only bundled aws-sdk-cpp.
 brew uninstall aws-sdk-cpp
 - name: Setup ccache
 run: |
 arrow/ci/scripts/ccache_setup.sh
 - name: Build C++ Libs
 run: |
 set -e
Expand DownExpand Up@@ -100,11 +107,13 @@ jobs:
 test -f arrow/java-dist/libarrow_cdata_jni.dylib
 test -f arrow/java-dist/libarrow_dataset_jni.dylib
 test -f arrow/java-dist/libgandiva_jni.dylib
 test -f arrow/java-dist/libplasma_java.dylib
 test -f arrow/java-dist/libarrow_orc_jni.dylib
 test -f arrow/java-dist/libarrow_cdata_jni.so
 test -f arrow/java-dist/libarrow_dataset_jni.so
 test -f arrow/java-dist/libarrow_orc_jni.so
 test -f arrow/java-dist/libgandiva_jni.so
 test -f arrow/java-dist/libplasma_java.so
 - name: Build Bundled Jar
 run: |
 set -e
Expand Down

40 changes: 6 additions & 34 deletions
docker-compose.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -122,7 +122,6 @@ x-hierarchy:
 - debian-go-cgo
 - debian-go-cgo-python
 - debian-java
 - debian-java-jni
 - debian-js
 - fedora-cpp:
 - fedora-python
Expand DownExpand Up@@ -999,10 +998,14 @@ services:
 <<: *ccache
 volumes:
 - .:/arrow:delegated
 - ${DOCKER_VOLUME_PREFIX}maven-cache:/root/.m2:delegated
 - ${DOCKER_VOLUME_PREFIX}python-wheel-manylinux2014-ccache:/ccache:delegated
 command:
 ["pip install -e /arrow/dev/archery &&
 /arrow/ci/scripts/java_jni_manylinux_build.sh /arrow /build /arrow/java-dist"]
 ["pip install -e /arrow/dev/archery && \
 /arrow/ci/scripts/java_jni_manylinux_build.sh /arrow /build /arrow/java-dist && \
 source /opt/rh/rh-maven35/enable && \
 /arrow/ci/scripts/java_build.sh /arrow /build /arrow/java-dist && \
 /arrow/ci/scripts/java_test.sh /arrow /build /arrow/java-dist"]

 ############################## Integration #################################

Expand DownExpand Up@@ -1541,37 +1544,6 @@ services:
 /arrow/ci/scripts/java_build.sh /arrow /build &&
 /arrow/ci/scripts/java_test.sh /arrow /build"

 debian-java-jni:
 # Includes plasma test, jni for gandiva and orc, and C data interface.
 # Usage:
 # docker-compose build debian-java
 # docker-compose build debian-java-jni
 # docker-compose run debian-java-jni
 image: ${REPO}:${ARCH}-debian-9-java-${JDK}-maven-${MAVEN}-jni
 build:
 context: .
 dockerfile: ci/docker/linux-apt-jni.dockerfile
 cache_from:
 - ${REPO}:${ARCH}-debian-9-java-${JDK}-maven-${MAVEN}-jni
 args:
 llvm: ${LLVM}
 shm_size: *shm-size
 environment:
 <<: *ccache
 ARROW_BUILD_TESTS: "OFF"
 ARROW_S3: "OFF"
 ARROW_SUBSTRAIT: "OFF"
@pitroupitrouAug 19, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you reuse these environment variables in the java-manylinux build?
@koukouAug 19, 2022•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use ARROW_BUILD_TESTS=OFF and ARROW_SUBSTRAIT=OFF and they are already used.

(But I want to enable ARROW_BUILD_TESTS in ARROW-17081 / #13911 to run a Dataset JNI test that uses arrow_testing.)
We can't use ARROW_S3=OFF because our JNI packages enable S3 recently: ARROW-16584 / #13157
 volumes:
 - .:/arrow:delegated
 - ${DOCKER_VOLUME_PREFIX}maven-cache:/root/.m2:delegated
 - ${DOCKER_VOLUME_PREFIX}debian-ccache:/ccache:delegated
 command:
 /bin/bash -c "
 /arrow/ci/scripts/cpp_build.sh /arrow /build &&
 /arrow/ci/scripts/java_jni_build.sh /arrow /build /tmp/java_dist &&
 /arrow/ci/scripts/java_build.sh /arrow /build /tmp/java_dist &&
 /arrow/ci/scripts/java_test.sh /arrow /build /tmp/java_dist"

 oracle-java:
 # Usage:
 # docker-compose build oracle-java
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-17451: [CI][Java] Use manylinux2014 image for JNI#13920

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

kou
merged 6 commits into

from

Aug 23, 2022
 Merged


ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
#13920
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
b319661

ARROW-17451: [CI][Java] Use manylinux2014 image for JNI


kou Aug 18, 2022
36b05d1

Add missing libplasma_java


kou Aug 19, 2022
31a65eb

Show ccache statistics


kou Aug 19, 2022
3e7930d

Add missing ccache related configurations


kou Aug 20, 2022
64cf4c7

Fix path


kou Aug 20, 2022
489f84e

Fix cache key


kou Aug 20, 2022

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


12 changes: 6 additions & 6 deletions
.github/workflows/java_jni.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -47,7 +47,7 @@ env:
jobs:

 docker:
 name: AMD64 Debian 9 Java JNI (Gandiva, Plasma, ORC, Dataset)
 name: AMD64 manylinux2014 Java JNI
 runs-on: ubuntu-latest
 if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
 timeout-minutes: 90
Expand All@@ -61,23 +61,23 @@ jobs:
 uses: actions/cache@v2
 with:
 path: .docker
 key: maven-${{ hashFiles('java/**') }}
 restore-keys: maven-
 key: java-jni-manylinux-2014-${{ hashFiles('cpp/**', 'java/**') }}
 restore-keys: java-jni-manylinux-2014-
 - name: Setup Python
 uses: actions/setup-python@v4
 with:
 python-version: 3.8
 - name: Setup Archery
 run: pip install -e dev/archery[docker]
 - name: Execute Docker Build
 run: archery docker run debian-java-jni
 run: archery docker run java-jni-manylinux-2014
 - name: Docker Push
 if: success() && github.event_name == 'push' && github.repository == 'apache/arrow'
 continue-on-error: true
 run: archery docker push debian-java-jni
 run: archery docker push java-jni-manylinux-2014

 docker_integration_python:
 name: AMD64 Debian 9 Java C Data Interface Integration
 name: AMD64 Conda Java C Data Interface Integration
 runs-on: ubuntu-latest
 if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
 timeout-minutes: 90
Expand Down

10 changes: 9 additions & 1 deletion
ci/docker/java-jni-manylinux-201x.dockerfile

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -33,5 +33,13 @@ RUN vcpkg install \

# Install Java
ARG java=1.8.0
RUN yum install -y java-$java-openjdk-devel && yum clean all
RUN yum install -y java-$java-openjdk-devel rh-maven35 && yum clean all
ENV JAVA_HOME=/usr/lib/jvm/java-$java-openjdk/

# For ci/scripts/java_*.sh
ENV ARROW_GANDIVA_JAVA=ON \
 ARROW_HOME=/tmp/local \
 ARROW_JAVA_CDATA=ON \
 ARROW_JNI=ON \
 ARROW_PLASMA=ON \
 ARROW_USE_CCACHE=ON

7 changes: 5 additions & 2 deletions
ci/scripts/java_build.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -18,10 +18,13 @@

set -ex

if [[ "${ARROW_JAVA_BUILD:-ON}" != "ON" ]]; then
 exit
fi

arrow_dir=${1}
source_dir=${1}/java
build_dir=${2}
cpp_build_dir=${build_dir}/cpp/${ARROW_BUILD_TYPE:-debug}
java_jni_dist_dir=${3}

: ${BUILD_DOCS_JAVA:=OFF}
Expand DownExpand Up@@ -80,7 +83,7 @@ if [ "${ARROW_JAVA_CDATA}" = "ON" ]; then
fi

if [ "${ARROW_GANDIVA_JAVA}" = "ON" ]; then
 ${mvn} -Darrow.cpp.build.dir=${cpp_build_dir} -Parrow-jni install
 ${mvn} -Darrow.cpp.build.dir=${java_jni_dist_dir} -Parrow-jni install
fi

if [ "${ARROW_PLASMA}" = "ON" ]; then
Expand Down

25 changes: 20 additions & 5 deletions
ci/scripts/java_jni_macos_build.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -29,6 +29,7 @@ echo "=== Clear output directories and leftovers ==="
rm -rf ${build_dir}

echo "=== Building Arrow C++ libraries ==="
install_dir=${build_dir}/cpp-install
: ${ARROW_BUILD_TESTS:=OFF}
: ${ARROW_DATASET:=ON}
: ${ARROW_FILESYSTEM:=ON}
Expand All@@ -40,9 +41,15 @@ echo "=== Building Arrow C++ libraries ==="
: ${ARROW_PLASMA:=ON}
: ${ARROW_PYTHON:=OFF}
: ${ARROW_S3:=ON}
: ${ARROW_USE_CCACHE:=OFF}
@pitroupitrouAug 19, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not, but you should perhaps also pass it to CMake below?
@koukouAug 20, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I changed wrong file... I should have changed java_jni_manylinux_build.sh...
I've passed it to CMake arguments. I've updated java_jni_manylinux_build.sh too.
I chose ARROW_USE_CCACHE:=OFF here because cpp_build.sh uses it. I set ARROW_USE_CCACHE=ON in caller like other jobs.
: ${CMAKE_BUILD_TYPE:=Release}
: ${CMAKE_UNITY_BUILD:=ON}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics before build ==="
 ccache -s
fi

export ARROW_TEST_DATA="${arrow_dir}/testing/data"
export PARQUET_TEST_DATA="${arrow_dir}/cpp/submodules/parquet-testing/data"
export AWS_EC2_METADATA_DISABLED=TRUE
Expand DownExpand Up@@ -75,12 +82,13 @@ cmake \
 -DARROW_S3=${ARROW_S3} \
 -DARROW_SNAPPY_USE_SHARED=OFF \
 -DARROW_THRIFT_USE_SHARED=OFF \
 -DARROW_USE_CCACHE=${ARROW_USE_CCACHE} \
 -DARROW_UTF8PROC_USE_SHARED=OFF \
 -DARROW_ZSTD_USE_SHARED=OFF \
 -DAWSSDK_SOURCE=BUNDLED \
 -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
 -DCMAKE_INSTALL_LIBDIR=lib \
 -DCMAKE_INSTALL_PREFIX=${build_dir}/cpp \
 -DCMAKE_INSTALL_PREFIX=${install_dir} \
 -DCMAKE_UNITY_BUILD=${CMAKE_UNITY_BUILD} \
 -DPARQUET_BUILD_EXAMPLES=OFF \
 -DPARQUET_BUILD_EXECUTABLES=OFF \
Expand All@@ -102,12 +110,17 @@ ${arrow_dir}/ci/scripts/java_jni_build.sh \
 ${build_dir} \
 ${dist_dir}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics after build ==="
 ccache -s
fi

echo "=== Copying libraries to the distribution folder ==="
mkdir -p "${dist_dir}"
cp -L ${build_dir}/cpp/lib/libgandiva_jni.dylib ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_dataset_jni.dylib ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_orc_jni.dylib ${dist_dir}
cp -L ${install_dir}/lib/libarrow_dataset_jni.dylib ${dist_dir}
cp -L ${install_dir}/lib/libarrow_orc_jni.dylib ${dist_dir}
cp -L ${install_dir}/lib/libgandiva_jni.dylib ${dist_dir}
cp -L ${build_dir}/cpp/*/libplasma_java.dylib ${dist_dir}

echo "=== Checking shared dependencies for libraries ==="

Expand All@@ -122,9 +135,11 @@ archery linking check-dependencies \
 --allow libcurl \
 --allow libgandiva_jni \
 --allow libncurses \
 --allow libplasma_java \
 --allow libz \
 libarrow_cdata_jni.dylib \
 libarrow_dataset_jni.dylib \
 libarrow_orc_jni.dylib \
 libgandiva_jni.dylib
 libgandiva_jni.dylib \
 libplasma_java.dylib
popd

26 changes: 20 additions & 6 deletions
ci/scripts/java_jni_manylinux_build.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -32,6 +32,7 @@ echo "=== Building Arrow C++ libraries ==="
devtoolset_version=$(rpm -qa "devtoolset-*-gcc" --queryformat %{VERSION} | \
 grep -o "^[0-9]*")
devtoolset_include_cpp="/opt/rh/devtoolset-${devtoolset_version}/root/usr/include/c++/${devtoolset_version}"
: ${ARROW_BUILD_TESTS:=OFF}
: ${ARROW_DATASET:=ON}
: ${ARROW_GANDIVA:=ON}
: ${ARROW_GANDIVA_JAVA:=ON}
Expand All@@ -44,14 +45,19 @@ devtoolset_include_cpp="/opt/rh/devtoolset-${devtoolset_version}/root/usr/includ
: ${ARROW_PLASMA_JAVA_CLIENT:=ON}
: ${ARROW_PYTHON:=OFF}
: ${ARROW_S3:=ON}
: ${ARROW_BUILD_TESTS:=OFF}
: ${ARROW_USE_CCACHE:=OFF}
: ${CMAKE_BUILD_TYPE:=Release}
: ${CMAKE_UNITY_BUILD:=ON}
: ${VCPKG_ROOT:=/opt/vcpkg}
: ${VCPKG_FEATURE_FLAGS:=-manifests}
: ${VCPKG_TARGET_TRIPLET:=${VCPKG_DEFAULT_TRIPLET:-x64-linux-static-${CMAKE_BUILD_TYPE}}}
: ${GANDIVA_CXX_FLAGS:=-isystem;${devtoolset_include_cpp};-isystem;${devtoolset_include_cpp}/x86_64-redhat-linux;-isystem;-lpthread}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics before build ==="
 ccache -s
fi

export ARROW_TEST_DATA="${arrow_dir}/testing/data"
export PARQUET_TEST_DATA="${arrow_dir}/cpp/submodules/parquet-testing/data"
export AWS_EC2_METADATA_DISABLED=TRUE
Expand DownExpand Up@@ -87,11 +93,12 @@ cmake \
 -DARROW_S3=${ARROW_S3} \
 -DARROW_SNAPPY_USE_SHARED=OFF \
 -DARROW_THRIFT_USE_SHARED=OFF \
 -DARROW_USE_CCACHE=${ARROW_USE_CCACHE} \
 -DARROW_UTF8PROC_USE_SHARED=OFF \
 -DARROW_ZSTD_USE_SHARED=OFF \
 -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
 -DCMAKE_INSTALL_LIBDIR=lib \
 -DCMAKE_INSTALL_PREFIX=${build_dir}/cpp \
 -DCMAKE_INSTALL_PREFIX=${ARROW_HOME} \
 -DCMAKE_UNITY_BUILD=${CMAKE_UNITY_BUILD} \
 -DORC_SOURCE=BUNDLED \
 -DORC_PROTOBUF_EXECUTABLE=${VCPKG_ROOT}/installed/${VCPKG_TARGET_TRIPLET}/tools/protobuf/protoc \
Expand DownExpand Up@@ -126,11 +133,17 @@ ${arrow_dir}/ci/scripts/java_jni_build.sh \
 ${build_dir} \
 ${dist_dir}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics after build ==="
 ccache -s
fi


echo "=== Copying libraries to the distribution folder ==="
cp -L ${build_dir}/cpp/lib/libgandiva_jni.so ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_dataset_jni.so ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_orc_jni.so ${dist_dir}
cp -L ${ARROW_HOME}/lib/libarrow_dataset_jni.so ${dist_dir}
cp -L ${ARROW_HOME}/lib/libarrow_orc_jni.so ${dist_dir}
cp -L ${ARROW_HOME}/lib/libgandiva_jni.so ${dist_dir}
cp -L ${build_dir}/cpp/*/libplasma_java.so ${dist_dir}

echo "=== Checking shared dependencies for libraries ==="

Expand All@@ -149,5 +162,6 @@ archery linking check-dependencies \
 libarrow_cdata_jni.so \
 libarrow_dataset_jni.so \
 libarrow_orc_jni.so \
 libgandiva_jni.so
 libgandiva_jni.so \
 libplasma_java.so
popd

9 changes: 6 additions & 3 deletions
ci/scripts/java_test.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -18,9 +18,12 @@

set -ex

if [[ "${ARROW_JAVA_TEST:-ON}" != "ON" ]]; then
 exit
fi

arrow_dir=${1}
source_dir=${1}/java
cpp_build_dir=${2}/cpp/${ARROW_BUILD_TYPE:-debug}
java_jni_dist_dir=${3}

# For JNI and Plasma tests
Expand All@@ -36,7 +39,7 @@ pushd ${source_dir}
${mvn} test

if [ "${ARROW_JNI}" = "ON" ]; then
 ${mvn} test -Parrow-jni -pl adapter/orc,gandiva,dataset -Darrow.cpp.build.dir=${cpp_build_dir}
 ${mvn} test -Parrow-jni -pl adapter/orc,gandiva,dataset -Darrow.cpp.build.dir=${java_jni_dist_dir}
fi

if [ "${ARROW_JAVA_CDATA}" = "ON" ]; then
Expand All@@ -46,7 +49,7 @@ fi
if [ "${ARROW_PLASMA}" = "ON" ]; then
 pushd ${source_dir}/plasma
 java -cp target/test-classes:target/classes \
 -Djava.library.path=${cpp_build_dir} \
 -Djava.library.path=${java_jni_dist_dir} \
 org.apache.arrow.plasma.PlasmaClientTest
 popd
fi
Expand Down

10 changes: 5 additions & 5 deletions
cpp/src/plasma/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -185,14 +185,14 @@ if(ARROW_PLASMA_JAVA_CLIENT)

 if(APPLE)
 target_link_libraries(plasma_java
plasma_shared
 ${PLASMA_LINK_LIBS}
plasma_static
 ${PLASMA_STATIC_LINK_LIBS}
 "-undefined dynamic_lookup"
 ${PTHREAD_LIBRARY})
 else(APPLE)
 target_link_libraries(plasma_java plasma_shared ${PLASMA_LINK_LIBS}
 else()
 target_link_libraries(plasma_java plasma_static ${PLASMA_STATIC_LINK_LIBS}
 ${PTHREAD_LIBRARY})
 endif(APPLE)
 endif()
endif()
#
# Unit tests
Expand Down

11 changes: 10 additions & 1 deletion
dev/tasks/java-jars/github.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -28,7 +28,11 @@ jobs:
 {{ macros.github_checkout_arrow()|indent }}
 {{ macros.github_install_archery()|indent }}
 - name: Build C++ Libs
 run: archery docker run java-jni-manylinux-2014
 run: |
 archery docker run \
 -e ARROW_JAVA_BUILD=OFF \
 -e ARROW_JAVA_TEST=OFF \
 java-jni-manylinux-2014
 - name: Compress into single artifact to keep directory structure
 run: tar -cvzf arrow-shared-libs-linux.tar.gz arrow/java-dist/
 - name: Upload Artifacts
Expand DownExpand Up@@ -61,6 +65,9 @@ jobs:
 # aws-sdk-cpp and bundled aws-sdk-cpp. We uninstall Homebrew's
 # aws-sdk-cpp to ensure using only bundled aws-sdk-cpp.
 brew uninstall aws-sdk-cpp
 - name: Setup ccache
 run: |
 arrow/ci/scripts/ccache_setup.sh
 - name: Build C++ Libs
 run: |
 set -e
Expand DownExpand Up@@ -100,11 +107,13 @@ jobs:
 test -f arrow/java-dist/libarrow_cdata_jni.dylib
 test -f arrow/java-dist/libarrow_dataset_jni.dylib
 test -f arrow/java-dist/libgandiva_jni.dylib
 test -f arrow/java-dist/libplasma_java.dylib
 test -f arrow/java-dist/libarrow_orc_jni.dylib
 test -f arrow/java-dist/libarrow_cdata_jni.so
 test -f arrow/java-dist/libarrow_dataset_jni.so
 test -f arrow/java-dist/libarrow_orc_jni.so
 test -f arrow/java-dist/libgandiva_jni.so
 test -f arrow/java-dist/libplasma_java.so
 - name: Build Bundled Jar
 run: |
 set -e
Expand Down

40 changes: 6 additions & 34 deletions
docker-compose.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -122,7 +122,6 @@ x-hierarchy:
 - debian-go-cgo
 - debian-go-cgo-python
 - debian-java
 - debian-java-jni
 - debian-js
 - fedora-cpp:
 - fedora-python
Expand DownExpand Up@@ -999,10 +998,14 @@ services:
 <<: *ccache
 volumes:
 - .:/arrow:delegated
 - ${DOCKER_VOLUME_PREFIX}maven-cache:/root/.m2:delegated
 - ${DOCKER_VOLUME_PREFIX}python-wheel-manylinux2014-ccache:/ccache:delegated
 command:
 ["pip install -e /arrow/dev/archery &&
 /arrow/ci/scripts/java_jni_manylinux_build.sh /arrow /build /arrow/java-dist"]
 ["pip install -e /arrow/dev/archery && \
 /arrow/ci/scripts/java_jni_manylinux_build.sh /arrow /build /arrow/java-dist && \
 source /opt/rh/rh-maven35/enable && \
 /arrow/ci/scripts/java_build.sh /arrow /build /arrow/java-dist && \
 /arrow/ci/scripts/java_test.sh /arrow /build /arrow/java-dist"]

 ############################## Integration #################################

Expand DownExpand Up@@ -1541,37 +1544,6 @@ services:
 /arrow/ci/scripts/java_build.sh /arrow /build &&
 /arrow/ci/scripts/java_test.sh /arrow /build"

 debian-java-jni:
 # Includes plasma test, jni for gandiva and orc, and C data interface.
 # Usage:
 # docker-compose build debian-java
 # docker-compose build debian-java-jni
 # docker-compose run debian-java-jni
 image: ${REPO}:${ARCH}-debian-9-java-${JDK}-maven-${MAVEN}-jni
 build:
 context: .
 dockerfile: ci/docker/linux-apt-jni.dockerfile
 cache_from:
 - ${REPO}:${ARCH}-debian-9-java-${JDK}-maven-${MAVEN}-jni
 args:
 llvm: ${LLVM}
 shm_size: *shm-size
 environment:
 <<: *ccache
 ARROW_BUILD_TESTS: "OFF"
 ARROW_S3: "OFF"
 ARROW_SUBSTRAIT: "OFF"
@pitroupitrouAug 19, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you reuse these environment variables in the java-manylinux build?
@koukouAug 19, 2022•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use ARROW_BUILD_TESTS=OFF and ARROW_SUBSTRAIT=OFF and they are already used.

(But I want to enable ARROW_BUILD_TESTS in ARROW-17081 / #13911 to run a Dataset JNI test that uses arrow_testing.)
We can't use ARROW_S3=OFF because our JNI packages enable S3 recently: ARROW-16584 / #13157
 volumes:
 - .:/arrow:delegated
 - ${DOCKER_VOLUME_PREFIX}maven-cache:/root/.m2:delegated
 - ${DOCKER_VOLUME_PREFIX}debian-ccache:/ccache:delegated
 command:
 /bin/bash -c "
 /arrow/ci/scripts/cpp_build.sh /arrow /build &&
 /arrow/ci/scripts/java_jni_build.sh /arrow /build /tmp/java_dist &&
 /arrow/ci/scripts/java_build.sh /arrow /build /tmp/java_dist &&
 /arrow/ci/scripts/java_test.sh /arrow /build /tmp/java_dist"

 oracle-java:
 # Usage:
 # docker-compose build oracle-java
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-17451: [CI][Java] Use manylinux2014 image for JNI#13920

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

kou
merged 6 commits into

from

Aug 23, 2022
 Merged


ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
#13920
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
b319661

ARROW-17451: [CI][Java] Use manylinux2014 image for JNI


kou Aug 18, 2022
36b05d1

Add missing libplasma_java


kou Aug 19, 2022
31a65eb

Show ccache statistics


kou Aug 19, 2022
3e7930d

Add missing ccache related configurations


kou Aug 20, 2022
64cf4c7

Fix path


kou Aug 20, 2022
489f84e

Fix cache key


kou Aug 20, 2022

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


12 changes: 6 additions & 6 deletions
.github/workflows/java_jni.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -47,7 +47,7 @@ env:
jobs:

 docker:
 name: AMD64 Debian 9 Java JNI (Gandiva, Plasma, ORC, Dataset)
 name: AMD64 manylinux2014 Java JNI
 runs-on: ubuntu-latest
 if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
 timeout-minutes: 90
Expand All@@ -61,23 +61,23 @@ jobs:
 uses: actions/cache@v2
 with:
 path: .docker
 key: maven-${{ hashFiles('java/**') }}
 restore-keys: maven-
 key: java-jni-manylinux-2014-${{ hashFiles('cpp/**', 'java/**') }}
 restore-keys: java-jni-manylinux-2014-
 - name: Setup Python
 uses: actions/setup-python@v4
 with:
 python-version: 3.8
 - name: Setup Archery
 run: pip install -e dev/archery[docker]
 - name: Execute Docker Build
 run: archery docker run debian-java-jni
 run: archery docker run java-jni-manylinux-2014
 - name: Docker Push
 if: success() && github.event_name == 'push' && github.repository == 'apache/arrow'
 continue-on-error: true
 run: archery docker push debian-java-jni
 run: archery docker push java-jni-manylinux-2014

 docker_integration_python:
 name: AMD64 Debian 9 Java C Data Interface Integration
 name: AMD64 Conda Java C Data Interface Integration
 runs-on: ubuntu-latest
 if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
 timeout-minutes: 90
Expand Down

10 changes: 9 additions & 1 deletion
ci/docker/java-jni-manylinux-201x.dockerfile

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -33,5 +33,13 @@ RUN vcpkg install \

# Install Java
ARG java=1.8.0
RUN yum install -y java-$java-openjdk-devel && yum clean all
RUN yum install -y java-$java-openjdk-devel rh-maven35 && yum clean all
ENV JAVA_HOME=/usr/lib/jvm/java-$java-openjdk/

# For ci/scripts/java_*.sh
ENV ARROW_GANDIVA_JAVA=ON \
 ARROW_HOME=/tmp/local \
 ARROW_JAVA_CDATA=ON \
 ARROW_JNI=ON \
 ARROW_PLASMA=ON \
 ARROW_USE_CCACHE=ON

7 changes: 5 additions & 2 deletions
ci/scripts/java_build.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -18,10 +18,13 @@

set -ex

if [[ "${ARROW_JAVA_BUILD:-ON}" != "ON" ]]; then
 exit
fi

arrow_dir=${1}
source_dir=${1}/java
build_dir=${2}
cpp_build_dir=${build_dir}/cpp/${ARROW_BUILD_TYPE:-debug}
java_jni_dist_dir=${3}

: ${BUILD_DOCS_JAVA:=OFF}
Expand DownExpand Up@@ -80,7 +83,7 @@ if [ "${ARROW_JAVA_CDATA}" = "ON" ]; then
fi

if [ "${ARROW_GANDIVA_JAVA}" = "ON" ]; then
 ${mvn} -Darrow.cpp.build.dir=${cpp_build_dir} -Parrow-jni install
 ${mvn} -Darrow.cpp.build.dir=${java_jni_dist_dir} -Parrow-jni install
fi

if [ "${ARROW_PLASMA}" = "ON" ]; then
Expand Down

25 changes: 20 additions & 5 deletions
ci/scripts/java_jni_macos_build.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -29,6 +29,7 @@ echo "=== Clear output directories and leftovers ==="
rm -rf ${build_dir}

echo "=== Building Arrow C++ libraries ==="
install_dir=${build_dir}/cpp-install
: ${ARROW_BUILD_TESTS:=OFF}
: ${ARROW_DATASET:=ON}
: ${ARROW_FILESYSTEM:=ON}
Expand All@@ -40,9 +41,15 @@ echo "=== Building Arrow C++ libraries ==="
: ${ARROW_PLASMA:=ON}
: ${ARROW_PYTHON:=OFF}
: ${ARROW_S3:=ON}
: ${ARROW_USE_CCACHE:=OFF}
@pitroupitrouAug 19, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not, but you should perhaps also pass it to CMake below?
@koukouAug 20, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I changed wrong file... I should have changed java_jni_manylinux_build.sh...
I've passed it to CMake arguments. I've updated java_jni_manylinux_build.sh too.
I chose ARROW_USE_CCACHE:=OFF here because cpp_build.sh uses it. I set ARROW_USE_CCACHE=ON in caller like other jobs.
: ${CMAKE_BUILD_TYPE:=Release}
: ${CMAKE_UNITY_BUILD:=ON}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics before build ==="
 ccache -s
fi

export ARROW_TEST_DATA="${arrow_dir}/testing/data"
export PARQUET_TEST_DATA="${arrow_dir}/cpp/submodules/parquet-testing/data"
export AWS_EC2_METADATA_DISABLED=TRUE
Expand DownExpand Up@@ -75,12 +82,13 @@ cmake \
 -DARROW_S3=${ARROW_S3} \
 -DARROW_SNAPPY_USE_SHARED=OFF \
 -DARROW_THRIFT_USE_SHARED=OFF \
 -DARROW_USE_CCACHE=${ARROW_USE_CCACHE} \
 -DARROW_UTF8PROC_USE_SHARED=OFF \
 -DARROW_ZSTD_USE_SHARED=OFF \
 -DAWSSDK_SOURCE=BUNDLED \
 -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
 -DCMAKE_INSTALL_LIBDIR=lib \
 -DCMAKE_INSTALL_PREFIX=${build_dir}/cpp \
 -DCMAKE_INSTALL_PREFIX=${install_dir} \
 -DCMAKE_UNITY_BUILD=${CMAKE_UNITY_BUILD} \
 -DPARQUET_BUILD_EXAMPLES=OFF \
 -DPARQUET_BUILD_EXECUTABLES=OFF \
Expand All@@ -102,12 +110,17 @@ ${arrow_dir}/ci/scripts/java_jni_build.sh \
 ${build_dir} \
 ${dist_dir}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics after build ==="
 ccache -s
fi

echo "=== Copying libraries to the distribution folder ==="
mkdir -p "${dist_dir}"
cp -L ${build_dir}/cpp/lib/libgandiva_jni.dylib ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_dataset_jni.dylib ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_orc_jni.dylib ${dist_dir}
cp -L ${install_dir}/lib/libarrow_dataset_jni.dylib ${dist_dir}
cp -L ${install_dir}/lib/libarrow_orc_jni.dylib ${dist_dir}
cp -L ${install_dir}/lib/libgandiva_jni.dylib ${dist_dir}
cp -L ${build_dir}/cpp/*/libplasma_java.dylib ${dist_dir}

echo "=== Checking shared dependencies for libraries ==="

Expand All@@ -122,9 +135,11 @@ archery linking check-dependencies \
 --allow libcurl \
 --allow libgandiva_jni \
 --allow libncurses \
 --allow libplasma_java \
 --allow libz \
 libarrow_cdata_jni.dylib \
 libarrow_dataset_jni.dylib \
 libarrow_orc_jni.dylib \
 libgandiva_jni.dylib
 libgandiva_jni.dylib \
 libplasma_java.dylib
popd

26 changes: 20 additions & 6 deletions
ci/scripts/java_jni_manylinux_build.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -32,6 +32,7 @@ echo "=== Building Arrow C++ libraries ==="
devtoolset_version=$(rpm -qa "devtoolset-*-gcc" --queryformat %{VERSION} | \
 grep -o "^[0-9]*")
devtoolset_include_cpp="/opt/rh/devtoolset-${devtoolset_version}/root/usr/include/c++/${devtoolset_version}"
: ${ARROW_BUILD_TESTS:=OFF}
: ${ARROW_DATASET:=ON}
: ${ARROW_GANDIVA:=ON}
: ${ARROW_GANDIVA_JAVA:=ON}
Expand All@@ -44,14 +45,19 @@ devtoolset_include_cpp="/opt/rh/devtoolset-${devtoolset_version}/root/usr/includ
: ${ARROW_PLASMA_JAVA_CLIENT:=ON}
: ${ARROW_PYTHON:=OFF}
: ${ARROW_S3:=ON}
: ${ARROW_BUILD_TESTS:=OFF}
: ${ARROW_USE_CCACHE:=OFF}
: ${CMAKE_BUILD_TYPE:=Release}
: ${CMAKE_UNITY_BUILD:=ON}
: ${VCPKG_ROOT:=/opt/vcpkg}
: ${VCPKG_FEATURE_FLAGS:=-manifests}
: ${VCPKG_TARGET_TRIPLET:=${VCPKG_DEFAULT_TRIPLET:-x64-linux-static-${CMAKE_BUILD_TYPE}}}
: ${GANDIVA_CXX_FLAGS:=-isystem;${devtoolset_include_cpp};-isystem;${devtoolset_include_cpp}/x86_64-redhat-linux;-isystem;-lpthread}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics before build ==="
 ccache -s
fi

export ARROW_TEST_DATA="${arrow_dir}/testing/data"
export PARQUET_TEST_DATA="${arrow_dir}/cpp/submodules/parquet-testing/data"
export AWS_EC2_METADATA_DISABLED=TRUE
Expand DownExpand Up@@ -87,11 +93,12 @@ cmake \
 -DARROW_S3=${ARROW_S3} \
 -DARROW_SNAPPY_USE_SHARED=OFF \
 -DARROW_THRIFT_USE_SHARED=OFF \
 -DARROW_USE_CCACHE=${ARROW_USE_CCACHE} \
 -DARROW_UTF8PROC_USE_SHARED=OFF \
 -DARROW_ZSTD_USE_SHARED=OFF \
 -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
 -DCMAKE_INSTALL_LIBDIR=lib \
 -DCMAKE_INSTALL_PREFIX=${build_dir}/cpp \
 -DCMAKE_INSTALL_PREFIX=${ARROW_HOME} \
 -DCMAKE_UNITY_BUILD=${CMAKE_UNITY_BUILD} \
 -DORC_SOURCE=BUNDLED \
 -DORC_PROTOBUF_EXECUTABLE=${VCPKG_ROOT}/installed/${VCPKG_TARGET_TRIPLET}/tools/protobuf/protoc \
Expand DownExpand Up@@ -126,11 +133,17 @@ ${arrow_dir}/ci/scripts/java_jni_build.sh \
 ${build_dir} \
 ${dist_dir}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics after build ==="
 ccache -s
fi


echo "=== Copying libraries to the distribution folder ==="
cp -L ${build_dir}/cpp/lib/libgandiva_jni.so ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_dataset_jni.so ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_orc_jni.so ${dist_dir}
cp -L ${ARROW_HOME}/lib/libarrow_dataset_jni.so ${dist_dir}
cp -L ${ARROW_HOME}/lib/libarrow_orc_jni.so ${dist_dir}
cp -L ${ARROW_HOME}/lib/libgandiva_jni.so ${dist_dir}
cp -L ${build_dir}/cpp/*/libplasma_java.so ${dist_dir}

echo "=== Checking shared dependencies for libraries ==="

Expand All@@ -149,5 +162,6 @@ archery linking check-dependencies \
 libarrow_cdata_jni.so \
 libarrow_dataset_jni.so \
 libarrow_orc_jni.so \
 libgandiva_jni.so
 libgandiva_jni.so \
 libplasma_java.so
popd

9 changes: 6 additions & 3 deletions
ci/scripts/java_test.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -18,9 +18,12 @@

set -ex

if [[ "${ARROW_JAVA_TEST:-ON}" != "ON" ]]; then
 exit
fi

arrow_dir=${1}
source_dir=${1}/java
cpp_build_dir=${2}/cpp/${ARROW_BUILD_TYPE:-debug}
java_jni_dist_dir=${3}

# For JNI and Plasma tests
Expand All@@ -36,7 +39,7 @@ pushd ${source_dir}
${mvn} test

if [ "${ARROW_JNI}" = "ON" ]; then
 ${mvn} test -Parrow-jni -pl adapter/orc,gandiva,dataset -Darrow.cpp.build.dir=${cpp_build_dir}
 ${mvn} test -Parrow-jni -pl adapter/orc,gandiva,dataset -Darrow.cpp.build.dir=${java_jni_dist_dir}
fi

if [ "${ARROW_JAVA_CDATA}" = "ON" ]; then
Expand All@@ -46,7 +49,7 @@ fi
if [ "${ARROW_PLASMA}" = "ON" ]; then
 pushd ${source_dir}/plasma
 java -cp target/test-classes:target/classes \
 -Djava.library.path=${cpp_build_dir} \
 -Djava.library.path=${java_jni_dist_dir} \
 org.apache.arrow.plasma.PlasmaClientTest
 popd
fi
Expand Down

10 changes: 5 additions & 5 deletions
cpp/src/plasma/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -185,14 +185,14 @@ if(ARROW_PLASMA_JAVA_CLIENT)

 if(APPLE)
 target_link_libraries(plasma_java
plasma_shared
 ${PLASMA_LINK_LIBS}
plasma_static
 ${PLASMA_STATIC_LINK_LIBS}
 "-undefined dynamic_lookup"
 ${PTHREAD_LIBRARY})
 else(APPLE)
 target_link_libraries(plasma_java plasma_shared ${PLASMA_LINK_LIBS}
 else()
 target_link_libraries(plasma_java plasma_static ${PLASMA_STATIC_LINK_LIBS}
 ${PTHREAD_LIBRARY})
 endif(APPLE)
 endif()
endif()
#
# Unit tests
Expand Down

11 changes: 10 additions & 1 deletion
dev/tasks/java-jars/github.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -28,7 +28,11 @@ jobs:
 {{ macros.github_checkout_arrow()|indent }}
 {{ macros.github_install_archery()|indent }}
 - name: Build C++ Libs
 run: archery docker run java-jni-manylinux-2014
 run: |
 archery docker run \
 -e ARROW_JAVA_BUILD=OFF \
 -e ARROW_JAVA_TEST=OFF \
 java-jni-manylinux-2014
 - name: Compress into single artifact to keep directory structure
 run: tar -cvzf arrow-shared-libs-linux.tar.gz arrow/java-dist/
 - name: Upload Artifacts
Expand DownExpand Up@@ -61,6 +65,9 @@ jobs:
 # aws-sdk-cpp and bundled aws-sdk-cpp. We uninstall Homebrew's
 # aws-sdk-cpp to ensure using only bundled aws-sdk-cpp.
 brew uninstall aws-sdk-cpp
 - name: Setup ccache
 run: |
 arrow/ci/scripts/ccache_setup.sh
 - name: Build C++ Libs
 run: |
 set -e
Expand DownExpand Up@@ -100,11 +107,13 @@ jobs:
 test -f arrow/java-dist/libarrow_cdata_jni.dylib
 test -f arrow/java-dist/libarrow_dataset_jni.dylib
 test -f arrow/java-dist/libgandiva_jni.dylib
 test -f arrow/java-dist/libplasma_java.dylib
 test -f arrow/java-dist/libarrow_orc_jni.dylib
 test -f arrow/java-dist/libarrow_cdata_jni.so
 test -f arrow/java-dist/libarrow_dataset_jni.so
 test -f arrow/java-dist/libarrow_orc_jni.so
 test -f arrow/java-dist/libgandiva_jni.so
 test -f arrow/java-dist/libplasma_java.so
 - name: Build Bundled Jar
 run: |
 set -e
Expand Down

40 changes: 6 additions & 34 deletions
docker-compose.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -122,7 +122,6 @@ x-hierarchy:
 - debian-go-cgo
 - debian-go-cgo-python
 - debian-java
 - debian-java-jni
 - debian-js
 - fedora-cpp:
 - fedora-python
Expand DownExpand Up@@ -999,10 +998,14 @@ services:
 <<: *ccache
 volumes:
 - .:/arrow:delegated
 - ${DOCKER_VOLUME_PREFIX}maven-cache:/root/.m2:delegated
 - ${DOCKER_VOLUME_PREFIX}python-wheel-manylinux2014-ccache:/ccache:delegated
 command:
 ["pip install -e /arrow/dev/archery &&
 /arrow/ci/scripts/java_jni_manylinux_build.sh /arrow /build /arrow/java-dist"]
 ["pip install -e /arrow/dev/archery && \
 /arrow/ci/scripts/java_jni_manylinux_build.sh /arrow /build /arrow/java-dist && \
 source /opt/rh/rh-maven35/enable && \
 /arrow/ci/scripts/java_build.sh /arrow /build /arrow/java-dist && \
 /arrow/ci/scripts/java_test.sh /arrow /build /arrow/java-dist"]

 ############################## Integration #################################

Expand DownExpand Up@@ -1541,37 +1544,6 @@ services:
 /arrow/ci/scripts/java_build.sh /arrow /build &&
 /arrow/ci/scripts/java_test.sh /arrow /build"

 debian-java-jni:
 # Includes plasma test, jni for gandiva and orc, and C data interface.
 # Usage:
 # docker-compose build debian-java
 # docker-compose build debian-java-jni
 # docker-compose run debian-java-jni
 image: ${REPO}:${ARCH}-debian-9-java-${JDK}-maven-${MAVEN}-jni
 build:
 context: .
 dockerfile: ci/docker/linux-apt-jni.dockerfile
 cache_from:
 - ${REPO}:${ARCH}-debian-9-java-${JDK}-maven-${MAVEN}-jni
 args:
 llvm: ${LLVM}
 shm_size: *shm-size
 environment:
 <<: *ccache
 ARROW_BUILD_TESTS: "OFF"
 ARROW_S3: "OFF"
 ARROW_SUBSTRAIT: "OFF"
@pitroupitrouAug 19, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you reuse these environment variables in the java-manylinux build?
@koukouAug 19, 2022•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use ARROW_BUILD_TESTS=OFF and ARROW_SUBSTRAIT=OFF and they are already used.

(But I want to enable ARROW_BUILD_TESTS in ARROW-17081 / #13911 to run a Dataset JNI test that uses arrow_testing.)
We can't use ARROW_S3=OFF because our JNI packages enable S3 recently: ARROW-16584 / #13157
 volumes:
 - .:/arrow:delegated
 - ${DOCKER_VOLUME_PREFIX}maven-cache:/root/.m2:delegated
 - ${DOCKER_VOLUME_PREFIX}debian-ccache:/ccache:delegated
 command:
 /bin/bash -c "
 /arrow/ci/scripts/cpp_build.sh /arrow /build &&
 /arrow/ci/scripts/java_jni_build.sh /arrow /build /tmp/java_dist &&
 /arrow/ci/scripts/java_build.sh /arrow /build /tmp/java_dist &&
 /arrow/ci/scripts/java_test.sh /arrow /build /tmp/java_dist"

 oracle-java:
 # Usage:
 # docker-compose build oracle-java
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
ARROW-17451: [CI][Java] Use manylinux2014 image for JNI#13920

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

kou
merged 6 commits into

from

Aug 23, 2022
 Merged


ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
#13920
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
b319661

ARROW-17451: [CI][Java] Use manylinux2014 image for JNI


kou Aug 18, 2022
36b05d1

Add missing libplasma_java


kou Aug 19, 2022
31a65eb

Show ccache statistics


kou Aug 19, 2022
3e7930d

Add missing ccache related configurations


kou Aug 20, 2022
64cf4c7

Fix path


kou Aug 20, 2022
489f84e

Fix cache key


kou Aug 20, 2022

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


12 changes: 6 additions & 6 deletions
.github/workflows/java_jni.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -47,7 +47,7 @@ env:
jobs:

 docker:
 name: AMD64 Debian 9 Java JNI (Gandiva, Plasma, ORC, Dataset)
 name: AMD64 manylinux2014 Java JNI
 runs-on: ubuntu-latest
 if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
 timeout-minutes: 90
Expand All@@ -61,23 +61,23 @@ jobs:
 uses: actions/cache@v2
 with:
 path: .docker
 key: maven-${{ hashFiles('java/**') }}
 restore-keys: maven-
 key: java-jni-manylinux-2014-${{ hashFiles('cpp/**', 'java/**') }}
 restore-keys: java-jni-manylinux-2014-
 - name: Setup Python
 uses: actions/setup-python@v4
 with:
 python-version: 3.8
 - name: Setup Archery
 run: pip install -e dev/archery[docker]
 - name: Execute Docker Build
 run: archery docker run debian-java-jni
 run: archery docker run java-jni-manylinux-2014
 - name: Docker Push
 if: success() && github.event_name == 'push' && github.repository == 'apache/arrow'
 continue-on-error: true
 run: archery docker push debian-java-jni
 run: archery docker push java-jni-manylinux-2014

 docker_integration_python:
 name: AMD64 Debian 9 Java C Data Interface Integration
 name: AMD64 Conda Java C Data Interface Integration
 runs-on: ubuntu-latest
 if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
 timeout-minutes: 90
Expand Down

10 changes: 9 additions & 1 deletion
ci/docker/java-jni-manylinux-201x.dockerfile

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -33,5 +33,13 @@ RUN vcpkg install \

# Install Java
ARG java=1.8.0
RUN yum install -y java-$java-openjdk-devel && yum clean all
RUN yum install -y java-$java-openjdk-devel rh-maven35 && yum clean all
ENV JAVA_HOME=/usr/lib/jvm/java-$java-openjdk/

# For ci/scripts/java_*.sh
ENV ARROW_GANDIVA_JAVA=ON \
 ARROW_HOME=/tmp/local \
 ARROW_JAVA_CDATA=ON \
 ARROW_JNI=ON \
 ARROW_PLASMA=ON \
 ARROW_USE_CCACHE=ON

7 changes: 5 additions & 2 deletions
ci/scripts/java_build.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -18,10 +18,13 @@

set -ex

if [[ "${ARROW_JAVA_BUILD:-ON}" != "ON" ]]; then
 exit
fi

arrow_dir=${1}
source_dir=${1}/java
build_dir=${2}
cpp_build_dir=${build_dir}/cpp/${ARROW_BUILD_TYPE:-debug}
java_jni_dist_dir=${3}

: ${BUILD_DOCS_JAVA:=OFF}
Expand DownExpand Up@@ -80,7 +83,7 @@ if [ "${ARROW_JAVA_CDATA}" = "ON" ]; then
fi

if [ "${ARROW_GANDIVA_JAVA}" = "ON" ]; then
 ${mvn} -Darrow.cpp.build.dir=${cpp_build_dir} -Parrow-jni install
 ${mvn} -Darrow.cpp.build.dir=${java_jni_dist_dir} -Parrow-jni install
fi

if [ "${ARROW_PLASMA}" = "ON" ]; then
Expand Down

25 changes: 20 additions & 5 deletions
ci/scripts/java_jni_macos_build.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -29,6 +29,7 @@ echo "=== Clear output directories and leftovers ==="
rm -rf ${build_dir}

echo "=== Building Arrow C++ libraries ==="
install_dir=${build_dir}/cpp-install
: ${ARROW_BUILD_TESTS:=OFF}
: ${ARROW_DATASET:=ON}
: ${ARROW_FILESYSTEM:=ON}
Expand All@@ -40,9 +41,15 @@ echo "=== Building Arrow C++ libraries ==="
: ${ARROW_PLASMA:=ON}
: ${ARROW_PYTHON:=OFF}
: ${ARROW_S3:=ON}
: ${ARROW_USE_CCACHE:=OFF}
@pitroupitrouAug 19, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not, but you should perhaps also pass it to CMake below?
@koukouAug 20, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I changed wrong file... I should have changed java_jni_manylinux_build.sh...
I've passed it to CMake arguments. I've updated java_jni_manylinux_build.sh too.
I chose ARROW_USE_CCACHE:=OFF here because cpp_build.sh uses it. I set ARROW_USE_CCACHE=ON in caller like other jobs.
: ${CMAKE_BUILD_TYPE:=Release}
: ${CMAKE_UNITY_BUILD:=ON}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics before build ==="
 ccache -s
fi

export ARROW_TEST_DATA="${arrow_dir}/testing/data"
export PARQUET_TEST_DATA="${arrow_dir}/cpp/submodules/parquet-testing/data"
export AWS_EC2_METADATA_DISABLED=TRUE
Expand DownExpand Up@@ -75,12 +82,13 @@ cmake \
 -DARROW_S3=${ARROW_S3} \
 -DARROW_SNAPPY_USE_SHARED=OFF \
 -DARROW_THRIFT_USE_SHARED=OFF \
 -DARROW_USE_CCACHE=${ARROW_USE_CCACHE} \
 -DARROW_UTF8PROC_USE_SHARED=OFF \
 -DARROW_ZSTD_USE_SHARED=OFF \
 -DAWSSDK_SOURCE=BUNDLED \
 -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
 -DCMAKE_INSTALL_LIBDIR=lib \
 -DCMAKE_INSTALL_PREFIX=${build_dir}/cpp \
 -DCMAKE_INSTALL_PREFIX=${install_dir} \
 -DCMAKE_UNITY_BUILD=${CMAKE_UNITY_BUILD} \
 -DPARQUET_BUILD_EXAMPLES=OFF \
 -DPARQUET_BUILD_EXECUTABLES=OFF \
Expand All@@ -102,12 +110,17 @@ ${arrow_dir}/ci/scripts/java_jni_build.sh \
 ${build_dir} \
 ${dist_dir}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics after build ==="
 ccache -s
fi

echo "=== Copying libraries to the distribution folder ==="
mkdir -p "${dist_dir}"
cp -L ${build_dir}/cpp/lib/libgandiva_jni.dylib ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_dataset_jni.dylib ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_orc_jni.dylib ${dist_dir}
cp -L ${install_dir}/lib/libarrow_dataset_jni.dylib ${dist_dir}
cp -L ${install_dir}/lib/libarrow_orc_jni.dylib ${dist_dir}
cp -L ${install_dir}/lib/libgandiva_jni.dylib ${dist_dir}
cp -L ${build_dir}/cpp/*/libplasma_java.dylib ${dist_dir}

echo "=== Checking shared dependencies for libraries ==="

Expand All@@ -122,9 +135,11 @@ archery linking check-dependencies \
 --allow libcurl \
 --allow libgandiva_jni \
 --allow libncurses \
 --allow libplasma_java \
 --allow libz \
 libarrow_cdata_jni.dylib \
 libarrow_dataset_jni.dylib \
 libarrow_orc_jni.dylib \
 libgandiva_jni.dylib
 libgandiva_jni.dylib \
 libplasma_java.dylib
popd

26 changes: 20 additions & 6 deletions
ci/scripts/java_jni_manylinux_build.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -32,6 +32,7 @@ echo "=== Building Arrow C++ libraries ==="
devtoolset_version=$(rpm -qa "devtoolset-*-gcc" --queryformat %{VERSION} | \
 grep -o "^[0-9]*")
devtoolset_include_cpp="/opt/rh/devtoolset-${devtoolset_version}/root/usr/include/c++/${devtoolset_version}"
: ${ARROW_BUILD_TESTS:=OFF}
: ${ARROW_DATASET:=ON}
: ${ARROW_GANDIVA:=ON}
: ${ARROW_GANDIVA_JAVA:=ON}
Expand All@@ -44,14 +45,19 @@ devtoolset_include_cpp="/opt/rh/devtoolset-${devtoolset_version}/root/usr/includ
: ${ARROW_PLASMA_JAVA_CLIENT:=ON}
: ${ARROW_PYTHON:=OFF}
: ${ARROW_S3:=ON}
: ${ARROW_BUILD_TESTS:=OFF}
: ${ARROW_USE_CCACHE:=OFF}
: ${CMAKE_BUILD_TYPE:=Release}
: ${CMAKE_UNITY_BUILD:=ON}
: ${VCPKG_ROOT:=/opt/vcpkg}
: ${VCPKG_FEATURE_FLAGS:=-manifests}
: ${VCPKG_TARGET_TRIPLET:=${VCPKG_DEFAULT_TRIPLET:-x64-linux-static-${CMAKE_BUILD_TYPE}}}
: ${GANDIVA_CXX_FLAGS:=-isystem;${devtoolset_include_cpp};-isystem;${devtoolset_include_cpp}/x86_64-redhat-linux;-isystem;-lpthread}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics before build ==="
 ccache -s
fi

export ARROW_TEST_DATA="${arrow_dir}/testing/data"
export PARQUET_TEST_DATA="${arrow_dir}/cpp/submodules/parquet-testing/data"
export AWS_EC2_METADATA_DISABLED=TRUE
Expand DownExpand Up@@ -87,11 +93,12 @@ cmake \
 -DARROW_S3=${ARROW_S3} \
 -DARROW_SNAPPY_USE_SHARED=OFF \
 -DARROW_THRIFT_USE_SHARED=OFF \
 -DARROW_USE_CCACHE=${ARROW_USE_CCACHE} \
 -DARROW_UTF8PROC_USE_SHARED=OFF \
 -DARROW_ZSTD_USE_SHARED=OFF \
 -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
 -DCMAKE_INSTALL_LIBDIR=lib \
 -DCMAKE_INSTALL_PREFIX=${build_dir}/cpp \
 -DCMAKE_INSTALL_PREFIX=${ARROW_HOME} \
 -DCMAKE_UNITY_BUILD=${CMAKE_UNITY_BUILD} \
 -DORC_SOURCE=BUNDLED \
 -DORC_PROTOBUF_EXECUTABLE=${VCPKG_ROOT}/installed/${VCPKG_TARGET_TRIPLET}/tools/protobuf/protoc \
Expand DownExpand Up@@ -126,11 +133,17 @@ ${arrow_dir}/ci/scripts/java_jni_build.sh \
 ${build_dir} \
 ${dist_dir}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics after build ==="
 ccache -s
fi


echo "=== Copying libraries to the distribution folder ==="
cp -L ${build_dir}/cpp/lib/libgandiva_jni.so ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_dataset_jni.so ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_orc_jni.so ${dist_dir}
cp -L ${ARROW_HOME}/lib/libarrow_dataset_jni.so ${dist_dir}
cp -L ${ARROW_HOME}/lib/libarrow_orc_jni.so ${dist_dir}
cp -L ${ARROW_HOME}/lib/libgandiva_jni.so ${dist_dir}
cp -L ${build_dir}/cpp/*/libplasma_java.so ${dist_dir}

echo "=== Checking shared dependencies for libraries ==="

Expand All@@ -149,5 +162,6 @@ archery linking check-dependencies \
 libarrow_cdata_jni.so \
 libarrow_dataset_jni.so \
 libarrow_orc_jni.so \
 libgandiva_jni.so
 libgandiva_jni.so \
 libplasma_java.so
popd

9 changes: 6 additions & 3 deletions
ci/scripts/java_test.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -18,9 +18,12 @@

set -ex

if [[ "${ARROW_JAVA_TEST:-ON}" != "ON" ]]; then
 exit
fi

arrow_dir=${1}
source_dir=${1}/java
cpp_build_dir=${2}/cpp/${ARROW_BUILD_TYPE:-debug}
java_jni_dist_dir=${3}

# For JNI and Plasma tests
Expand All@@ -36,7 +39,7 @@ pushd ${source_dir}
${mvn} test

if [ "${ARROW_JNI}" = "ON" ]; then
 ${mvn} test -Parrow-jni -pl adapter/orc,gandiva,dataset -Darrow.cpp.build.dir=${cpp_build_dir}
 ${mvn} test -Parrow-jni -pl adapter/orc,gandiva,dataset -Darrow.cpp.build.dir=${java_jni_dist_dir}
fi

if [ "${ARROW_JAVA_CDATA}" = "ON" ]; then
Expand All@@ -46,7 +49,7 @@ fi
if [ "${ARROW_PLASMA}" = "ON" ]; then
 pushd ${source_dir}/plasma
 java -cp target/test-classes:target/classes \
 -Djava.library.path=${cpp_build_dir} \
 -Djava.library.path=${java_jni_dist_dir} \
 org.apache.arrow.plasma.PlasmaClientTest
 popd
fi
Expand Down

10 changes: 5 additions & 5 deletions
cpp/src/plasma/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -185,14 +185,14 @@ if(ARROW_PLASMA_JAVA_CLIENT)

 if(APPLE)
 target_link_libraries(plasma_java
plasma_shared
 ${PLASMA_LINK_LIBS}
plasma_static
 ${PLASMA_STATIC_LINK_LIBS}
 "-undefined dynamic_lookup"
 ${PTHREAD_LIBRARY})
 else(APPLE)
 target_link_libraries(plasma_java plasma_shared ${PLASMA_LINK_LIBS}
 else()
 target_link_libraries(plasma_java plasma_static ${PLASMA_STATIC_LINK_LIBS}
 ${PTHREAD_LIBRARY})
 endif(APPLE)
 endif()
endif()
#
# Unit tests
Expand Down

11 changes: 10 additions & 1 deletion
dev/tasks/java-jars/github.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -28,7 +28,11 @@ jobs:
 {{ macros.github_checkout_arrow()|indent }}
 {{ macros.github_install_archery()|indent }}
 - name: Build C++ Libs
 run: archery docker run java-jni-manylinux-2014
 run: |
 archery docker run \
 -e ARROW_JAVA_BUILD=OFF \
 -e ARROW_JAVA_TEST=OFF \
 java-jni-manylinux-2014
 - name: Compress into single artifact to keep directory structure
 run: tar -cvzf arrow-shared-libs-linux.tar.gz arrow/java-dist/
 - name: Upload Artifacts
Expand DownExpand Up@@ -61,6 +65,9 @@ jobs:
 # aws-sdk-cpp and bundled aws-sdk-cpp. We uninstall Homebrew's
 # aws-sdk-cpp to ensure using only bundled aws-sdk-cpp.
 brew uninstall aws-sdk-cpp
 - name: Setup ccache
 run: |
 arrow/ci/scripts/ccache_setup.sh
 - name: Build C++ Libs
 run: |
 set -e
Expand DownExpand Up@@ -100,11 +107,13 @@ jobs:
 test -f arrow/java-dist/libarrow_cdata_jni.dylib
 test -f arrow/java-dist/libarrow_dataset_jni.dylib
 test -f arrow/java-dist/libgandiva_jni.dylib
 test -f arrow/java-dist/libplasma_java.dylib
 test -f arrow/java-dist/libarrow_orc_jni.dylib
 test -f arrow/java-dist/libarrow_cdata_jni.so
 test -f arrow/java-dist/libarrow_dataset_jni.so
 test -f arrow/java-dist/libarrow_orc_jni.so
 test -f arrow/java-dist/libgandiva_jni.so
 test -f arrow/java-dist/libplasma_java.so
 - name: Build Bundled Jar
 run: |
 set -e
Expand Down

40 changes: 6 additions & 34 deletions
docker-compose.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -122,7 +122,6 @@ x-hierarchy:
 - debian-go-cgo
 - debian-go-cgo-python
 - debian-java
 - debian-java-jni
 - debian-js
 - fedora-cpp:
 - fedora-python
Expand DownExpand Up@@ -999,10 +998,14 @@ services:
 <<: *ccache
 volumes:
 - .:/arrow:delegated
 - ${DOCKER_VOLUME_PREFIX}maven-cache:/root/.m2:delegated
 - ${DOCKER_VOLUME_PREFIX}python-wheel-manylinux2014-ccache:/ccache:delegated
 command:
 ["pip install -e /arrow/dev/archery &&
 /arrow/ci/scripts/java_jni_manylinux_build.sh /arrow /build /arrow/java-dist"]
 ["pip install -e /arrow/dev/archery && \
 /arrow/ci/scripts/java_jni_manylinux_build.sh /arrow /build /arrow/java-dist && \
 source /opt/rh/rh-maven35/enable && \
 /arrow/ci/scripts/java_build.sh /arrow /build /arrow/java-dist && \
 /arrow/ci/scripts/java_test.sh /arrow /build /arrow/java-dist"]

 ############################## Integration #################################

Expand DownExpand Up@@ -1541,37 +1544,6 @@ services:
 /arrow/ci/scripts/java_build.sh /arrow /build &&
 /arrow/ci/scripts/java_test.sh /arrow /build"

 debian-java-jni:
 # Includes plasma test, jni for gandiva and orc, and C data interface.
 # Usage:
 # docker-compose build debian-java
 # docker-compose build debian-java-jni
 # docker-compose run debian-java-jni
 image: ${REPO}:${ARCH}-debian-9-java-${JDK}-maven-${MAVEN}-jni
 build:
 context: .
 dockerfile: ci/docker/linux-apt-jni.dockerfile
 cache_from:
 - ${REPO}:${ARCH}-debian-9-java-${JDK}-maven-${MAVEN}-jni
 args:
 llvm: ${LLVM}
 shm_size: *shm-size
 environment:
 <<: *ccache
 ARROW_BUILD_TESTS: "OFF"
 ARROW_S3: "OFF"
 ARROW_SUBSTRAIT: "OFF"
@pitroupitrouAug 19, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you reuse these environment variables in the java-manylinux build?
@koukouAug 19, 2022•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use ARROW_BUILD_TESTS=OFF and ARROW_SUBSTRAIT=OFF and they are already used.

(But I want to enable ARROW_BUILD_TESTS in ARROW-17081 / #13911 to run a Dataset JNI test that uses arrow_testing.)
We can't use ARROW_S3=OFF because our JNI packages enable S3 recently: ARROW-16584 / #13157
 volumes:
 - .:/arrow:delegated
 - ${DOCKER_VOLUME_PREFIX}maven-cache:/root/.m2:delegated
 - ${DOCKER_VOLUME_PREFIX}debian-ccache:/ccache:delegated
 command:
 /bin/bash -c "
 /arrow/ci/scripts/cpp_build.sh /arrow /build &&
 /arrow/ci/scripts/java_jni_build.sh /arrow /build /tmp/java_dist &&
 /arrow/ci/scripts/java_build.sh /arrow /build /tmp/java_dist &&
 /arrow/ci/scripts/java_test.sh /arrow /build /tmp/java_dist"

 oracle-java:
 # Usage:
 # docker-compose build oracle-java
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-17451: [CI][Java] Use manylinux2014 image for JNI#13920

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

kou
merged 6 commits into

from

Aug 23, 2022
 Merged


ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
#13920
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
b319661

ARROW-17451: [CI][Java] Use manylinux2014 image for JNI


kou Aug 18, 2022
36b05d1

Add missing libplasma_java


kou Aug 19, 2022
31a65eb

Show ccache statistics


kou Aug 19, 2022
3e7930d

Add missing ccache related configurations


kou Aug 20, 2022
64cf4c7

Fix path


kou Aug 20, 2022
489f84e

Fix cache key


kou Aug 20, 2022

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


12 changes: 6 additions & 6 deletions
.github/workflows/java_jni.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -47,7 +47,7 @@ env:
jobs:

 docker:
 name: AMD64 Debian 9 Java JNI (Gandiva, Plasma, ORC, Dataset)
 name: AMD64 manylinux2014 Java JNI
 runs-on: ubuntu-latest
 if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
 timeout-minutes: 90
Expand All@@ -61,23 +61,23 @@ jobs:
 uses: actions/cache@v2
 with:
 path: .docker
 key: maven-${{ hashFiles('java/**') }}
 restore-keys: maven-
 key: java-jni-manylinux-2014-${{ hashFiles('cpp/**', 'java/**') }}
 restore-keys: java-jni-manylinux-2014-
 - name: Setup Python
 uses: actions/setup-python@v4
 with:
 python-version: 3.8
 - name: Setup Archery
 run: pip install -e dev/archery[docker]
 - name: Execute Docker Build
 run: archery docker run debian-java-jni
 run: archery docker run java-jni-manylinux-2014
 - name: Docker Push
 if: success() && github.event_name == 'push' && github.repository == 'apache/arrow'
 continue-on-error: true
 run: archery docker push debian-java-jni
 run: archery docker push java-jni-manylinux-2014

 docker_integration_python:
 name: AMD64 Debian 9 Java C Data Interface Integration
 name: AMD64 Conda Java C Data Interface Integration
 runs-on: ubuntu-latest
 if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
 timeout-minutes: 90
Expand Down

10 changes: 9 additions & 1 deletion
ci/docker/java-jni-manylinux-201x.dockerfile

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -33,5 +33,13 @@ RUN vcpkg install \

# Install Java
ARG java=1.8.0
RUN yum install -y java-$java-openjdk-devel && yum clean all
RUN yum install -y java-$java-openjdk-devel rh-maven35 && yum clean all
ENV JAVA_HOME=/usr/lib/jvm/java-$java-openjdk/

# For ci/scripts/java_*.sh
ENV ARROW_GANDIVA_JAVA=ON \
 ARROW_HOME=/tmp/local \
 ARROW_JAVA_CDATA=ON \
 ARROW_JNI=ON \
 ARROW_PLASMA=ON \
 ARROW_USE_CCACHE=ON

7 changes: 5 additions & 2 deletions
ci/scripts/java_build.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -18,10 +18,13 @@

set -ex

if [[ "${ARROW_JAVA_BUILD:-ON}" != "ON" ]]; then
 exit
fi

arrow_dir=${1}
source_dir=${1}/java
build_dir=${2}
cpp_build_dir=${build_dir}/cpp/${ARROW_BUILD_TYPE:-debug}
java_jni_dist_dir=${3}

: ${BUILD_DOCS_JAVA:=OFF}
Expand DownExpand Up@@ -80,7 +83,7 @@ if [ "${ARROW_JAVA_CDATA}" = "ON" ]; then
fi

if [ "${ARROW_GANDIVA_JAVA}" = "ON" ]; then
 ${mvn} -Darrow.cpp.build.dir=${cpp_build_dir} -Parrow-jni install
 ${mvn} -Darrow.cpp.build.dir=${java_jni_dist_dir} -Parrow-jni install
fi

if [ "${ARROW_PLASMA}" = "ON" ]; then
Expand Down

25 changes: 20 additions & 5 deletions
ci/scripts/java_jni_macos_build.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -29,6 +29,7 @@ echo "=== Clear output directories and leftovers ==="
rm -rf ${build_dir}

echo "=== Building Arrow C++ libraries ==="
install_dir=${build_dir}/cpp-install
: ${ARROW_BUILD_TESTS:=OFF}
: ${ARROW_DATASET:=ON}
: ${ARROW_FILESYSTEM:=ON}
Expand All@@ -40,9 +41,15 @@ echo "=== Building Arrow C++ libraries ==="
: ${ARROW_PLASMA:=ON}
: ${ARROW_PYTHON:=OFF}
: ${ARROW_S3:=ON}
: ${ARROW_USE_CCACHE:=OFF}
@pitroupitrouAug 19, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not, but you should perhaps also pass it to CMake below?
@koukouAug 20, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I changed wrong file... I should have changed java_jni_manylinux_build.sh...
I've passed it to CMake arguments. I've updated java_jni_manylinux_build.sh too.
I chose ARROW_USE_CCACHE:=OFF here because cpp_build.sh uses it. I set ARROW_USE_CCACHE=ON in caller like other jobs.
: ${CMAKE_BUILD_TYPE:=Release}
: ${CMAKE_UNITY_BUILD:=ON}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics before build ==="
 ccache -s
fi

export ARROW_TEST_DATA="${arrow_dir}/testing/data"
export PARQUET_TEST_DATA="${arrow_dir}/cpp/submodules/parquet-testing/data"
export AWS_EC2_METADATA_DISABLED=TRUE
Expand DownExpand Up@@ -75,12 +82,13 @@ cmake \
 -DARROW_S3=${ARROW_S3} \
 -DARROW_SNAPPY_USE_SHARED=OFF \
 -DARROW_THRIFT_USE_SHARED=OFF \
 -DARROW_USE_CCACHE=${ARROW_USE_CCACHE} \
 -DARROW_UTF8PROC_USE_SHARED=OFF \
 -DARROW_ZSTD_USE_SHARED=OFF \
 -DAWSSDK_SOURCE=BUNDLED \
 -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
 -DCMAKE_INSTALL_LIBDIR=lib \
 -DCMAKE_INSTALL_PREFIX=${build_dir}/cpp \
 -DCMAKE_INSTALL_PREFIX=${install_dir} \
 -DCMAKE_UNITY_BUILD=${CMAKE_UNITY_BUILD} \
 -DPARQUET_BUILD_EXAMPLES=OFF \
 -DPARQUET_BUILD_EXECUTABLES=OFF \
Expand All@@ -102,12 +110,17 @@ ${arrow_dir}/ci/scripts/java_jni_build.sh \
 ${build_dir} \
 ${dist_dir}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics after build ==="
 ccache -s
fi

echo "=== Copying libraries to the distribution folder ==="
mkdir -p "${dist_dir}"
cp -L ${build_dir}/cpp/lib/libgandiva_jni.dylib ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_dataset_jni.dylib ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_orc_jni.dylib ${dist_dir}
cp -L ${install_dir}/lib/libarrow_dataset_jni.dylib ${dist_dir}
cp -L ${install_dir}/lib/libarrow_orc_jni.dylib ${dist_dir}
cp -L ${install_dir}/lib/libgandiva_jni.dylib ${dist_dir}
cp -L ${build_dir}/cpp/*/libplasma_java.dylib ${dist_dir}

echo "=== Checking shared dependencies for libraries ==="

Expand All@@ -122,9 +135,11 @@ archery linking check-dependencies \
 --allow libcurl \
 --allow libgandiva_jni \
 --allow libncurses \
 --allow libplasma_java \
 --allow libz \
 libarrow_cdata_jni.dylib \
 libarrow_dataset_jni.dylib \
 libarrow_orc_jni.dylib \
 libgandiva_jni.dylib
 libgandiva_jni.dylib \
 libplasma_java.dylib
popd

26 changes: 20 additions & 6 deletions
ci/scripts/java_jni_manylinux_build.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -32,6 +32,7 @@ echo "=== Building Arrow C++ libraries ==="
devtoolset_version=$(rpm -qa "devtoolset-*-gcc" --queryformat %{VERSION} | \
 grep -o "^[0-9]*")
devtoolset_include_cpp="/opt/rh/devtoolset-${devtoolset_version}/root/usr/include/c++/${devtoolset_version}"
: ${ARROW_BUILD_TESTS:=OFF}
: ${ARROW_DATASET:=ON}
: ${ARROW_GANDIVA:=ON}
: ${ARROW_GANDIVA_JAVA:=ON}
Expand All@@ -44,14 +45,19 @@ devtoolset_include_cpp="/opt/rh/devtoolset-${devtoolset_version}/root/usr/includ
: ${ARROW_PLASMA_JAVA_CLIENT:=ON}
: ${ARROW_PYTHON:=OFF}
: ${ARROW_S3:=ON}
: ${ARROW_BUILD_TESTS:=OFF}
: ${ARROW_USE_CCACHE:=OFF}
: ${CMAKE_BUILD_TYPE:=Release}
: ${CMAKE_UNITY_BUILD:=ON}
: ${VCPKG_ROOT:=/opt/vcpkg}
: ${VCPKG_FEATURE_FLAGS:=-manifests}
: ${VCPKG_TARGET_TRIPLET:=${VCPKG_DEFAULT_TRIPLET:-x64-linux-static-${CMAKE_BUILD_TYPE}}}
: ${GANDIVA_CXX_FLAGS:=-isystem;${devtoolset_include_cpp};-isystem;${devtoolset_include_cpp}/x86_64-redhat-linux;-isystem;-lpthread}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics before build ==="
 ccache -s
fi

export ARROW_TEST_DATA="${arrow_dir}/testing/data"
export PARQUET_TEST_DATA="${arrow_dir}/cpp/submodules/parquet-testing/data"
export AWS_EC2_METADATA_DISABLED=TRUE
Expand DownExpand Up@@ -87,11 +93,12 @@ cmake \
 -DARROW_S3=${ARROW_S3} \
 -DARROW_SNAPPY_USE_SHARED=OFF \
 -DARROW_THRIFT_USE_SHARED=OFF \
 -DARROW_USE_CCACHE=${ARROW_USE_CCACHE} \
 -DARROW_UTF8PROC_USE_SHARED=OFF \
 -DARROW_ZSTD_USE_SHARED=OFF \
 -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
 -DCMAKE_INSTALL_LIBDIR=lib \
 -DCMAKE_INSTALL_PREFIX=${build_dir}/cpp \
 -DCMAKE_INSTALL_PREFIX=${ARROW_HOME} \
 -DCMAKE_UNITY_BUILD=${CMAKE_UNITY_BUILD} \
 -DORC_SOURCE=BUNDLED \
 -DORC_PROTOBUF_EXECUTABLE=${VCPKG_ROOT}/installed/${VCPKG_TARGET_TRIPLET}/tools/protobuf/protoc \
Expand DownExpand Up@@ -126,11 +133,17 @@ ${arrow_dir}/ci/scripts/java_jni_build.sh \
 ${build_dir} \
 ${dist_dir}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics after build ==="
 ccache -s
fi


echo "=== Copying libraries to the distribution folder ==="
cp -L ${build_dir}/cpp/lib/libgandiva_jni.so ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_dataset_jni.so ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_orc_jni.so ${dist_dir}
cp -L ${ARROW_HOME}/lib/libarrow_dataset_jni.so ${dist_dir}
cp -L ${ARROW_HOME}/lib/libarrow_orc_jni.so ${dist_dir}
cp -L ${ARROW_HOME}/lib/libgandiva_jni.so ${dist_dir}
cp -L ${build_dir}/cpp/*/libplasma_java.so ${dist_dir}

echo "=== Checking shared dependencies for libraries ==="

Expand All@@ -149,5 +162,6 @@ archery linking check-dependencies \
 libarrow_cdata_jni.so \
 libarrow_dataset_jni.so \
 libarrow_orc_jni.so \
 libgandiva_jni.so
 libgandiva_jni.so \
 libplasma_java.so
popd

9 changes: 6 additions & 3 deletions
ci/scripts/java_test.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -18,9 +18,12 @@

set -ex

if [[ "${ARROW_JAVA_TEST:-ON}" != "ON" ]]; then
 exit
fi

arrow_dir=${1}
source_dir=${1}/java
cpp_build_dir=${2}/cpp/${ARROW_BUILD_TYPE:-debug}
java_jni_dist_dir=${3}

# For JNI and Plasma tests
Expand All@@ -36,7 +39,7 @@ pushd ${source_dir}
${mvn} test

if [ "${ARROW_JNI}" = "ON" ]; then
 ${mvn} test -Parrow-jni -pl adapter/orc,gandiva,dataset -Darrow.cpp.build.dir=${cpp_build_dir}
 ${mvn} test -Parrow-jni -pl adapter/orc,gandiva,dataset -Darrow.cpp.build.dir=${java_jni_dist_dir}
fi

if [ "${ARROW_JAVA_CDATA}" = "ON" ]; then
Expand All@@ -46,7 +49,7 @@ fi
if [ "${ARROW_PLASMA}" = "ON" ]; then
 pushd ${source_dir}/plasma
 java -cp target/test-classes:target/classes \
 -Djava.library.path=${cpp_build_dir} \
 -Djava.library.path=${java_jni_dist_dir} \
 org.apache.arrow.plasma.PlasmaClientTest
 popd
fi
Expand Down

10 changes: 5 additions & 5 deletions
cpp/src/plasma/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -185,14 +185,14 @@ if(ARROW_PLASMA_JAVA_CLIENT)

 if(APPLE)
 target_link_libraries(plasma_java
plasma_shared
 ${PLASMA_LINK_LIBS}
plasma_static
 ${PLASMA_STATIC_LINK_LIBS}
 "-undefined dynamic_lookup"
 ${PTHREAD_LIBRARY})
 else(APPLE)
 target_link_libraries(plasma_java plasma_shared ${PLASMA_LINK_LIBS}
 else()
 target_link_libraries(plasma_java plasma_static ${PLASMA_STATIC_LINK_LIBS}
 ${PTHREAD_LIBRARY})
 endif(APPLE)
 endif()
endif()
#
# Unit tests
Expand Down

11 changes: 10 additions & 1 deletion
dev/tasks/java-jars/github.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -28,7 +28,11 @@ jobs:
 {{ macros.github_checkout_arrow()|indent }}
 {{ macros.github_install_archery()|indent }}
 - name: Build C++ Libs
 run: archery docker run java-jni-manylinux-2014
 run: |
 archery docker run \
 -e ARROW_JAVA_BUILD=OFF \
 -e ARROW_JAVA_TEST=OFF \
 java-jni-manylinux-2014
 - name: Compress into single artifact to keep directory structure
 run: tar -cvzf arrow-shared-libs-linux.tar.gz arrow/java-dist/
 - name: Upload Artifacts
Expand DownExpand Up@@ -61,6 +65,9 @@ jobs:
 # aws-sdk-cpp and bundled aws-sdk-cpp. We uninstall Homebrew's
 # aws-sdk-cpp to ensure using only bundled aws-sdk-cpp.
 brew uninstall aws-sdk-cpp
 - name: Setup ccache
 run: |
 arrow/ci/scripts/ccache_setup.sh
 - name: Build C++ Libs
 run: |
 set -e
Expand DownExpand Up@@ -100,11 +107,13 @@ jobs:
 test -f arrow/java-dist/libarrow_cdata_jni.dylib
 test -f arrow/java-dist/libarrow_dataset_jni.dylib
 test -f arrow/java-dist/libgandiva_jni.dylib
 test -f arrow/java-dist/libplasma_java.dylib
 test -f arrow/java-dist/libarrow_orc_jni.dylib
 test -f arrow/java-dist/libarrow_cdata_jni.so
 test -f arrow/java-dist/libarrow_dataset_jni.so
 test -f arrow/java-dist/libarrow_orc_jni.so
 test -f arrow/java-dist/libgandiva_jni.so
 test -f arrow/java-dist/libplasma_java.so
 - name: Build Bundled Jar
 run: |
 set -e
Expand Down

40 changes: 6 additions & 34 deletions
docker-compose.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -122,7 +122,6 @@ x-hierarchy:
 - debian-go-cgo
 - debian-go-cgo-python
 - debian-java
 - debian-java-jni
 - debian-js
 - fedora-cpp:
 - fedora-python
Expand DownExpand Up@@ -999,10 +998,14 @@ services:
 <<: *ccache
 volumes:
 - .:/arrow:delegated
 - ${DOCKER_VOLUME_PREFIX}maven-cache:/root/.m2:delegated
 - ${DOCKER_VOLUME_PREFIX}python-wheel-manylinux2014-ccache:/ccache:delegated
 command:
 ["pip install -e /arrow/dev/archery &&
 /arrow/ci/scripts/java_jni_manylinux_build.sh /arrow /build /arrow/java-dist"]
 ["pip install -e /arrow/dev/archery && \
 /arrow/ci/scripts/java_jni_manylinux_build.sh /arrow /build /arrow/java-dist && \
 source /opt/rh/rh-maven35/enable && \
 /arrow/ci/scripts/java_build.sh /arrow /build /arrow/java-dist && \
 /arrow/ci/scripts/java_test.sh /arrow /build /arrow/java-dist"]

 ############################## Integration #################################

Expand DownExpand Up@@ -1541,37 +1544,6 @@ services:
 /arrow/ci/scripts/java_build.sh /arrow /build &&
 /arrow/ci/scripts/java_test.sh /arrow /build"

 debian-java-jni:
 # Includes plasma test, jni for gandiva and orc, and C data interface.
 # Usage:
 # docker-compose build debian-java
 # docker-compose build debian-java-jni
 # docker-compose run debian-java-jni
 image: ${REPO}:${ARCH}-debian-9-java-${JDK}-maven-${MAVEN}-jni
 build:
 context: .
 dockerfile: ci/docker/linux-apt-jni.dockerfile
 cache_from:
 - ${REPO}:${ARCH}-debian-9-java-${JDK}-maven-${MAVEN}-jni
 args:
 llvm: ${LLVM}
 shm_size: *shm-size
 environment:
 <<: *ccache
 ARROW_BUILD_TESTS: "OFF"
 ARROW_S3: "OFF"
 ARROW_SUBSTRAIT: "OFF"
@pitroupitrouAug 19, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you reuse these environment variables in the java-manylinux build?
@koukouAug 19, 2022•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use ARROW_BUILD_TESTS=OFF and ARROW_SUBSTRAIT=OFF and they are already used.

(But I want to enable ARROW_BUILD_TESTS in ARROW-17081 / #13911 to run a Dataset JNI test that uses arrow_testing.)
We can't use ARROW_S3=OFF because our JNI packages enable S3 recently: ARROW-16584 / #13157
 volumes:
 - .:/arrow:delegated
 - ${DOCKER_VOLUME_PREFIX}maven-cache:/root/.m2:delegated
 - ${DOCKER_VOLUME_PREFIX}debian-ccache:/ccache:delegated
 command:
 /bin/bash -c "
 /arrow/ci/scripts/cpp_build.sh /arrow /build &&
 /arrow/ci/scripts/java_jni_build.sh /arrow /build /tmp/java_dist &&
 /arrow/ci/scripts/java_build.sh /arrow /build /tmp/java_dist &&
 /arrow/ci/scripts/java_test.sh /arrow /build /tmp/java_dist"

 oracle-java:
 # Usage:
 # docker-compose build oracle-java
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-17451: [CI][Java] Use manylinux2014 image for JNI#13920

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

kou
merged 6 commits into

from

Aug 23, 2022
 Merged


ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
#13920
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
b319661

ARROW-17451: [CI][Java] Use manylinux2014 image for JNI


kou Aug 18, 2022
36b05d1

Add missing libplasma_java


kou Aug 19, 2022
31a65eb

Show ccache statistics


kou Aug 19, 2022
3e7930d

Add missing ccache related configurations


kou Aug 20, 2022
64cf4c7

Fix path


kou Aug 20, 2022
489f84e

Fix cache key


kou Aug 20, 2022

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


12 changes: 6 additions & 6 deletions
.github/workflows/java_jni.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -47,7 +47,7 @@ env:
jobs:

 docker:
 name: AMD64 Debian 9 Java JNI (Gandiva, Plasma, ORC, Dataset)
 name: AMD64 manylinux2014 Java JNI
 runs-on: ubuntu-latest
 if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
 timeout-minutes: 90
Expand All@@ -61,23 +61,23 @@ jobs:
 uses: actions/cache@v2
 with:
 path: .docker
 key: maven-${{ hashFiles('java/**') }}
 restore-keys: maven-
 key: java-jni-manylinux-2014-${{ hashFiles('cpp/**', 'java/**') }}
 restore-keys: java-jni-manylinux-2014-
 - name: Setup Python
 uses: actions/setup-python@v4
 with:
 python-version: 3.8
 - name: Setup Archery
 run: pip install -e dev/archery[docker]
 - name: Execute Docker Build
 run: archery docker run debian-java-jni
 run: archery docker run java-jni-manylinux-2014
 - name: Docker Push
 if: success() && github.event_name == 'push' && github.repository == 'apache/arrow'
 continue-on-error: true
 run: archery docker push debian-java-jni
 run: archery docker push java-jni-manylinux-2014

 docker_integration_python:
 name: AMD64 Debian 9 Java C Data Interface Integration
 name: AMD64 Conda Java C Data Interface Integration
 runs-on: ubuntu-latest
 if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
 timeout-minutes: 90
Expand Down

10 changes: 9 additions & 1 deletion
ci/docker/java-jni-manylinux-201x.dockerfile

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -33,5 +33,13 @@ RUN vcpkg install \

# Install Java
ARG java=1.8.0
RUN yum install -y java-$java-openjdk-devel && yum clean all
RUN yum install -y java-$java-openjdk-devel rh-maven35 && yum clean all
ENV JAVA_HOME=/usr/lib/jvm/java-$java-openjdk/

# For ci/scripts/java_*.sh
ENV ARROW_GANDIVA_JAVA=ON \
 ARROW_HOME=/tmp/local \
 ARROW_JAVA_CDATA=ON \
 ARROW_JNI=ON \
 ARROW_PLASMA=ON \
 ARROW_USE_CCACHE=ON

7 changes: 5 additions & 2 deletions
ci/scripts/java_build.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -18,10 +18,13 @@

set -ex

if [[ "${ARROW_JAVA_BUILD:-ON}" != "ON" ]]; then
 exit
fi

arrow_dir=${1}
source_dir=${1}/java
build_dir=${2}
cpp_build_dir=${build_dir}/cpp/${ARROW_BUILD_TYPE:-debug}
java_jni_dist_dir=${3}

: ${BUILD_DOCS_JAVA:=OFF}
Expand DownExpand Up@@ -80,7 +83,7 @@ if [ "${ARROW_JAVA_CDATA}" = "ON" ]; then
fi

if [ "${ARROW_GANDIVA_JAVA}" = "ON" ]; then
 ${mvn} -Darrow.cpp.build.dir=${cpp_build_dir} -Parrow-jni install
 ${mvn} -Darrow.cpp.build.dir=${java_jni_dist_dir} -Parrow-jni install
fi

if [ "${ARROW_PLASMA}" = "ON" ]; then
Expand Down

25 changes: 20 additions & 5 deletions
ci/scripts/java_jni_macos_build.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -29,6 +29,7 @@ echo "=== Clear output directories and leftovers ==="
rm -rf ${build_dir}

echo "=== Building Arrow C++ libraries ==="
install_dir=${build_dir}/cpp-install
: ${ARROW_BUILD_TESTS:=OFF}
: ${ARROW_DATASET:=ON}
: ${ARROW_FILESYSTEM:=ON}
Expand All@@ -40,9 +41,15 @@ echo "=== Building Arrow C++ libraries ==="
: ${ARROW_PLASMA:=ON}
: ${ARROW_PYTHON:=OFF}
: ${ARROW_S3:=ON}
: ${ARROW_USE_CCACHE:=OFF}
@pitroupitrouAug 19, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not, but you should perhaps also pass it to CMake below?
@koukouAug 20, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I changed wrong file... I should have changed java_jni_manylinux_build.sh...
I've passed it to CMake arguments. I've updated java_jni_manylinux_build.sh too.
I chose ARROW_USE_CCACHE:=OFF here because cpp_build.sh uses it. I set ARROW_USE_CCACHE=ON in caller like other jobs.
: ${CMAKE_BUILD_TYPE:=Release}
: ${CMAKE_UNITY_BUILD:=ON}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics before build ==="
 ccache -s
fi

export ARROW_TEST_DATA="${arrow_dir}/testing/data"
export PARQUET_TEST_DATA="${arrow_dir}/cpp/submodules/parquet-testing/data"
export AWS_EC2_METADATA_DISABLED=TRUE
Expand DownExpand Up@@ -75,12 +82,13 @@ cmake \
 -DARROW_S3=${ARROW_S3} \
 -DARROW_SNAPPY_USE_SHARED=OFF \
 -DARROW_THRIFT_USE_SHARED=OFF \
 -DARROW_USE_CCACHE=${ARROW_USE_CCACHE} \
 -DARROW_UTF8PROC_USE_SHARED=OFF \
 -DARROW_ZSTD_USE_SHARED=OFF \
 -DAWSSDK_SOURCE=BUNDLED \
 -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
 -DCMAKE_INSTALL_LIBDIR=lib \
 -DCMAKE_INSTALL_PREFIX=${build_dir}/cpp \
 -DCMAKE_INSTALL_PREFIX=${install_dir} \
 -DCMAKE_UNITY_BUILD=${CMAKE_UNITY_BUILD} \
 -DPARQUET_BUILD_EXAMPLES=OFF \
 -DPARQUET_BUILD_EXECUTABLES=OFF \
Expand All@@ -102,12 +110,17 @@ ${arrow_dir}/ci/scripts/java_jni_build.sh \
 ${build_dir} \
 ${dist_dir}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics after build ==="
 ccache -s
fi

echo "=== Copying libraries to the distribution folder ==="
mkdir -p "${dist_dir}"
cp -L ${build_dir}/cpp/lib/libgandiva_jni.dylib ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_dataset_jni.dylib ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_orc_jni.dylib ${dist_dir}
cp -L ${install_dir}/lib/libarrow_dataset_jni.dylib ${dist_dir}
cp -L ${install_dir}/lib/libarrow_orc_jni.dylib ${dist_dir}
cp -L ${install_dir}/lib/libgandiva_jni.dylib ${dist_dir}
cp -L ${build_dir}/cpp/*/libplasma_java.dylib ${dist_dir}

echo "=== Checking shared dependencies for libraries ==="

Expand All@@ -122,9 +135,11 @@ archery linking check-dependencies \
 --allow libcurl \
 --allow libgandiva_jni \
 --allow libncurses \
 --allow libplasma_java \
 --allow libz \
 libarrow_cdata_jni.dylib \
 libarrow_dataset_jni.dylib \
 libarrow_orc_jni.dylib \
 libgandiva_jni.dylib
 libgandiva_jni.dylib \
 libplasma_java.dylib
popd

26 changes: 20 additions & 6 deletions
ci/scripts/java_jni_manylinux_build.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -32,6 +32,7 @@ echo "=== Building Arrow C++ libraries ==="
devtoolset_version=$(rpm -qa "devtoolset-*-gcc" --queryformat %{VERSION} | \
 grep -o "^[0-9]*")
devtoolset_include_cpp="/opt/rh/devtoolset-${devtoolset_version}/root/usr/include/c++/${devtoolset_version}"
: ${ARROW_BUILD_TESTS:=OFF}
: ${ARROW_DATASET:=ON}
: ${ARROW_GANDIVA:=ON}
: ${ARROW_GANDIVA_JAVA:=ON}
Expand All@@ -44,14 +45,19 @@ devtoolset_include_cpp="/opt/rh/devtoolset-${devtoolset_version}/root/usr/includ
: ${ARROW_PLASMA_JAVA_CLIENT:=ON}
: ${ARROW_PYTHON:=OFF}
: ${ARROW_S3:=ON}
: ${ARROW_BUILD_TESTS:=OFF}
: ${ARROW_USE_CCACHE:=OFF}
: ${CMAKE_BUILD_TYPE:=Release}
: ${CMAKE_UNITY_BUILD:=ON}
: ${VCPKG_ROOT:=/opt/vcpkg}
: ${VCPKG_FEATURE_FLAGS:=-manifests}
: ${VCPKG_TARGET_TRIPLET:=${VCPKG_DEFAULT_TRIPLET:-x64-linux-static-${CMAKE_BUILD_TYPE}}}
: ${GANDIVA_CXX_FLAGS:=-isystem;${devtoolset_include_cpp};-isystem;${devtoolset_include_cpp}/x86_64-redhat-linux;-isystem;-lpthread}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics before build ==="
 ccache -s
fi

export ARROW_TEST_DATA="${arrow_dir}/testing/data"
export PARQUET_TEST_DATA="${arrow_dir}/cpp/submodules/parquet-testing/data"
export AWS_EC2_METADATA_DISABLED=TRUE
Expand DownExpand Up@@ -87,11 +93,12 @@ cmake \
 -DARROW_S3=${ARROW_S3} \
 -DARROW_SNAPPY_USE_SHARED=OFF \
 -DARROW_THRIFT_USE_SHARED=OFF \
 -DARROW_USE_CCACHE=${ARROW_USE_CCACHE} \
 -DARROW_UTF8PROC_USE_SHARED=OFF \
 -DARROW_ZSTD_USE_SHARED=OFF \
 -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
 -DCMAKE_INSTALL_LIBDIR=lib \
 -DCMAKE_INSTALL_PREFIX=${build_dir}/cpp \
 -DCMAKE_INSTALL_PREFIX=${ARROW_HOME} \
 -DCMAKE_UNITY_BUILD=${CMAKE_UNITY_BUILD} \
 -DORC_SOURCE=BUNDLED \
 -DORC_PROTOBUF_EXECUTABLE=${VCPKG_ROOT}/installed/${VCPKG_TARGET_TRIPLET}/tools/protobuf/protoc \
Expand DownExpand Up@@ -126,11 +133,17 @@ ${arrow_dir}/ci/scripts/java_jni_build.sh \
 ${build_dir} \
 ${dist_dir}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics after build ==="
 ccache -s
fi


echo "=== Copying libraries to the distribution folder ==="
cp -L ${build_dir}/cpp/lib/libgandiva_jni.so ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_dataset_jni.so ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_orc_jni.so ${dist_dir}
cp -L ${ARROW_HOME}/lib/libarrow_dataset_jni.so ${dist_dir}
cp -L ${ARROW_HOME}/lib/libarrow_orc_jni.so ${dist_dir}
cp -L ${ARROW_HOME}/lib/libgandiva_jni.so ${dist_dir}
cp -L ${build_dir}/cpp/*/libplasma_java.so ${dist_dir}

echo "=== Checking shared dependencies for libraries ==="

Expand All@@ -149,5 +162,6 @@ archery linking check-dependencies \
 libarrow_cdata_jni.so \
 libarrow_dataset_jni.so \
 libarrow_orc_jni.so \
 libgandiva_jni.so
 libgandiva_jni.so \
 libplasma_java.so
popd

9 changes: 6 additions & 3 deletions
ci/scripts/java_test.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -18,9 +18,12 @@

set -ex

if [[ "${ARROW_JAVA_TEST:-ON}" != "ON" ]]; then
 exit
fi

arrow_dir=${1}
source_dir=${1}/java
cpp_build_dir=${2}/cpp/${ARROW_BUILD_TYPE:-debug}
java_jni_dist_dir=${3}

# For JNI and Plasma tests
Expand All@@ -36,7 +39,7 @@ pushd ${source_dir}
${mvn} test

if [ "${ARROW_JNI}" = "ON" ]; then
 ${mvn} test -Parrow-jni -pl adapter/orc,gandiva,dataset -Darrow.cpp.build.dir=${cpp_build_dir}
 ${mvn} test -Parrow-jni -pl adapter/orc,gandiva,dataset -Darrow.cpp.build.dir=${java_jni_dist_dir}
fi

if [ "${ARROW_JAVA_CDATA}" = "ON" ]; then
Expand All@@ -46,7 +49,7 @@ fi
if [ "${ARROW_PLASMA}" = "ON" ]; then
 pushd ${source_dir}/plasma
 java -cp target/test-classes:target/classes \
 -Djava.library.path=${cpp_build_dir} \
 -Djava.library.path=${java_jni_dist_dir} \
 org.apache.arrow.plasma.PlasmaClientTest
 popd
fi
Expand Down

10 changes: 5 additions & 5 deletions
cpp/src/plasma/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -185,14 +185,14 @@ if(ARROW_PLASMA_JAVA_CLIENT)

 if(APPLE)
 target_link_libraries(plasma_java
plasma_shared
 ${PLASMA_LINK_LIBS}
plasma_static
 ${PLASMA_STATIC_LINK_LIBS}
 "-undefined dynamic_lookup"
 ${PTHREAD_LIBRARY})
 else(APPLE)
 target_link_libraries(plasma_java plasma_shared ${PLASMA_LINK_LIBS}
 else()
 target_link_libraries(plasma_java plasma_static ${PLASMA_STATIC_LINK_LIBS}
 ${PTHREAD_LIBRARY})
 endif(APPLE)
 endif()
endif()
#
# Unit tests
Expand Down

11 changes: 10 additions & 1 deletion
dev/tasks/java-jars/github.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -28,7 +28,11 @@ jobs:
 {{ macros.github_checkout_arrow()|indent }}
 {{ macros.github_install_archery()|indent }}
 - name: Build C++ Libs
 run: archery docker run java-jni-manylinux-2014
 run: |
 archery docker run \
 -e ARROW_JAVA_BUILD=OFF \
 -e ARROW_JAVA_TEST=OFF \
 java-jni-manylinux-2014
 - name: Compress into single artifact to keep directory structure
 run: tar -cvzf arrow-shared-libs-linux.tar.gz arrow/java-dist/
 - name: Upload Artifacts
Expand DownExpand Up@@ -61,6 +65,9 @@ jobs:
 # aws-sdk-cpp and bundled aws-sdk-cpp. We uninstall Homebrew's
 # aws-sdk-cpp to ensure using only bundled aws-sdk-cpp.
 brew uninstall aws-sdk-cpp
 - name: Setup ccache
 run: |
 arrow/ci/scripts/ccache_setup.sh
 - name: Build C++ Libs
 run: |
 set -e
Expand DownExpand Up@@ -100,11 +107,13 @@ jobs:
 test -f arrow/java-dist/libarrow_cdata_jni.dylib
 test -f arrow/java-dist/libarrow_dataset_jni.dylib
 test -f arrow/java-dist/libgandiva_jni.dylib
 test -f arrow/java-dist/libplasma_java.dylib
 test -f arrow/java-dist/libarrow_orc_jni.dylib
 test -f arrow/java-dist/libarrow_cdata_jni.so
 test -f arrow/java-dist/libarrow_dataset_jni.so
 test -f arrow/java-dist/libarrow_orc_jni.so
 test -f arrow/java-dist/libgandiva_jni.so
 test -f arrow/java-dist/libplasma_java.so
 - name: Build Bundled Jar
 run: |
 set -e
Expand Down

40 changes: 6 additions & 34 deletions
docker-compose.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -122,7 +122,6 @@ x-hierarchy:
 - debian-go-cgo
 - debian-go-cgo-python
 - debian-java
 - debian-java-jni
 - debian-js
 - fedora-cpp:
 - fedora-python
Expand DownExpand Up@@ -999,10 +998,14 @@ services:
 <<: *ccache
 volumes:
 - .:/arrow:delegated
 - ${DOCKER_VOLUME_PREFIX}maven-cache:/root/.m2:delegated
 - ${DOCKER_VOLUME_PREFIX}python-wheel-manylinux2014-ccache:/ccache:delegated
 command:
 ["pip install -e /arrow/dev/archery &&
 /arrow/ci/scripts/java_jni_manylinux_build.sh /arrow /build /arrow/java-dist"]
 ["pip install -e /arrow/dev/archery && \
 /arrow/ci/scripts/java_jni_manylinux_build.sh /arrow /build /arrow/java-dist && \
 source /opt/rh/rh-maven35/enable && \
 /arrow/ci/scripts/java_build.sh /arrow /build /arrow/java-dist && \
 /arrow/ci/scripts/java_test.sh /arrow /build /arrow/java-dist"]

 ############################## Integration #################################

Expand DownExpand Up@@ -1541,37 +1544,6 @@ services:
 /arrow/ci/scripts/java_build.sh /arrow /build &&
 /arrow/ci/scripts/java_test.sh /arrow /build"

 debian-java-jni:
 # Includes plasma test, jni for gandiva and orc, and C data interface.
 # Usage:
 # docker-compose build debian-java
 # docker-compose build debian-java-jni
 # docker-compose run debian-java-jni
 image: ${REPO}:${ARCH}-debian-9-java-${JDK}-maven-${MAVEN}-jni
 build:
 context: .
 dockerfile: ci/docker/linux-apt-jni.dockerfile
 cache_from:
 - ${REPO}:${ARCH}-debian-9-java-${JDK}-maven-${MAVEN}-jni
 args:
 llvm: ${LLVM}
 shm_size: *shm-size
 environment:
 <<: *ccache
 ARROW_BUILD_TESTS: "OFF"
 ARROW_S3: "OFF"
 ARROW_SUBSTRAIT: "OFF"
@pitroupitrouAug 19, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you reuse these environment variables in the java-manylinux build?
@koukouAug 19, 2022•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use ARROW_BUILD_TESTS=OFF and ARROW_SUBSTRAIT=OFF and they are already used.

(But I want to enable ARROW_BUILD_TESTS in ARROW-17081 / #13911 to run a Dataset JNI test that uses arrow_testing.)
We can't use ARROW_S3=OFF because our JNI packages enable S3 recently: ARROW-16584 / #13157
 volumes:
 - .:/arrow:delegated
 - ${DOCKER_VOLUME_PREFIX}maven-cache:/root/.m2:delegated
 - ${DOCKER_VOLUME_PREFIX}debian-ccache:/ccache:delegated
 command:
 /bin/bash -c "
 /arrow/ci/scripts/cpp_build.sh /arrow /build &&
 /arrow/ci/scripts/java_jni_build.sh /arrow /build /tmp/java_dist &&
 /arrow/ci/scripts/java_build.sh /arrow /build /tmp/java_dist &&
 /arrow/ci/scripts/java_test.sh /arrow /build /tmp/java_dist"

 oracle-java:
 # Usage:
 # docker-compose build oracle-java
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
ARROW-17451: [CI][Java] Use manylinux2014 image for JNI#13920

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

kou
merged 6 commits into

from

Aug 23, 2022
 Merged


ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
#13920
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
b319661

ARROW-17451: [CI][Java] Use manylinux2014 image for JNI


kou Aug 18, 2022
36b05d1

Add missing libplasma_java


kou Aug 19, 2022
31a65eb

Show ccache statistics


kou Aug 19, 2022
3e7930d

Add missing ccache related configurations


kou Aug 20, 2022
64cf4c7

Fix path


kou Aug 20, 2022
489f84e

Fix cache key


kou Aug 20, 2022

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


12 changes: 6 additions & 6 deletions
.github/workflows/java_jni.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -47,7 +47,7 @@ env:
jobs:

 docker:
 name: AMD64 Debian 9 Java JNI (Gandiva, Plasma, ORC, Dataset)
 name: AMD64 manylinux2014 Java JNI
 runs-on: ubuntu-latest
 if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
 timeout-minutes: 90
Expand All@@ -61,23 +61,23 @@ jobs:
 uses: actions/cache@v2
 with:
 path: .docker
 key: maven-${{ hashFiles('java/**') }}
 restore-keys: maven-
 key: java-jni-manylinux-2014-${{ hashFiles('cpp/**', 'java/**') }}
 restore-keys: java-jni-manylinux-2014-
 - name: Setup Python
 uses: actions/setup-python@v4
 with:
 python-version: 3.8
 - name: Setup Archery
 run: pip install -e dev/archery[docker]
 - name: Execute Docker Build
 run: archery docker run debian-java-jni
 run: archery docker run java-jni-manylinux-2014
 - name: Docker Push
 if: success() && github.event_name == 'push' && github.repository == 'apache/arrow'
 continue-on-error: true
 run: archery docker push debian-java-jni
 run: archery docker push java-jni-manylinux-2014

 docker_integration_python:
 name: AMD64 Debian 9 Java C Data Interface Integration
 name: AMD64 Conda Java C Data Interface Integration
 runs-on: ubuntu-latest
 if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
 timeout-minutes: 90
Expand Down

10 changes: 9 additions & 1 deletion
ci/docker/java-jni-manylinux-201x.dockerfile

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -33,5 +33,13 @@ RUN vcpkg install \

# Install Java
ARG java=1.8.0
RUN yum install -y java-$java-openjdk-devel && yum clean all
RUN yum install -y java-$java-openjdk-devel rh-maven35 && yum clean all
ENV JAVA_HOME=/usr/lib/jvm/java-$java-openjdk/

# For ci/scripts/java_*.sh
ENV ARROW_GANDIVA_JAVA=ON \
 ARROW_HOME=/tmp/local \
 ARROW_JAVA_CDATA=ON \
 ARROW_JNI=ON \
 ARROW_PLASMA=ON \
 ARROW_USE_CCACHE=ON

7 changes: 5 additions & 2 deletions
ci/scripts/java_build.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -18,10 +18,13 @@

set -ex

if [[ "${ARROW_JAVA_BUILD:-ON}" != "ON" ]]; then
 exit
fi

arrow_dir=${1}
source_dir=${1}/java
build_dir=${2}
cpp_build_dir=${build_dir}/cpp/${ARROW_BUILD_TYPE:-debug}
java_jni_dist_dir=${3}

: ${BUILD_DOCS_JAVA:=OFF}
Expand DownExpand Up@@ -80,7 +83,7 @@ if [ "${ARROW_JAVA_CDATA}" = "ON" ]; then
fi

if [ "${ARROW_GANDIVA_JAVA}" = "ON" ]; then
 ${mvn} -Darrow.cpp.build.dir=${cpp_build_dir} -Parrow-jni install
 ${mvn} -Darrow.cpp.build.dir=${java_jni_dist_dir} -Parrow-jni install
fi

if [ "${ARROW_PLASMA}" = "ON" ]; then
Expand Down

25 changes: 20 additions & 5 deletions
ci/scripts/java_jni_macos_build.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -29,6 +29,7 @@ echo "=== Clear output directories and leftovers ==="
rm -rf ${build_dir}

echo "=== Building Arrow C++ libraries ==="
install_dir=${build_dir}/cpp-install
: ${ARROW_BUILD_TESTS:=OFF}
: ${ARROW_DATASET:=ON}
: ${ARROW_FILESYSTEM:=ON}
Expand All@@ -40,9 +41,15 @@ echo "=== Building Arrow C++ libraries ==="
: ${ARROW_PLASMA:=ON}
: ${ARROW_PYTHON:=OFF}
: ${ARROW_S3:=ON}
: ${ARROW_USE_CCACHE:=OFF}
@pitroupitrouAug 19, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not, but you should perhaps also pass it to CMake below?
@koukouAug 20, 2022

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I changed wrong file... I should have changed java_jni_manylinux_build.sh...
I've passed it to CMake arguments. I've updated java_jni_manylinux_build.sh too.
I chose ARROW_USE_CCACHE:=OFF here because cpp_build.sh uses it. I set ARROW_USE_CCACHE=ON in caller like other jobs.
: ${CMAKE_BUILD_TYPE:=Release}
: ${CMAKE_UNITY_BUILD:=ON}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics before build ==="
 ccache -s
fi

export ARROW_TEST_DATA="${arrow_dir}/testing/data"
export PARQUET_TEST_DATA="${arrow_dir}/cpp/submodules/parquet-testing/data"
export AWS_EC2_METADATA_DISABLED=TRUE
Expand DownExpand Up@@ -75,12 +82,13 @@ cmake \
 -DARROW_S3=${ARROW_S3} \
 -DARROW_SNAPPY_USE_SHARED=OFF \
 -DARROW_THRIFT_USE_SHARED=OFF \
 -DARROW_USE_CCACHE=${ARROW_USE_CCACHE} \
 -DARROW_UTF8PROC_USE_SHARED=OFF \
 -DARROW_ZSTD_USE_SHARED=OFF \
 -DAWSSDK_SOURCE=BUNDLED \
 -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
 -DCMAKE_INSTALL_LIBDIR=lib \
 -DCMAKE_INSTALL_PREFIX=${build_dir}/cpp \
 -DCMAKE_INSTALL_PREFIX=${install_dir} \
 -DCMAKE_UNITY_BUILD=${CMAKE_UNITY_BUILD} \
 -DPARQUET_BUILD_EXAMPLES=OFF \
 -DPARQUET_BUILD_EXECUTABLES=OFF \
Expand All@@ -102,12 +110,17 @@ ${arrow_dir}/ci/scripts/java_jni_build.sh \
 ${build_dir} \
 ${dist_dir}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics after build ==="
 ccache -s
fi

echo "=== Copying libraries to the distribution folder ==="
mkdir -p "${dist_dir}"
cp -L ${build_dir}/cpp/lib/libgandiva_jni.dylib ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_dataset_jni.dylib ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_orc_jni.dylib ${dist_dir}
cp -L ${install_dir}/lib/libarrow_dataset_jni.dylib ${dist_dir}
cp -L ${install_dir}/lib/libarrow_orc_jni.dylib ${dist_dir}
cp -L ${install_dir}/lib/libgandiva_jni.dylib ${dist_dir}
cp -L ${build_dir}/cpp/*/libplasma_java.dylib ${dist_dir}

echo "=== Checking shared dependencies for libraries ==="

Expand All@@ -122,9 +135,11 @@ archery linking check-dependencies \
 --allow libcurl \
 --allow libgandiva_jni \
 --allow libncurses \
 --allow libplasma_java \
 --allow libz \
 libarrow_cdata_jni.dylib \
 libarrow_dataset_jni.dylib \
 libarrow_orc_jni.dylib \
 libgandiva_jni.dylib
 libgandiva_jni.dylib \
 libplasma_java.dylib
popd

26 changes: 20 additions & 6 deletions
ci/scripts/java_jni_manylinux_build.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -32,6 +32,7 @@ echo "=== Building Arrow C++ libraries ==="
devtoolset_version=$(rpm -qa "devtoolset-*-gcc" --queryformat %{VERSION} | \
 grep -o "^[0-9]*")
devtoolset_include_cpp="/opt/rh/devtoolset-${devtoolset_version}/root/usr/include/c++/${devtoolset_version}"
: ${ARROW_BUILD_TESTS:=OFF}
: ${ARROW_DATASET:=ON}
: ${ARROW_GANDIVA:=ON}
: ${ARROW_GANDIVA_JAVA:=ON}
Expand All@@ -44,14 +45,19 @@ devtoolset_include_cpp="/opt/rh/devtoolset-${devtoolset_version}/root/usr/includ
: ${ARROW_PLASMA_JAVA_CLIENT:=ON}
: ${ARROW_PYTHON:=OFF}
: ${ARROW_S3:=ON}
: ${ARROW_BUILD_TESTS:=OFF}
: ${ARROW_USE_CCACHE:=OFF}
: ${CMAKE_BUILD_TYPE:=Release}
: ${CMAKE_UNITY_BUILD:=ON}
: ${VCPKG_ROOT:=/opt/vcpkg}
: ${VCPKG_FEATURE_FLAGS:=-manifests}
: ${VCPKG_TARGET_TRIPLET:=${VCPKG_DEFAULT_TRIPLET:-x64-linux-static-${CMAKE_BUILD_TYPE}}}
: ${GANDIVA_CXX_FLAGS:=-isystem;${devtoolset_include_cpp};-isystem;${devtoolset_include_cpp}/x86_64-redhat-linux;-isystem;-lpthread}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics before build ==="
 ccache -s
fi

export ARROW_TEST_DATA="${arrow_dir}/testing/data"
export PARQUET_TEST_DATA="${arrow_dir}/cpp/submodules/parquet-testing/data"
export AWS_EC2_METADATA_DISABLED=TRUE
Expand DownExpand Up@@ -87,11 +93,12 @@ cmake \
 -DARROW_S3=${ARROW_S3} \
 -DARROW_SNAPPY_USE_SHARED=OFF \
 -DARROW_THRIFT_USE_SHARED=OFF \
 -DARROW_USE_CCACHE=${ARROW_USE_CCACHE} \
 -DARROW_UTF8PROC_USE_SHARED=OFF \
 -DARROW_ZSTD_USE_SHARED=OFF \
 -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
 -DCMAKE_INSTALL_LIBDIR=lib \
 -DCMAKE_INSTALL_PREFIX=${build_dir}/cpp \
 -DCMAKE_INSTALL_PREFIX=${ARROW_HOME} \
 -DCMAKE_UNITY_BUILD=${CMAKE_UNITY_BUILD} \
 -DORC_SOURCE=BUNDLED \
 -DORC_PROTOBUF_EXECUTABLE=${VCPKG_ROOT}/installed/${VCPKG_TARGET_TRIPLET}/tools/protobuf/protoc \
Expand DownExpand Up@@ -126,11 +133,17 @@ ${arrow_dir}/ci/scripts/java_jni_build.sh \
 ${build_dir} \
 ${dist_dir}

if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
 echo "=== ccache statistics after build ==="
 ccache -s
fi


echo "=== Copying libraries to the distribution folder ==="
cp -L ${build_dir}/cpp/lib/libgandiva_jni.so ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_dataset_jni.so ${dist_dir}
cp -L ${build_dir}/cpp/lib/libarrow_orc_jni.so ${dist_dir}
cp -L ${ARROW_HOME}/lib/libarrow_dataset_jni.so ${dist_dir}
cp -L ${ARROW_HOME}/lib/libarrow_orc_jni.so ${dist_dir}
cp -L ${ARROW_HOME}/lib/libgandiva_jni.so ${dist_dir}
cp -L ${build_dir}/cpp/*/libplasma_java.so ${dist_dir}

echo "=== Checking shared dependencies for libraries ==="

Expand All@@ -149,5 +162,6 @@ archery linking check-dependencies \
 libarrow_cdata_jni.so \
 libarrow_dataset_jni.so \
 libarrow_orc_jni.so \
 libgandiva_jni.so
 libgandiva_jni.so \
 libplasma_java.so
popd

9 changes: 6 additions & 3 deletions
ci/scripts/java_test.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -18,9 +18,12 @@

set -ex

if [[ "${ARROW_JAVA_TEST:-ON}" != "ON" ]]; then
 exit
fi

arrow_dir=${1}
source_dir=${1}/java
cpp_build_dir=${2}/cpp/${ARROW_BUILD_TYPE:-debug}
java_jni_dist_dir=${3}

# For JNI and Plasma tests
Expand All@@ -36,7 +39,7 @@ pushd ${source_dir}
${mvn} test

if [ "${ARROW_JNI}" = "ON" ]; then
 ${mvn} test -Parrow-jni -pl adapter/orc,gandiva,dataset -Darrow.cpp.build.dir=${cpp_build_dir}
 ${mvn} test -Parrow-jni -pl adapter/orc,gandiva,dataset -Darrow.cpp.build.dir=${java_jni_dist_dir}
fi

if [ "${ARROW_JAVA_CDATA}" = "ON" ]; then
Expand All@@ -46,7 +49,7 @@ fi
if [ "${ARROW_PLASMA}" = "ON" ]; then
 pushd ${source_dir}/plasma
 java -cp target/test-classes:target/classes \
 -Djava.library.path=${cpp_build_dir} \
 -Djava.library.path=${java_jni_dist_dir} \
 org.apache.arrow.plasma.PlasmaClientTest
 popd
fi
Expand Down

10 changes: 5 additions & 5 deletions
cpp/src/plasma/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -185,14 +185,14 @@ if(ARROW_PLASMA_JAVA_CLIENT)

 if(APPLE)
 target_link_libraries(plasma_java
plasma_shared
 ${PLASMA_LINK_LIBS}
plasma_static
 ${PLASMA_STATIC_LINK_LIBS}
 "-undefined dynamic_lookup"
 ${PTHREAD_LIBRARY})
 else(APPLE)
 target_link_libraries(plasma_java plasma_shared ${PLASMA_LINK_LIBS}
 else()
 target_link_libraries(plasma_java plasma_static ${PLASMA_STATIC_LINK_LIBS}
 ${PTHREAD_LIBRARY})
 endif(APPLE)
 endif()
endif()
#
# Unit tests
Expand Down

11 changes: 10 additions & 1 deletion
dev/tasks/java-jars/github.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -28,7 +28,11 @@ jobs:
 {{ macros.github_checkout_arrow()|indent }}
 {{ macros.github_install_archery()|indent }}
 - name: Build C++ Libs
 run: archery docker run java-jni-manylinux-2014
 run: |
 archery docker run \
 -e ARROW_JAVA_BUILD=OFF \
 -e ARROW_JAVA_TEST=OFF \
 java-jni-manylinux-2014
 - name: Compress into single artifact to keep directory structure
 run: tar -cvzf arrow-shared-libs-linux.tar.gz arrow/java-dist/
 - name: Upload Artifacts
Expand DownExpand Up@@ -61,6 +65,9 @@ jobs:
 # aws-sdk-cpp and bundled aws-sdk-cpp. We uninstall Homebrew's
 # aws-sdk-cpp to ensure using only bundled aws-sdk-cpp.
 brew uninstall aws-sdk-cpp
 - name: Setup ccache
 run: |
 arrow/ci/scripts/ccache_setup.sh
 - name: Build C++ Libs
 run: |
 set -e
Expand DownExpand Up@@ -100,11 +107,13 @@ jobs:
 test -f arrow/java-dist/libarrow_cdata_jni.dylib
 test -f arrow/java-dist/libarrow_dataset_jni.dylib
 test -f arrow/java-dist/libgandiva_jni.dylib
 test -f arrow/java-dist/libplasma_java.dylib
 test -f arrow/java-dist/libarrow_orc_jni.dylib
 test -f arrow/java-dist/libarrow_cdata_jni.so
 test -f arrow/java-dist/libarrow_dataset_jni.so
 test -f arrow/java-dist/libarrow_orc_jni.so
 test -f arrow/java-dist/libgandiva_jni.so
 test -f arrow/java-dist/libplasma_java.so
 - name: Build Bundled Jar
 run: |
 set -e
Expand Down

40 changes: 6 additions & 34 deletions
docker-compose.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -122,7 +122,6 @@ x-hierarchy:
 - debian-go-cgo
 - debian-go-cgo-python
 - debian-java
 - debian-java-jni
 - debian-js
 - fedora-cpp:
 - fedora-python
Expand DownExpand Up@@ -999,10 +998,14 @@ services:
 <<: *ccache
 volumes:
 - .:/arrow:delegated
 - ${DOCKER_VOLUME_PREFIX}maven-cache:/root/.m2:delegated
 - ${DOCKER_VOLUME_PREFIX}python-wheel-manylinux2014-ccache:/ccache:delegated
 command:
 ["pip install -e /arrow/dev/archery &&
 /arrow/ci/scripts/java_jni_manylinux_build.sh /arrow /build /arrow/java-dist"]
 ["pip install -e /arrow/dev/archery && \
 /arrow/ci/scripts/java_jni_manylinux_build.sh /arrow /build /arrow/java-dist && \
 source /opt/rh/rh-maven35/enable && \
 /arrow/ci/scripts/java_build.sh /arrow /build /arrow/java-dist && \
 /arrow/ci/scripts/java_test.sh /arrow /build /arrow/java-dist"]

 ############################## Integration #################################

Expand DownExpand Up@@ -1541,37 +1544,6 @@ services:
 /arrow/ci/scripts/java_build.sh /arrow /build &&
 /arrow/ci/scripts/java_test.sh /arrow /build"

 debian-java-jni:
 # Includes plasma test, jni for gandiva and orc, and C data interface.
 # Usage:
 # docker-compose build debian-java
 # docker-compose build debian-java-jni
 # docker-compose run debian-java-jni
 image: ${REPO}:${ARCH}-debian-9-java-${JDK}-maven-${MAVEN}-jni
 build:
 context: .
 dockerfile: ci/docker/linux-apt-jni.dockerfile
 cache_from:
 - ${REPO}:${ARCH}-debian-9-java-${JDK}-maven-${MAVEN}-jni
 args:
 llvm: ${LLVM}
 shm_size: *shm-size
 environment:
 <<: *ccache
 ARROW_BUILD_TESTS: "OFF"
 ARROW_S3: "OFF"
 ARROW_SUBSTRAIT: "OFF"
@pitroupitrouAug 19, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you reuse these environment variables in the java-manylinux build?
@koukouAug 19, 2022•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use ARROW_BUILD_TESTS=OFF and ARROW_SUBSTRAIT=OFF and they are already used.

(But I want to enable ARROW_BUILD_TESTS in ARROW-17081 / #13911 to run a Dataset JNI test that uses arrow_testing.)
We can't use ARROW_S3=OFF because our JNI packages enable S3 recently: ARROW-16584 / #13157
 volumes:
 - .:/arrow:delegated
 - ${DOCKER_VOLUME_PREFIX}maven-cache:/root/.m2:delegated
 - ${DOCKER_VOLUME_PREFIX}debian-ccache:/ccache:delegated
 command:
 /bin/bash -c "
 /arrow/ci/scripts/cpp_build.sh /arrow /build &&
 /arrow/ci/scripts/java_jni_build.sh /arrow /build /tmp/java_dist &&
 /arrow/ci/scripts/java_build.sh /arrow /build /tmp/java_dist &&
 /arrow/ci/scripts/java_test.sh /arrow /build /tmp/java_dist"

 oracle-java:
 # Usage:
 # docker-compose build oracle-java
Expand Down