[R] Linux arm64 binary hosting

[eitsupi]

Describe the enhancement requested

It would be great to speed up the installation of the arrow package in R on arm64 Linux if the following script could be adapted to arm64 Linux.

arrow/r/tools/nixlibs.R

Lines 154 to 155 in 72343f5

	if (identical(os, "linux") && identical(arch, "x86_64")) {
	# We only host x86 linux binaries today

Component(s)

Packaging, R

[paleolimbot]

Just a note that I've also wished this in the past...arm64 Linux is what you get when using Docker on MacOS M1 and is useful for doing things like local testing with valgrind.

[eitsupi]

I am curious about this, where are the jobs defined that build existing amd64 binaries?

(As a side note, I have experienced that when I tried to install the arrow package on a Raspberry Pi 4 with 4GB RAM, the build stopped halfway through due to lack of machine power......)

[paleolimbot]

I don't think we currently build arm binaries for R at all (although I think there is an M1 Mac nightly build). I think that Apache recently made some Linux arm runners available (but I'm not familiar with the details).

[eitsupi]

I don't think we currently build arm binaries for R at all (although I think there is an M1 Mac nightly build). I think that Apache recently made some Linux arm runners available (but I'm not familiar with the details).

Oh, sorry.
My question was to know about the definition of a job that is building amd64 binaries.

[paleolimbot]

I think it happens in a few places:

• https://github.com/apache/arrow/blob/main/dev/tasks/tasks.yml#L922-L938
• https://github.com/apache/arrow/blob/main/dev/tasks/r/github.packages.yml

@assignUser is much more likely to get the details right as to whether or not this is easy to do!

[assignUser]

We have arm runners now so this should be doable with some additions.to the nixlibs script.

@raulcd was looking into cross compiling for big endian/s390x and mentioned that this might also be applicable for rpi etc.

[kou]

@eitsupi If you want to work on this, the following steps will be good:

1. Run ARCH=arm64v8 UBUNTU=22.04 archery docker run ubuntu-cpp-static on local
2. Add a job to build binary package for 1.:

   diff --git a/dev/tasks/r/github.packages.yml b/dev/tasks/r/github.packages.yml
   index e3e3d34e1..2b89a27a5 100644
   --- a/dev/tasks/r/github.packages.yml
   +++ b/dev/tasks/r/github.packages.yml
   @@ -58,21 +58,30 @@ jobs:
   
    linux-cpp:
   name: C++ Binary Linux OpenSSL {{ '${{ matrix.openssl }}' }}
   - runs-on: ubuntu-latest
   + runs-on: {{ '${{ matrix.runs-on }}' }}
   needs: source
   strategy:
    fail-fast: false
    matrix:
    include:
   - - openssl: "3.0"
   + - arch: amd64
   + openssl: "3.0"
   + os: ubuntu
   + runs-on: ubuntu-latest
   + ubuntu: "22.04"
   + - arch: arm64
   + openssl: "3.0"
   	os: ubuntu
   + runs-on: ["self-hosted", "Linux", "arm64"]
   	ubuntu: "22.04"
   - - extra-cmake-flags: >-
   + - arch: amd64
   + extra-cmake-flags: >-
   	 -DCMAKE_INCLUDE_PATH=/usr/include/openssl11
   	 -DCMAKE_LIBRARY_PATH=/usr/lib64/openssl11
   	openssl: "1.1"
   	os: centos
   - - openssl: "1.0"
   + - arch: amd64
   + openssl: "1.0"
   	os: centos
   steps:
    {{ macros.github_checkout_arrow()|indent }}
   @@ -103,7 +112,7 @@ jobs:
    - name: Upload binary artifact
    uses: actions/upload-artifact@v3
    with:
   - name: r-lib__libarrow__bin__linux-openssl-{{ '${{ matrix.openssl }}' }}
   + name: r-lib__libarrow__bin__linux-openssl-{{ '${{ matrix.openssl }}' }}__{{ '${{ matrix.arch }}' }}
    path: arrow/r/libarrow/dist/arrow-*.zip
   
    windows-cpp:

   We can run it on a PR by @github-actions crossbow submit r-binary-packages.
   We may need to change r/tools/nixlibs.R something like:

   diff --git a/r/tools/nixlibs.R b/r/tools/nixlibs.R
   index f8e42ecea..aa2729d3e 100644
   --- a/r/tools/nixlibs.R
   +++ b/r/tools/nixlibs.R
   @@ -87,7 +87,7 @@ thirdparty_dependency_dir <- Sys.getenv("ARROW_THIRDPARTY_DEPENDENCY_DIR", "tool
   
    download_binary <- function(lib) {
    libfile <- tempfile()
   - binary_url <- paste0(arrow_repo, "bin/", lib, "/arrow-", VERSION, ".zip")
   + binary_url <- paste0(arrow_repo, "bin/", lib, "/", detect_arch(), "/arrow-", VERSION, ".zip")
    if (try_download(binary_url, libfile)) {
    if (!quietly) {
    cat(sprintf("*** Successfully retrieved C++ binaries (%s)\n", lib))

3. Add a test job for the binary package to dev/tasks/r/github.packages.yml.
4. (We may want to add more patterns for old Linux distributions such as CentOS/AlmaLinux/Ubuntu < 22.04.)