GH-34843: [R] Fix R build failed caused by Acero refactor by icexelloss · Pull Request #34844 · apache/arrow

icexelloss · 2023-04-02T16:02:43Z

Rationale for this change

There was an issue with window R build introduced by this PR:
#34711

What changes are included in this PR?

Many changes to the build files

Are these changes tested?

crossbow builds

Are there any user-facing changes?

Closes: [R] Windows R build fails #34843

github-actions · 2023-04-02T16:03:04Z

Closes: [R] Windows R build fails #34843

github-actions · 2023-04-02T16:03:06Z

⚠️ GitHub issue #34843has been automatically assigned in GitHub to PR creator.

icexelloss · 2023-04-02T16:30:21Z

@github-actions crossbow submit -g r -g cpp

github-actions · 2023-04-02T16:35:32Z

Revision: db6b52b

Submitted crossbow builds: ursacomputing/crossbow @ actions-315577d6e3

Task	Status
conda-linux-aarch64-cpu-r41
conda-linux-aarch64-cpu-r42
conda-linux-x64-cpu-r41
conda-linux-x64-cpu-r42
conda-osx-arm64-cpu-r41
conda-osx-arm64-cpu-r42
conda-osx-x64-cpu-r41
conda-osx-x64-cpu-r42
conda-win-x64-cpu-r41
homebrew-r-autobrew
homebrew-r-brew
r-binary-packages
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp
test-debian-10-cpp-amd64
test-debian-10-cpp-i386
test-debian-11-cpp-amd64
test-debian-11-cpp-i386
test-fedora-35-cpp
test-fedora-r-clang-sanitizer
test-r-arrow-backwards-compatibility
test-r-depsource-bundled
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-gcc-11
test-r-gcc-12
test-r-install-local
test-r-install-local-minsizerel
test-r-library-r-base-latest
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-valgrind
test-r-minimal-build
test-r-offline-maximal
test-r-offline-minimal
test-r-rhub-debian-gcc-devel-lto-latest
test-r-rhub-debian-gcc-release-custom-ccache
test-r-rhub-ubuntu-gcc-release-latest
test-r-rstudio-r-base-4.1-opensuse153
test-r-rstudio-r-base-4.2-centos7-devtoolset-8
test-r-rstudio-r-base-4.2-focal
test-r-ubuntu-22.04
test-r-versions
test-ubuntu-20.04-cpp
test-ubuntu-20.04-cpp-20
test-ubuntu-20.04-cpp-bundled
test-ubuntu-20.04-cpp-minimal-with-formats
test-ubuntu-20.04-cpp-thread-sanitizer
test-ubuntu-22.04-cpp
test-ubuntu-r-sanitizer

icexelloss · 2023-04-03T12:45:27Z

@github-actions crossbow submit -g r -g cpp

github-actions · 2023-04-03T12:51:42Z

Revision: 0c234bea39b22795628685dc787f73596c8a60e1

Submitted crossbow builds: ursacomputing/crossbow @ actions-eeab399aca

Task	Status
conda-linux-aarch64-cpu-r41
conda-linux-aarch64-cpu-r42
conda-linux-x64-cpu-r41
conda-linux-x64-cpu-r42
conda-osx-arm64-cpu-r41
conda-osx-arm64-cpu-r42
conda-osx-x64-cpu-r41
conda-osx-x64-cpu-r42
conda-win-x64-cpu-r41
homebrew-r-autobrew
homebrew-r-brew
r-binary-packages
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp
test-debian-10-cpp-amd64
test-debian-10-cpp-i386
test-debian-11-cpp-amd64
test-debian-11-cpp-i386
test-fedora-35-cpp
test-fedora-r-clang-sanitizer
test-r-arrow-backwards-compatibility
test-r-depsource-bundled
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-gcc-11
test-r-gcc-12
test-r-install-local
test-r-install-local-minsizerel
test-r-library-r-base-latest
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-valgrind
test-r-minimal-build
test-r-offline-maximal
test-r-offline-minimal
test-r-rhub-debian-gcc-devel-lto-latest
test-r-rhub-debian-gcc-release-custom-ccache
test-r-rhub-ubuntu-gcc-release-latest
test-r-rstudio-r-base-4.1-opensuse153
test-r-rstudio-r-base-4.2-centos7-devtoolset-8
test-r-rstudio-r-base-4.2-focal
test-r-ubuntu-22.04
test-r-versions
test-ubuntu-20.04-cpp
test-ubuntu-20.04-cpp-20
test-ubuntu-20.04-cpp-bundled
test-ubuntu-20.04-cpp-minimal-with-formats
test-ubuntu-20.04-cpp-thread-sanitizer
test-ubuntu-22.04-cpp
test-ubuntu-r-sanitizer

icexelloss · 2023-04-03T14:26:20Z

@github-actions crossbow submit -g r -g cpp

github-actions · 2023-04-03T14:32:22Z

Revision: 13de798

Submitted crossbow builds: ursacomputing/crossbow @ actions-d25bb2c4f9

Task	Status
conda-linux-aarch64-cpu-r41
conda-linux-aarch64-cpu-r42
conda-linux-x64-cpu-r41
conda-linux-x64-cpu-r42
conda-osx-arm64-cpu-r41
conda-osx-arm64-cpu-r42
conda-osx-x64-cpu-r41
conda-osx-x64-cpu-r42
conda-win-x64-cpu-r41
homebrew-r-autobrew
homebrew-r-brew
r-binary-packages
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp
test-debian-10-cpp-amd64
test-debian-10-cpp-i386
test-debian-11-cpp-amd64
test-debian-11-cpp-i386
test-fedora-35-cpp
test-fedora-r-clang-sanitizer
test-r-arrow-backwards-compatibility
test-r-depsource-bundled
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-gcc-11
test-r-gcc-12
test-r-install-local
test-r-install-local-minsizerel
test-r-library-r-base-latest
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-valgrind
test-r-minimal-build
test-r-offline-maximal
test-r-offline-minimal
test-r-rhub-debian-gcc-devel-lto-latest
test-r-rhub-debian-gcc-release-custom-ccache
test-r-rhub-ubuntu-gcc-release-latest
test-r-rstudio-r-base-4.1-opensuse153
test-r-rstudio-r-base-4.2-centos7-devtoolset-8
test-r-rstudio-r-base-4.2-focal
test-r-ubuntu-22.04
test-r-versions
test-ubuntu-20.04-cpp
test-ubuntu-20.04-cpp-20
test-ubuntu-20.04-cpp-bundled
test-ubuntu-20.04-cpp-minimal-with-formats
test-ubuntu-20.04-cpp-thread-sanitizer
test-ubuntu-22.04-cpp
test-ubuntu-r-sanitizer

icexelloss · 2023-04-03T14:41:58Z

For the mac build: https://github.com/ursacomputing/crossbow/actions/runs/4596858239/jobs/8118844443

I noticed "ARROW_ACERO" is not on from log on
(L815)

==> Installing apache-arrow-static --HEAD

(L900)

==> cmake ../cpp -DCMAKE_INSTALL_PREFIX=/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/build-apache-arrow-static/Cellar/apache-arrow-static/HEAD-0c234be -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_BUILD_TYPE=Release -DCMAKE_FIND_FRAMEWORK=LAST -DCMAKE_VERBOSE_MAKEFILE=ON -Wno-dev -DBUILD_TESTING=OFF -DCMAKE_OSX_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk -DARROW_BUILD_SHARED=OFF -DARROW_BUILD_UTILITIES=ON -DARROW_COMPUTE=ON -DARROW_CSV=ON -DARROW_DATASET=ON -DARROW_FILESYSTEM=ON -DARROW_GCS=ON -DARROW_JEMALLOC=ON -DARROW_JSON=ON -DARROW_MIMALLOC=ON -DARROW_PARQUET=ON -DARROW_S3=ON -DARROW_VERBOSE_THIRDPARTY_BUILD=ON -DARROW_WITH_BROTLI=ON -DARROW_WITH_BZ2=ON -DARROW_WITH_LZ4=ON -DARROW_WITH_SNAPPY=ON -DARROW_WITH_ZLIB=ON -DARROW_WITH_ZSTD=ON -DLZ4_HOME=/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/build-apache-arrow-static/opt/lz4 -DPARQUET_BUILD_EXECUTABLES=ON -DTHRIFT_HOME=/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/build-apache-arrow-static/opt/thrift

But I have yet to figure out where is the macOS build file for this.

icexelloss · 2023-04-03T14:45:01Z

@assignUser per comments here: #34844 (comment)

I am not too familiar with the build system - do you by chance know where the build script for "R on mac" is? (Or knows who might know that)?

assignUser · 2023-04-03T14:47:49Z

Yes they are in dev/tasks/homebrew-formulae there are multiple once for homebrew and autpbrew (the shadow version of homebrew for use on CRAN/EOL macOS)

icexelloss · 2023-04-03T15:27:22Z

@github-actions crossbow submit -g r -g cpp

icexelloss · 2023-04-03T15:29:51Z

Thanks @assignUser! Sorry to bother u with another question, looks like my attempt to fix window build is unsuccessful after passing (-larrow_acero) to the linking option. I suspect that the Arrow C++ library is not build with Acero in the window R build. Do you by any chance knows which file controls the Arrow C++ build flag in the "windows R build?

github-actions · 2023-04-03T15:32:33Z

Revision: 2d48623

Submitted crossbow builds: ursacomputing/crossbow @ actions-fdf767b324

Task	Status
conda-linux-aarch64-cpu-r41
conda-linux-aarch64-cpu-r42
conda-linux-x64-cpu-r41
conda-linux-x64-cpu-r42
conda-osx-arm64-cpu-r41
conda-osx-arm64-cpu-r42
conda-osx-x64-cpu-r41
conda-osx-x64-cpu-r42
conda-win-x64-cpu-r41
homebrew-r-autobrew
homebrew-r-brew
r-binary-packages
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp
test-debian-10-cpp-amd64
test-debian-10-cpp-i386
test-debian-11-cpp-amd64
test-debian-11-cpp-i386
test-fedora-35-cpp
test-fedora-r-clang-sanitizer
test-r-arrow-backwards-compatibility
test-r-depsource-bundled
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-gcc-11
test-r-gcc-12
test-r-install-local
test-r-install-local-minsizerel
test-r-library-r-base-latest
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-valgrind
test-r-minimal-build
test-r-offline-maximal
test-r-offline-minimal
test-r-rhub-debian-gcc-devel-lto-latest
test-r-rhub-debian-gcc-release-custom-ccache
test-r-rhub-ubuntu-gcc-release-latest
test-r-rstudio-r-base-4.1-opensuse153
test-r-rstudio-r-base-4.2-centos7-devtoolset-8
test-r-rstudio-r-base-4.2-focal
test-r-ubuntu-22.04
test-r-versions
test-ubuntu-20.04-cpp
test-ubuntu-20.04-cpp-20
test-ubuntu-20.04-cpp-bundled
test-ubuntu-20.04-cpp-minimal-with-formats
test-ubuntu-20.04-cpp-thread-sanitizer
test-ubuntu-22.04-cpp
test-ubuntu-r-sanitizer

icexelloss · 2023-04-03T15:40:29Z

@github-actions crossbow submit -g r -g cpp

github-actions · 2023-04-03T15:46:42Z

Revision: 94ab634

Submitted crossbow builds: ursacomputing/crossbow @ actions-6f5022991e

Task	Status
conda-linux-aarch64-cpu-r41
conda-linux-aarch64-cpu-r42
conda-linux-x64-cpu-r41
conda-linux-x64-cpu-r42
conda-osx-arm64-cpu-r41
conda-osx-arm64-cpu-r42
conda-osx-x64-cpu-r41
conda-osx-x64-cpu-r42
conda-win-x64-cpu-r41
homebrew-r-autobrew
homebrew-r-brew
r-binary-packages
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp
test-debian-10-cpp-amd64
test-debian-10-cpp-i386
test-debian-11-cpp-amd64
test-debian-11-cpp-i386
test-fedora-35-cpp
test-fedora-r-clang-sanitizer
test-r-arrow-backwards-compatibility
test-r-depsource-bundled
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-gcc-11
test-r-gcc-12
test-r-install-local
test-r-install-local-minsizerel
test-r-library-r-base-latest
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-valgrind
test-r-minimal-build
test-r-offline-maximal
test-r-offline-minimal
test-r-rhub-debian-gcc-devel-lto-latest
test-r-rhub-debian-gcc-release-custom-ccache
test-r-rhub-ubuntu-gcc-release-latest
test-r-rstudio-r-base-4.1-opensuse153
test-r-rstudio-r-base-4.2-centos7-devtoolset-8
test-r-rstudio-r-base-4.2-focal
test-r-ubuntu-22.04
test-r-versions
test-ubuntu-20.04-cpp
test-ubuntu-20.04-cpp-20
test-ubuntu-20.04-cpp-bundled
test-ubuntu-20.04-cpp-minimal-with-formats
test-ubuntu-20.04-cpp-thread-sanitizer
test-ubuntu-22.04-cpp
test-ubuntu-r-sanitizer

icexelloss · 2023-04-03T16:24:02Z

cc @westonpace - add u as collaborator to my repo - feel free to push

westonpace · 2023-04-03T16:31:16Z

Ok. I'm reproducing the test-r-minimal-build failure now. I'll try pulling these changes and then see what else needs done.

assignUser · 2023-04-03T17:26:39Z

@icexelloss it looks like you need to add these changes to ci/scripts/r_windows_build.sh and ci/scripts/PKGBUILD

westonpace · 2023-04-03T21:09:00Z

These jobs are incredibly long running 😲 . In the meantime I verified that the test-alpine-linux-cpp failure is unrelated and have a fix proposed in #34870.

* Making Acero off on minimal builds. Adding 'skip if no snappy' to parquet test that needs snappy. * Various changes to allow R to build without acero * Skipping query engine tests if acero not enabled * Disabling example that depended on Acero if Acero is not enabled * Add arrow_with_acero function * Turns out @examplesIf has to replace @examples

…onfuses mingw when statically linking

westonpace · 2023-04-05T14:07:40Z

@github-actions crossbow submit test-r-minimal-build

github-actions · 2023-04-05T14:10:24Z

Revision: fdbc370

Submitted crossbow builds: ursacomputing/crossbow @ actions-1eeda5ce75

Task	Status
test-r-minimal-build

westonpace · 2023-04-05T14:30:20Z

@github-actions crossbow submit test-r-minimal-build

github-actions · 2023-04-05T14:32:51Z

Revision: fdbc370

Submitted crossbow builds: ursacomputing/crossbow @ actions-ede3ef8fbe

Task	Status
test-r-minimal-build

assignUser

+1 merging this to unblock the tests Thanks working on this!

icexelloss · 2023-04-05T15:43:50Z

@github-actions crossbow submit test-r-minimal-build r-binary-packages homebrew-r-autobrew

github-actions · 2023-04-05T15:46:14Z

Revision: fdbc370

Submitted crossbow builds: ursacomputing/crossbow @ actions-82ec0861a3

Task	Status
homebrew-r-autobrew
r-binary-packages
test-r-minimal-build

assignUser · 2023-04-05T17:48:34Z

I'll merge this to get things going again!

ursabot · 2023-04-06T13:23:40Z

Benchmark runs are scheduled for baseline = 2678db3 and contender = c219863. c219863 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.0% ⬆️0.0%] test-mac-arm
[Finished ⬇️1.28% ⬆️0.26%] ursa-i9-9960x
[Failed ⬇️0.0% ⬆️0.0%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] c2198630 ec2-t3-xlarge-us-east-2
[Failed] c2198630 test-mac-arm
[Finished] c2198630 ursa-i9-9960x
[Failed] c2198630 ursa-thinkcentre-m75q
[Finished] 2678db3d ec2-t3-xlarge-us-east-2
[Failed] 2678db3d test-mac-arm
[Finished] 2678db3d ursa-i9-9960x
[Failed] 2678db3d ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

nealrichardson · 2023-04-06T17:27:26Z

👍 @thisisnic @assignUser reminder that these changes will need to be upstreamed to autobrew at CRAN release time (fyi @jeroen)

#34844 inadvertently resulted in `skip_if_not_available("acero")` to always skip.

apache#34844 inadvertently resulted in `skip_if_not_available("acero")` to always skip.

…he#34844) ### Rationale for this change There was an issue with window R build introduced by this PR: apache#34711 ### What changes are included in this PR? Many changes to the build files ### Are these changes tested? crossbow builds ### Are there any user-facing changes? * Closes: apache#34843 Lead-authored-by: Weston Pace <weston.pace@gmail.com> Co-authored-by: Li Jin <ice.xelloss@gmail.com> Signed-off-by: Jacob Wujciak-Jens <jacob@wujciak.de>

apache#34844 inadvertently resulted in `skip_if_not_available("acero")` to always skip.

icexelloss requested review from paleolimbot and thisisnic as code owners April 2, 2023 16:02

github-actionsBot added Component: R awaiting committer review Awaiting committer review labels Apr 2, 2023

icexelloss changed the title ~~GH-34843: Fix windows R build~~GH-34843: [R} Fix windows R buildApr 3, 2023

thisisnic changed the title ~~GH-34843: [R} Fix windows R build~~GH-34843: [R] Fix windows R buildApr 3, 2023

icexelloss requested a review from assignUser as a code owner April 3, 2023 12:44

icexelloss force-pushed the refactor-r-window branch from 0c234be to d7a4c2dCompare April 3, 2023 14:14

icexelloss requested review from kou and raulcd as code owners April 3, 2023 15:26

westonpace added 6 commits April 5, 2023 07:06

Removed incorrectly added [[acero::export]]
26c73f2

Change acero/dataset linker order since dataset depends on acero
946c701

Make sure that Acero doesn't mark symbols with dllimport since that c…
8bfa93e
…onfuses mingw when statically linking

One more spot that needed -DARROW_R_WITH_ACERO
0f8daea

Removed excess blank line per lint.
fdbc370

westonpace force-pushed the refactor-r-window branch from 6a0d7fd to fdbc370Compare April 5, 2023 14:07

assignUser approved these changes Apr 5, 2023
View reviewed changes

github-actionsBot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Apr 5, 2023

assignUser merged commit c219863 into apache:mainApr 5, 2023

github-actionsBot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Apr 5, 2023

nealrichardson mentioned this pull request Apr 6, 2023
MINOR: [R] Unskip acero tests #34943
Merged

nealrichardson added a commit that referenced this pull request Apr 7, 2023
MINOR: [R] Unskip acero tests (#34943)
14e9e3c
#34844 inadvertently resulted in `skip_if_not_available("acero")` to always skip.

assignUser mentioned this pull request Apr 18, 2023
[R] CRAN packaging checklist for version 12.0.0 #35216
Closed
44 tasks

liujiacheng777 pushed a commit to LoongArch-Python/arrow that referenced this pull request May 11, 2023
MINOR: [R] Unskip acero tests (apache#34943)
85ed44e
apache#34844 inadvertently resulted in `skip_if_not_available("acero")` to always skip.

ArgusLi pushed a commit to Bit-Quill/arrow that referenced this pull request May 15, 2023
MINOR: [R] Unskip acero tests (apache#34943)
8e816b2
apache#34844 inadvertently resulted in `skip_if_not_available("acero")` to always skip.

Uh oh!

Conversation

icexelloss commented Apr 2, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actionsBot commented Apr 2, 2023

Uh oh!

github-actionsBot commented Apr 2, 2023

Uh oh!

icexelloss commented Apr 2, 2023

Uh oh!

github-actionsBot commented Apr 2, 2023

Uh oh!

icexelloss commented Apr 3, 2023

Uh oh!

github-actionsBot commented Apr 3, 2023

Uh oh!

icexelloss commented Apr 3, 2023

Uh oh!

github-actionsBot commented Apr 3, 2023

Uh oh!

icexelloss commented Apr 3, 2023

Uh oh!

icexelloss commented Apr 3, 2023

Uh oh!

assignUser commented Apr 3, 2023

Uh oh!

icexelloss commented Apr 3, 2023

Uh oh!

icexelloss commented Apr 3, 2023

Uh oh!

github-actionsBot commented Apr 3, 2023

Uh oh!

icexelloss commented Apr 3, 2023

Uh oh!

github-actionsBot commented Apr 3, 2023

Uh oh!

icexelloss commented Apr 3, 2023

Uh oh!

westonpace commented Apr 3, 2023

Uh oh!

assignUser commented Apr 3, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

westonpace commented Apr 3, 2023

Uh oh!

westonpace commented Apr 5, 2023

Uh oh!

github-actionsBot commented Apr 5, 2023

Uh oh!

westonpace commented Apr 5, 2023

Uh oh!

github-actionsBot commented Apr 5, 2023

Uh oh!

assignUser left a comment

Choose a reason for hiding this comment

Uh oh!

icexelloss commented Apr 5, 2023

Uh oh!

github-actionsBot commented Apr 5, 2023

Uh oh!

assignUser commented Apr 5, 2023

Uh oh!

ursabot commented Apr 6, 2023

Uh oh!

nealrichardson commented Apr 6, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

icexelloss commented Apr 2, 2023•
edited
Loading

assignUser commented Apr 3, 2023•
edited
Loading