GH-38043: [R] Enable all features by default on macOS

[assignUser]

Rationale for this change

Previously GCS/S3 support would need to be explicitly enabled in source builds (when they are build without NOT_CRAN). As we want the macos binaries to be fully featured we should turn the features on when the dependencies exists.

What changes are included in this PR?

This PR enables this behavior for macOS only, on Linux setting NOT_CRAN or LIBARROW_MINIMAL=false is still required.

Are these changes tested?

Crossbow and locally (thanks @paleolimbot )

• Closes: [R] Enable full feature libarrow build as default on macOS  #38043

[assignUser]

@github-actions crossbow submit r-binary-packages

[assignUser]

@github-actions crossbow submit r-binary-packages

[github-actions]

Revision: 9d40184

Submitted crossbow builds: ursacomputing/crossbow @ actions-3b4a613597

Task	Status
r-binary-packages

[github-actions]

Revision: 4ddbeaf

Submitted crossbow builds: ursacomputing/crossbow @ actions-28d06e21a7

Task	Status
r-binary-packages

[paleolimbot]

(After r-binary-packages passes I'll give a shot locally!)

[assignUser]

Some weirdness going on with the linux deps but mac works!

Arrow package version: 13.0.0.100000311
Capabilities:
acero TRUE
dataset TRUE
substrait FALSE
parquet TRUE
json TRUE
s3 TRUE
gcs TRUE
utf8proc TRUE
re2 TRUE
snappy TRUE
gzip TRUE
brotli TRUE
zstd TRUE
lz4 TRUE
lz4_frame TRUE
lzo FALSE
bz2 TRUE
jemalloc TRUE
mimalloc TRUE

[paleolimbot]

Ok I'll give it a go!

[assignUser]

Thanks for testing @jonkeane !

but setting this to use more cores might be nice + possible then.

In the follow up #38236 I automatically set not_cran when we are in a dev build, which should make the dev experience on especially linux but overall much smoother as you also won't have to fiddle with allow lists or such to get the binaries.

I am a bit hesitant to remove the 2 core cap by default for release builds as we could still have to build from source on cran (BDR special checks? Condition for use of binaries ...). Mac users will get the cran binary anyway and through the allow list a majority of the linux users will also get binaries, so I think the reward doesn't justify the risk of a policy violation. (I also think that the 2 core think is a philosophical stance not necessarily limited to builds on cran, at least going by recent r dev ml threads...)

but the linking step seemed to use all of my cores

😱 I didn't change anything there so I am pretty sure that was already happening (on linuc), so it looks like you are right and the short burst is within tolerance?

Thanks @paleolimbot !

I think my local 11.7 build fails because it's mixing homebrew and bundled dependencies

Hm that should not be a problem normally though unless there is some version mismatch? The 10.13 runner also has brew and even has the deps installed but no problem arises... as this is a definitive dev use case I think it's ok to have people manually set the deps source var if the problem happens. Could you try with ARROW_DEPENDENCY_SOURCE= BUNDELD?

[assignUser]

@github-actions crossbow submit -g r

[github-actions]

Revision: 9d209a6

Submitted crossbow builds: ursacomputing/crossbow @ actions-431cdc81fd

Task	Status
conda-linux-aarch64-cpu-r42
conda-linux-aarch64-cpu-r43
conda-linux-x64-cpu-r42
conda-linux-x64-cpu-r43
conda-osx-arm64-cpu-r42
conda-osx-arm64-cpu-r43
conda-osx-x64-cpu-r42
conda-osx-x64-cpu-r43
conda-win-x64-cpu-r41
r-binary-packages
test-fedora-r-clang-sanitizer
test-r-arrow-backwards-compatibility
test-r-depsource-bundled
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-gcc-11
test-r-gcc-12
test-r-install-local
test-r-install-local-minsizerel
test-r-library-r-base-latest
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-valgrind
test-r-minimal-build
test-r-offline-maximal
test-r-offline-minimal
test-r-rhub-debian-gcc-devel-lto-latest
test-r-rhub-debian-gcc-release-custom-ccache
test-r-rhub-ubuntu-gcc-release-latest
test-r-rstudio-r-base-4.1-opensuse153
test-r-rstudio-r-base-4.2-centos7-devtoolset-8
test-r-rstudio-r-base-4.2-focal
test-r-ubuntu-22.04
test-r-versions
test-ubuntu-r-sanitizer

[paleolimbot]

Could you try with ARROW_DEPENDENCY_SOURCE= BUNDELD?

I'll debug my source build separately from this PR

I am a bit hesitant to remove the 2 core cap

That primarily affects the development experience, and we're not planning to have that be the primary means by which any MacOS developer gets an Arrow build. If it turns out that it is, we can fix that incrementally after the release.

[paleolimbot]

With a green nightlies run (minus any "device out of space" errors), I think this particular chunk of the larger build system refactor is good to go! Thank you!

[paleolimbot]

I think it's reasonable to have some comment here explaing why this step exists; however, I think a wholistic review of comments on the entire script is better suited to #38236 (after rebase) (and the existing comment doesn't make this clear to me, at least).

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 4bf777a.

There was 1 benchmark result indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2023-10-13 20:07:12Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-02, scale_factor=1

The full Conbench report has more details. It also includes information about 4 possible false positives for unstable benchmarks that are known to sometimes produce them.