[R] R package is linking to the wrong OpenSSL on MacOS BigSur?

[paleolimbot]

Describe the bug, including details regarding any error messages, version, and platform.

From the r-binary-packages job failures: https://github.com/ursacomputing/crossbow/actions/runs/5442353558/jobs/9898961393

All of them look like this:

** testing if installed package can be loaded from temporary location
Error: package or namespace load failed for ‘arrow’ in dyn.load(file, DLLpath = DLLpath, ...):
ERROR: loading failed
 unable to load shared object '/Users/voltrondata/Library/R/arm64/4.1/library/00LOCK-arrow/00new/arrow/libs/arrow.so':
 dlopen(/Users/voltrondata/Library/R/arm64/4.1/library/00LOCK-arrow/00new/arrow/libs/arrow.so, 6): Symbol not found: _EVP_DigestSignUpdate
* removing ‘/Users/voltrondata/Library/R/arm64/4.1/library/arrow’
 Referenced from: /Users/voltrondata/Library/R/arm64/4.1/library/00LOCK-arrow/00new/arrow/libs/arrow.so
 Expected in: flat namespace
 in /Users/voltrondata/Library/R/arm64/4.1/library/00LOCK-arrow/00new/arrow/libs/arrow.so
Error: loading failed
Execution halted
Warning message:
In install.packages("arrow", type = "source", repos = sub("file://", :
 installation of package ‘arrow’ had non-zero exit status
Error in library(arrow) : there is no package called ‘arrow’
Execution halted

Component(s)

R

[paleolimbot]

I mention OpenSSL here because the symbol that is not found is EVP_DigestSignUpdate...I suppose it could also be failing to link to any OpenSSL.

[paleolimbot]

I wonder if:

arrow/r/configure

Lines 111 to 120 in 7ecec6c

	# find openssl on macos. macOS ships with libressl. openssl is installable
	# with brew, but it is generally not linked. We can over-ride this and find
	# openssl but setting OPENSSL_ROOT_DIR (which cmake will pick up later in
	# the installation process). FWIW, arrow's cmake process uses this
	# same process to find openssl, but doing it now allows us to catch it in
	# nixlibs.R and throw a nicer error.
	if [ "${OPENSSL_ROOT_DIR}" = "" ] && brew --prefix openssl >/dev/null 2>&1; then
	export OPENSSL_ROOT_DIR="`brew --prefix openssl`"
	export PKG_CONFIG_PATH="${OPENSSL_ROOT_DIR}/lib/pkgconfig${PKG_CONFIG_PATH:+:${PKG_CONFIG_PATH}}"
	fi

is at odds with:

arrow/cpp/cmake_modules/FindOpenSSLAlt.cmake

Lines 22 to 39 in 7ecec6c

	if(APPLE AND NOT OPENSSL_ROOT_DIR)
	find_program(BREW brew)
	if(BREW)
	foreach(BREW_OPENSSL_VERSION "" "3" "3.0" "1.1")
	set(BREW_OPENSSL_PACKAGE "openssl")
	if(BREW_OPENSSL_VERSION)
	string(APPEND BREW_OPENSSL_PACKAGE "@${BREW_OPENSSL_VERSION}")
	endif()
	execute_process(COMMAND ${BREW} --prefix --installed ${BREW_OPENSSL_PACKAGE}
	OUTPUT_VARIABLE BREW_OPENSSL_PREFIX
	OUTPUT_STRIP_TRAILING_WHITESPACE)
	if(BREW_OPENSSL_PREFIX)
	set(OPENSSL_ROOT_DIR ${BREW_OPENSSL_PREFIX})
	break()
	endif()
	endforeach()
	endif()
	endif()

(that was added in #36336 )

[kou]

Ah, this is caused by mixing multiple OpenSSL versions.
It seems that openssl/*.h of OpenSSL 1.1 is used but libcrypto.dylib of OpenSSL 3 is used when building Apache Arrow C++. (Is autobrew used to build Apache Arrow C++?)
Could you check it?

[paleolimbot]

Yes, it's autobrew in that job ( https://github.com/ursacomputing/crossbow/actions/runs/5442353558/jobs/9897977884#step:13:60 ). I think that script is here: https://github.com/apache/arrow/blob/main/r/tools/autobrew and the formula is here: https://github.com/apache/arrow/blob/main/dev/tasks/homebrew-formulae/autobrew/apache-arrow-static.rb .

I think I'm remembering correctly that we always try to link to system openssl on MacOS and not the one provided by homebrew/autobrew... @assignUser may be more familiar with the details!

Also worth nothing that we're considering dropping autobrew to reduce the number of steps needed for the R package release; however, I don't know if doing a bundled build (i.e., running cmake on the bundled cpp source instead of autobrew) is more or less complicated with respect to openssl.

[assignUser]

may be more familiar with the details

eh, not really due to lack of macos locally ^^ but I can also have a look at this!

[raulcd]

homebrew-r-autobrew nightly job seems to fail with the same issue. Should this be considered a blocker for the release? I am marking it as a blocker at the moment.

[paleolimbot]

Thanks...yes, it's a blocker for release! CRAN does some builds on MacOS BigSur which is how most people will install the R package.

[assignUser]

As discussed in the PR this seems to be an issue with the gh runners explicitly running brew link openssl which causes the isssue. It would be nice if someone could try to repo locally with a default brew openssl install to make sure. I would assume that cran doesn't do this...

edit: or not as that solution does not work afterall