[C++][CI] Flight tests fail running on some configurations

[pitrou]

Describe the bug, including details regarding any error messages, version, and platform.

For example on Ubuntu 22.04:
https://github.com/ursacomputing/crossbow/actions/runs/5799341398/job/15719127299#step:6:4848

or Debian 11:
https://github.com/ursacomputing/crossbow/actions/runs/5799341448/job/15719127520

Symptoms are the same:

 70/107 Test #71: arrow-flight-internals-test ...............***Failed 0.12 sec
Running arrow-flight-internals-test, redirecting output into /build/cpp/build/test-logs/arrow-flight-internals-test.txt (attempt 1/1)
/build/cpp/debug/arrow-flight-internals-test: error while loading shared libraries: libgmockd.so.1.11.0: cannot open shared object file: No such file or directory
/build/cpp/src/arrow/flight

 Start 72: arrow-flight-test
 71/107 Test #72: arrow-flight-test .........................***Failed 0.12 sec
Running arrow-flight-test, redirecting output into /build/cpp/build/test-logs/arrow-flight-test.txt (attempt 1/1)
/build/cpp/debug/arrow-flight-test: error while loading shared libraries: libgmockd.so.1.11.0: cannot open shared object file: No such file or directory
/build/cpp/src/arrow/flight

Component(s)

C++, Continuous Integration, FlightRPC

[pitrou]

@kou @lidavidm

[pitrou]

This seems to be caused by the LD_LIBRARY_PATH setting in ci/scripts/cpp_test.sh. If I disable LD_LIBRARY_PATH, the tests run fine.

[kou]

It seems that 37cb592 / GH-36956 is related.

[lidavidm]

Hmm, I guess we have to also list GMock here? https://github.com/apache/arrow/blob/main/cpp/src/arrow/flight/CMakeLists.txt#L255

[kou]

It seems that it doesn't solve this problem.

The link command line for arrow-flight-test has googletest_ep-prefix/lib/libgmockd.so but it's not linked to arrow-flight-test directly. (libgtestd.so and libgtest_maind.so are linked directly.)

$ objdump -p build/cpp/debug/arrow-flight-test | grep NEEDED
 NEEDED libarrow_flight_testing.so.1300
 NEEDED libarrow_testing.so.1300
 NEEDED libgtest_maind.so.1.11.0
 NEEDED libarrow_flight.so.1300
 NEEDED libgrpc.so.10
 NEEDED libarrow.so.1300
 NEEDED libgtestd.so.1.11.0
 NEEDED libstdc++.so.6
 NEEDED libgcc_s.so.1
 NEEDED libc.so.6
 NEEDED ld-linux-x86-64.so.2

arrow-flight-test has rpath to bundled GoogleTest:

$ objdump -p build/cpp/debug/arrow-flight-test | grep RUNPATH
 RUNPATH /build/cpp/debug:/build/cpp/googletest_ep-prefix/lib

So directed linked libgtestd.so and libgtest_maind.so can be found but libgmockd.so can't:

$ ldd build/cpp/debug/arrow-flight-test | grep libg
	libgtest_maind.so.1.11.0 => /build/cpp/googletest_ep-prefix/lib/libgtest_maind.so.1.11.0 (0x00007fc833eb2000)
	libgrpc.so.10 => /lib/x86_64-linux-gnu/libgrpc.so.10 (0x00007fc8337dc000)
	libgtestd.so.1.11.0 => /build/cpp/googletest_ep-prefix/lib/libgtestd.so.1.11.0 (0x00007fc82e1f8000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fc82dfac000)
	libgmockd.so.1.11.0 => /build/cpp/googletest_ep-prefix/lib/libgmockd.so.1.11.0 (0x00007fc82dd06000)
	libgrpc++.so.1 => /lib/x86_64-linux-gnu/libgrpc++.so.1 (0x00007fc82dba7000)
	libgpr.so.10 => /lib/x86_64-linux-gnu/libgpr.so.10 (0x00007fc82d054000)
	libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007fc82ceba000)
	libgnutls.so.30 => /lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007fc82caae000)
	libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007fc82c99e000)
$ LD_LIBRARY_PATH=${ARROW_HOME}/${CMAKE_INSTALL_LIBDIR:-lib}:${LD_LIBRARY_PATH} ldd build/cpp/debug/arrow-flight-test | grep libg
	libgtest_maind.so.1.11.0 => /build/cpp/googletest_ep-prefix/lib/libgtest_maind.so.1.11.0 (0x00007f4fe6589000)
	libgrpc.so.10 => /lib/x86_64-linux-gnu/libgrpc.so.10 (0x00007f4fe5eb3000)
	libgtestd.so.1.11.0 => /build/cpp/googletest_ep-prefix/lib/libgtestd.so.1.11.0 (0x00007f4fe08cf000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f4fe0683000)
	libgmockd.so.1.11.0 => not found
	libgrpc++.so.1 => /lib/x86_64-linux-gnu/libgrpc++.so.1 (0x00007f4fe02da000)
	libgpr.so.10 => /lib/x86_64-linux-gnu/libgpr.so.10 (0x00007f4fdf787000)
	libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f4fdf5ed000)
	libgnutls.so.30 => /lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f4fdf1e1000)
	libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f4fdf0d1000)

libgmocked.so is linked implicitly via libarrow_flight_testing.so:

$ lddtree build/cpp/debug/arrow-flight-test
arrow-flight-test => build/cpp/debug/arrow-flight-test (interpreter => /lib64/ld-linux-x86-64.so.2)
 libarrow_flight_testing.so.1300 => /build/cpp/debug/libarrow_flight_testing.so.1300
scanelf: rpath_security_checks(): Security problem NULL DT_RUNPATH in /build/cpp/debug/libarrow_flight_testing.so.1300
 libboost_filesystem.so.1.74.0 => /lib/x86_64-linux-gnu/libboost_filesystem.so.1.74.0
 libgmockd.so.1.11.0 => /build/cpp/googletest_ep-prefix/lib/libgmockd.so.1.11.0
 libarrow_testing.so.1300 => /build/cpp/debug/libarrow_testing.so.1300
scanelf: rpath_security_checks(): Security problem NULL DT_RUNPATH in /build/cpp/debug/libarrow_testing.so.1300
 libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
 libgtest_maind.so.1.11.0 => /build/cpp/googletest_ep-prefix/lib/libgtest_maind.so.1.11.0
 libarrow_flight.so.1300 => /build/cpp/debug/libarrow_flight.so.1300
scanelf: rpath_security_checks(): Security problem NULL DT_RUNPATH in /build/cpp/debug/libarrow_flight.so.1300
 libgrpc++.so.1 => /lib/x86_64-linux-gnu/libgrpc++.so.1
 libabsl_strings.so.20210324 => /lib/x86_64-linux-gnu/libabsl_strings.so.20210324
 libabsl_strings_internal.so.20210324 => /lib/x86_64-linux-gnu/libabsl_strings_internal.so.20210324
 libabsl_int128.so.20210324 => /lib/x86_64-linux-gnu/libabsl_int128.so.20210324
 libabsl_throw_delegate.so.20210324 => /lib/x86_64-linux-gnu/libabsl_throw_delegate.so.20210324
 libabsl_raw_logging_internal.so.20210324 => /lib/x86_64-linux-gnu/libabsl_raw_logging_internal.so.20210324
 libgpr.so.10 => /lib/x86_64-linux-gnu/libgpr.so.10
 libabsl_str_format_internal.so.20210324 => /lib/x86_64-linux-gnu/libabsl_str_format_internal.so.20210324
scanelf: rpath_security_checks(): Security problem NULL DT_RUNPATH in /build/cpp/debug/libarrow_flight.so.1300
 libprotobuf.so.23 => /lib/x86_64-linux-gnu/libprotobuf.so.23
 libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1
 libgrpc.so.10 => /lib/x86_64-linux-gnu/libgrpc.so.10
 libssl.so.3 => /lib/x86_64-linux-gnu/libssl.so.3
 libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3
 libabsl_time.so.20210324 => /lib/x86_64-linux-gnu/libabsl_time.so.20210324
 libabsl_time_zone.so.20210324 => /lib/x86_64-linux-gnu/libabsl_time_zone.so.20210324
 libabsl_base.so.20210324 => /lib/x86_64-linux-gnu/libabsl_base.so.20210324
 libabsl_spinlock_wait.so.20210324 => /lib/x86_64-linux-gnu/libabsl_spinlock_wait.so.20210324
 libabsl_bad_optional_access.so.20210324 => /lib/x86_64-linux-gnu/libabsl_bad_optional_access.so.20210324
 libcares.so.2 => /lib/x86_64-linux-gnu/libcares.so.2
 libarrow.so.1300 => /build/cpp/debug/libarrow.so.1300
 libbrotlienc.so.1 => /lib/x86_64-linux-gnu/libbrotlienc.so.1
 libbrotlicommon.so.1 => /lib/x86_64-linux-gnu/libbrotlicommon.so.1
 libbrotlidec.so.1 => /lib/x86_64-linux-gnu/libbrotlidec.so.1
 libutf8proc.so.2 => /lib/x86_64-linux-gnu/libutf8proc.so.2
 libre2.so.9 => /lib/x86_64-linux-gnu/libre2.so.9
 libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0
 liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1
 libsnappy.so.1 => /lib/x86_64-linux-gnu/libsnappy.so.1
 libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1
 libcurl.so.4 => /lib/x86_64-linux-gnu/libcurl.so.4
 libnghttp2.so.14 => /lib/x86_64-linux-gnu/libnghttp2.so.14
 libidn2.so.0 => /lib/x86_64-linux-gnu/libidn2.so.0
 libunistring.so.2 => /lib/x86_64-linux-gnu/libunistring.so.2
 librtmp.so.1 => /lib/x86_64-linux-gnu/librtmp.so.1
 libgnutls.so.30 => /lib/x86_64-linux-gnu/libgnutls.so.30
 libp11-kit.so.0 => /lib/x86_64-linux-gnu/libp11-kit.so.0
 libffi.so.8 => /lib/x86_64-linux-gnu/libffi.so.8
 libtasn1.so.6 => /lib/x86_64-linux-gnu/libtasn1.so.6
 libhogweed.so.6 => /lib/x86_64-linux-gnu/libhogweed.so.6
 libnettle.so.8 => /lib/x86_64-linux-gnu/libnettle.so.8
 libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10
 libssh.so.4 => /lib/x86_64-linux-gnu/libssh.so.4
 libpsl.so.5 => /lib/x86_64-linux-gnu/libpsl.so.5
 libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2
 libkrb5.so.3 => /lib/x86_64-linux-gnu/libkrb5.so.3
 libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1
 libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2
 libk5crypto.so.3 => /lib/x86_64-linux-gnu/libk5crypto.so.3
 libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2
 libkrb5support.so.0 => /lib/x86_64-linux-gnu/libkrb5support.so.0
 libldap-2.5.so.0 => /lib/x86_64-linux-gnu/libldap-2.5.so.0
 libsasl2.so.2 => /lib/x86_64-linux-gnu/libsasl2.so.2
 liblber-2.5.so.0 => /lib/x86_64-linux-gnu/liblber-2.5.so.0
 libgtestd.so.1.11.0 => /build/cpp/googletest_ep-prefix/lib/libgtestd.so.1.11.0
 libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6
 libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
 libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
 ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2

[kou]

The link command line:

/usr/lib/ccache/c++ \
-Wno-noexcept-type \
-fdiagnostics-color=always \
-Wall \
-Wno-conversion \
-Wno-sign-conversion \
-Wunused-result \
-Wdate-time \
-fno-semantic-interposition \
-msse4.2 \
-g \
-Werror \
-O0 \
-ggdb \
src/arrow/flight/CMakeFiles/arrow-flight-test.dir/flight_test.cc.o \
-o \
debug/arrow-flight-test \
-Wl,-rpath,/build/cpp/debug:/build/cpp/googletest_ep-prefix/lib \
debug/libarrow_flight_testing.so.1300.0.0 \
debug/libarrow_testing.so.1300.0.0 \
google_cloud_cpp_ep-install/lib/libgoogle_cloud_cpp_storage.a \
/usr/lib/x86_64-linux-gnu/libcrypto.so \
/usr/lib/x86_64-linux-gnu/libssl.so \
/usr/lib/x86_64-linux-gnu/libbrotlienc.so \
/usr/lib/x86_64-linux-gnu/libbrotlidec.so \
/usr/lib/x86_64-linux-gnu/libbrotlicommon.so \
orc_ep-install/lib/liborc.a \
/usr/lib/x86_64-linux-gnu/libprotobuf.so \
awssdk_ep-install/lib/libaws-cpp-sdk-identity-management.a \
awssdk_ep-install/lib/libaws-cpp-sdk-sts.a \
awssdk_ep-install/lib/libaws-cpp-sdk-cognito-identity.a \
awssdk_ep-install/lib/libaws-cpp-sdk-s3.a \
awssdk_ep-install/lib/libaws-cpp-sdk-core.a \
awssdk_ep-install/lib/libaws-crt-cpp.a \
awssdk_ep-install/lib/libaws-c-s3.a \
awssdk_ep-install/lib/libaws-c-auth.a \
awssdk_ep-install/lib/libaws-c-mqtt.a \
awssdk_ep-install/lib/libaws-c-http.a \
awssdk_ep-install/lib/libaws-c-compression.a \
awssdk_ep-install/lib/libaws-c-sdkutils.a \
awssdk_ep-install/lib/libaws-c-event-stream.a \
awssdk_ep-install/lib/libaws-c-io.a \
awssdk_ep-install/lib/libaws-c-cal.a \
awssdk_ep-install/lib/libaws-checksums.a \
awssdk_ep-install/lib/libaws-c-common.a \
awssdk_ep-install/lib/libs2n.a \
opentelemetry_ep-install/lib/libopentelemetry_trace.a \
opentelemetry_ep-install/lib/libopentelemetry_exporter_ostream_span.a \
opentelemetry_ep-install/lib/libopentelemetry_exporter_otlp_http.a \
/usr/lib/x86_64-linux-gnu/libutf8proc.so \
/usr/lib/x86_64-linux-gnu/libre2.so \
-ldl \
googletest_ep-prefix/lib/libgtest_maind.so \
googletest_ep-prefix/lib/libgtestd.so \
googletest_ep-prefix/lib/libgmockd.so \
-ldl \
/usr/lib/x86_64-linux-gnu/libgrpc++.so \
debug/libarrow_flight.so.1300.0.0 \
/usr/lib/x86_64-linux-gnu/libgrpc++.so \
/usr/lib/x86_64-linux-gnu/libgrpc.so \
/usr/lib/x86_64-linux-gnu/libgpr.so \
debug/libarrow.so.1300.0.0 \
google_cloud_cpp_ep-install/lib/libgoogle_cloud_cpp_storage.a \
google_cloud_cpp_ep-install/lib/libgoogle_cloud_cpp_rest_internal.a \
google_cloud_cpp_ep-install/lib/libgoogle_cloud_cpp_common.a \
absl_ep-install/lib/libabsl_cord.a \
absl_ep-install/lib/libabsl_cordz_info.a \
absl_ep-install/lib/libabsl_cord_internal.a \
absl_ep-install/lib/libabsl_cordz_functions.a \
absl_ep-install/lib/libabsl_exponential_biased.a \
absl_ep-install/lib/libabsl_cordz_handle.a \
absl_ep-install/lib/libabsl_synchronization.a \
absl_ep-install/lib/libabsl_stacktrace.a \
absl_ep-install/lib/libabsl_graphcycles_internal.a \
absl_ep-install/lib/libabsl_symbolize.a \
absl_ep-install/lib/libabsl_debugging_internal.a \
absl_ep-install/lib/libabsl_malloc_internal.a \
absl_ep-install/lib/libabsl_demangle_internal.a \
absl_ep-install/lib/libabsl_bad_optional_access.a \
absl_ep-install/lib/libabsl_str_format_internal.a \
absl_ep-install/lib/libabsl_time.a \
absl_ep-install/lib/libabsl_strings.a \
absl_ep-install/lib/libabsl_strings_internal.a \
absl_ep-install/lib/libabsl_throw_delegate.a \
absl_ep-install/lib/libabsl_base.a \
absl_ep-install/lib/libabsl_spinlock_wait.a \
absl_ep-install/lib/libabsl_int128.a \
absl_ep-install/lib/libabsl_civil_time.a \
absl_ep-install/lib/libabsl_time_zone.a \
absl_ep-install/lib/libabsl_bad_variant_access.a \
absl_ep-install/lib/libabsl_raw_logging_internal.a \
absl_ep-install/lib/libabsl_log_severity.a \
crc32c_ep-install/lib/libcrc32c.a \
/usr/lib/x86_64-linux-gnu/libbrotlienc.so \
/usr/lib/x86_64-linux-gnu/libbrotlidec.so \
/usr/lib/x86_64-linux-gnu/libbrotlicommon.so \
orc_ep-install/lib/liborc.a \
/usr/lib/x86_64-linux-gnu/libz.so \
/usr/lib/x86_64-linux-gnu/liblz4.so \
awssdk_ep-install/lib/libaws-cpp-sdk-identity-management.a \
awssdk_ep-install/lib/libaws-cpp-sdk-sts.a \
awssdk_ep-install/lib/libaws-cpp-sdk-cognito-identity.a \
awssdk_ep-install/lib/libaws-cpp-sdk-s3.a \
awssdk_ep-install/lib/libaws-cpp-sdk-core.a \
awssdk_ep-install/lib/libaws-crt-cpp.a \
awssdk_ep-install/lib/libaws-c-s3.a \
awssdk_ep-install/lib/libaws-c-auth.a \
awssdk_ep-install/lib/libaws-c-mqtt.a \
awssdk_ep-install/lib/libaws-c-http.a \
awssdk_ep-install/lib/libaws-c-compression.a \
awssdk_ep-install/lib/libaws-c-sdkutils.a \
awssdk_ep-install/lib/libaws-c-event-stream.a \
awssdk_ep-install/lib/libaws-c-io.a \
awssdk_ep-install/lib/libaws-c-cal.a \
/usr/lib/x86_64-linux-gnu/libssl.so \
/usr/lib/x86_64-linux-gnu/libcrypto.so \
awssdk_ep-install/lib/libaws-checksums.a \
awssdk_ep-install/lib/libaws-c-common.a \
awssdk_ep-install/lib/libs2n.a \
opentelemetry_ep-install/lib/libopentelemetry_exporter_ostream_span.a \
opentelemetry_ep-install/lib/libopentelemetry_exporter_otlp_http.a \
opentelemetry_ep-install/lib/libopentelemetry_otlp_recordable.a \
opentelemetry_ep-install/lib/libopentelemetry_trace.a \
opentelemetry_ep-install/lib/libopentelemetry_resources.a \
opentelemetry_ep-install/lib/libopentelemetry_common.a \
opentelemetry_ep-install/lib/libopentelemetry_exporter_otlp_http_client.a \
opentelemetry_ep-install/lib/libopentelemetry_proto.a \
/usr/lib/x86_64-linux-gnu/libprotobuf.so \
opentelemetry_ep-install/lib/libopentelemetry_http_client_curl.a \
/usr/lib/x86_64-linux-gnu/libcurl.so \
/usr/lib/x86_64-linux-gnu/libutf8proc.so \
/usr/lib/x86_64-linux-gnu/libre2.so \
-ldl \
jemalloc_ep-prefix/src/jemalloc_ep/dist//lib/libjemalloc_pic.a \
mimalloc_ep/src/mimalloc_ep/lib/mimalloc-2.0/libmimalloc-debug.a \
-lrt \
/usr/lib/x86_64-linux-gnu/libzstd.so \
/usr/lib/x86_64-linux-gnu/libsnappy.so.1.1.8 \
/usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.74.0 \
/usr/lib/x86_64-linux-gnu/libboost_system.so.1.74.0