GH-48926: [C++] Upgrade Abseil/Protobuf/GRPC/Google-Cloud-CPP bundled versions

[raulcd]

Rationale for this change

The versions we are currently using are really old. There are several advantages on using newer bundled versions but in general they'll bring bug fixes, security fixes and new features.

What changes are included in this PR?

• Update ABSL from 20211102.0 to 20250127.0
• Remove now unnecessary CRC32C
• Update Google Cloud CPP from v2.22.0 to v3.2.0
• Updatge gRPC from v1.46.3 to v1.76.0
• Update Protobuf from v21.3 to v31.1
• Older Ubuntu, Debian and RHEL will have to bundle newer ABSL due to CRC32C requirements.
• A couple of minor patches required for Google Cloud old MINGW and protobuf emscripten builds.
• Other minor changes and requirements to build.

Are these changes tested?

Yes via CI

Are there any user-facing changes?

Not directly but some old versions (like compiling Google Cloud with old system Abseil) might stop working.

• GitHub Issue: [C++] Building Flight with bundled gRPC / Abseil fails with gcc-15 and C++20 #48926

[github-actions]

⚠️ GitHub issue #48926has been automatically assigned in GitHub to PR creator.

[raulcd]

@kou I am submitting this to show some issues I am facing and see if you have any idea on how to solve them. Basically I've tried to bump protobuf/grpc and the related abseil used by default on that version of protobuf. Building flight with bundled dependencies works great but when I want to also build google cloud cpp I start having some problems because newer abseil defines a crc32c target but crc32c also defines the same target name and Google Cloud requires both abseil and crc32c.

-- google-cloud-cpp: CRC32C: Performing Test HAVE_WEAK_GETAUXVAL - Success
CMake Error at /build/cpp/_deps/crc32c-src/CMakeLists.txt:268 (add_library):
add_library cannot create target "crc32c" because another target with the
same name already exists. The existing target is a static library created
in source directory "/build/cpp/_deps/absl-src/absl/crc". See
documentation for policy CMP0002 for more details.
CMake Warning (dev) at /build/cpp/_deps/crc32c-src/CMakeLists.txt:274 (target_sources):
Policy CMP0076 is not set: target_sources() command converts relative paths
to absolute. Run "cmake --help-policy CMP0076" for policy details. Use
the cmake_policy command to set the policy and suppress this warning.
A private source from a directory other than that of target "crc32c" has a
relative path.
This warning is for project developers. Use -Wno-dev to suppress it.
-- google-cloud-cpp: Only building the google-cloud-cpp::storage component

I've tried several things locally but to be fair none of them seem to help me move forward. Any idea of how to proceed with those?

[kou]

googleapis/google-cloud-cpp#15402 may solve it.
I'll complete the PR...

[kou]

googleapis/google-cloud-cpp#15402 (comment)

The upcoming v3.0.0 release of the Cloud C++ SDK will use the crc32c provided by Abseil and will no longer require the separate crc32c package.

[raulcd]

@kou should we try patching ourselves with the patch (or similar) you proposed? I haven't been able to get a clear understanding on when are they planning to have v3

[kou]

If we want to try before the v3.0.0 release, we can use GIT_REPOSITORY and GIT_TAG in fetchcontent_declare(): https://cmake.org/cmake/help/latest/module/FetchContent.html#typical-case

We don't need to use my suggested patch.

[raulcd]

I gave that a quick try building from the tag and removing crc32c, see last commit: cdf2f31
I tried disabling opentelemetry and other auto-added features that require grpc on google cloud cpp but it seems that opentelemetry is unconditionally required on v3 (current failures on linux jobs)
Windows failures are related to building abseil time.

Some more work is required to upgrade :(

[raulcd]

@kou I've been able to have a fully bundled dependency build locally with the updated versions! 🥳 we are closer to having this done! Now we have to fix the rest of CI jobs!

The main friction is currently coming from the fact that Google Cloud CPP seems to unconditionally require OpenTelemetry to build and we have several jobs where we are not building with OpenTelemetry enabled.

Nonetheless I think this is a great step, I've been able to build with all our dependencies (--preset ninja-debug-maximal) with the newer versions locally!

[raulcd]

Note to self, I should fix this warning:

CMakeWarning (dev) atC:/hostedtoolcache/windows/Ruby/4.0.1/x64/msys64/ucrt64/share/cmake/Modules/FetchContent.cmake:1963 (message):CallingFetchContent_Populate(opentelemetry_cpp) isdeprecated,callFetchContent_MakeAvailable(opentelemetry_cpp) instead.PolicyCMP0169canbesettoOLDtoallowFetchContent_Populate(opentelemetry_cpp) tobecalleddirectlyfornow,buttheabilitytocallitwithdeclareddetailswillberemovedcompletelyinafutureversion.CallStack (mostrecentcallfirst):cmake_modules/ThirdpartyToolchain.cmake:3505 (fetchcontent_populate)
cmake_modules/ThirdpartyToolchain.cmake:193 (build_google_cloud_cpp_storage)
cmake_modules/ThirdpartyToolchain.cmake:304 (build_dependency)
cmake_modules/ThirdpartyToolchain.cmake:3577 (resolve_dependency)
CMakeLists.txt:538 (include)

[kou]

Great! I'll also take a look at the Google Cloud C++ and OpenTelemetry later.

[raulcd]

The MinGW failures seems related to something that happened in the past but I can't really understand what is happening:

FAILED: [code=1] _deps/google_cloud_cpp-build/google/cloud/CMakeFiles/google_cloud_cpp_common.dir/internal/sha256_hash.cc.obj D:\a\_temp\msys64\mingw64\bin\ccache.exe D:\a\_temp\msys64\mingw64\bin\c++.exe -DARROW_HAVE_RUNTIME_SSE4_2 -DARROW_HAVE_SSE4_2 -DNOMINMAX -DWIN32_LEAN_AND_MEAN -D_CRT_SECURE_NO_WARNINGS -ID:/a/arrow/arrow/build/cpp/_deps/google_cloud_cpp-src -isystem D:/a/arrow/arrow/build/cpp/_deps/google_cloud_cpp-build -isystem D:/a/arrow/arrow/build/cpp/_deps/opentelemetry_cpp-src/api/include -Wredundant-move -Wno-noexcept-type -Wno-self-move -fdiagnostics-color=always -Wa,-mbig-obj -Wall -Wno-conversion -Wno-sign-conversion -Wdate-time -Wimplicit-fallthrough -Wunused-result -fno-semantic-interposition -mxsave -msse4.2 -O3 -DNDEBUG -O2 -ftree-vectorize -std=c++20 -Wall -Wextra -Wconversion -Wno-sign-conversion -MD -MT _deps/google_cloud_cpp-build/google/cloud/CMakeFiles/google_cloud_cpp_common.dir/internal/sha256_hash.cc.obj -MF _deps\google_cloud_cpp-build\google\cloud\CMakeFiles\google_cloud_cpp_common.dir\internal\sha256_hash.cc.obj.d -fmodules-ts -fmodule-mapper=_deps\google_cloud_cpp-build\google\cloud\CMakeFiles\google_cloud_cpp_common.dir\internal\sha256_hash.cc.obj.modmap -MD -fdeps-format=p1689r5 -x c++ -o _deps/google_cloud_cpp-build/google/cloud/CMakeFiles/google_cloud_cpp_common.dir/internal/sha256_hash.cc.obj -c D:/a/arrow/arrow/build/cpp/_deps/google_cloud_cpp-src/google/cloud/internal/sha256_hash.cc
D:/a/arrow/arrow/build/cpp/_deps/google_cloud_cpp-src/google/cloud/internal/sha256_hash.cc: In function 'google::cloud::v3_2::internal::Sha256Type google::cloud::v3_2::internal::{anonymous}::Sha256Hash(const void*, std::size_t)':
D:/a/arrow/arrow/build/cpp/_deps/google_cloud_cpp-src/google/cloud/internal/sha256_hash.cc:38:3: error: 'BCryptHash' was not declared in this scope; did you mean 'BCryptSignHash'?
38 | BCryptHash(BCRYPT_SHA256_ALG_HANDLE, nullptr, 0,
| ^~~~~~~~~~
| BCryptSignHash

Seems related to old issues raised by us on both MinGW and GCP, @kou you did open the original issues, any idea?

• bcyrpt.h: CNG Algorithm Pseudo-handles are missing mingw-w64/mingw-w64#49
• Can't build with MinGW: BCRYPT_SHA256_ALG_HANDLE isn't defined googleapis/google-cloud-cpp#14436

The only thing I've found is that MinGW's bcrypt.h does declare BCryptHash but it's guarded by #if NTDDI_VERSION >= NTDDI_WINTHRESHOLD (Windows 10), maybe a regression on Google Cloud building on MinGW64?

On another note, I've been able to fix some more CI jobs, I'll continue pushing this.

[kou]

Hmm. I checked Google Cloud C++ changes but I couldn't find any related changes...

How about specifying _WIN32_WINNT=0x0A01 (_WIN32_WINNT_WIN10) in our end explicitly like the following?

arrow/cpp/cmake_modules/ThirdpartyToolchain.cmake

Lines 3878 to 3884 in 560ef02

	if(MINGW)
	# PPROCESSOR_NUMBER requires Windows 7 or later.
	#
	# 0x0601 == _WIN32_WINNT_WIN7
	string(APPENDCMAKE_C_FLAGS" -D_WIN32_WINNT=0x0601")
	string(APPENDCMAKE_CXX_FLAGS" -D_WIN32_WINNT=0x0601")
	endif()

BTW, we can switch to _WIN32_WINNT_WIN10 from _WIN32_WINNT_WIN7.

(Sorry. I haven't taken a look at the Google Cloud C++ and OpenTelemetry...)

[raulcd]

@kou that was a nice one. I also have had to apply a patch to google-cloud-cpp with the guard for some new bcrypt algorithms missing on old MinGW, probably something we can share upstream for a patch (similar to the previous one).
The current failures seem unrelated, I've opened:

• [CI][R] AMD64 Windows R release fails with IOError: Bucket 'ursa-labs-r-test' not found #49609

I'll start exercising extended CI (Linux Packaging, extra C++) next week to continue fixing CI but this is "looking better" than some weeks ago now.

[kou]

we can share upstream for a patch (similar to the previous one).

Let's try it. But they may reject it because the latest MinGW doesn't need the workaround. If it's reject, let's keep the patch in our end until we drop support for old MinGW.

Or we may want to disable GCS on Windows with R build like we did for macOS: #49068

@thisisnic@jonkeane Do you have any opinion for GCS on Windows?

[thisisnic]

we can share upstream for a patch (similar to the previous one).

Let's try it. But they may reject it because the latest MinGW doesn't need the workaround. If it's reject, let's keep the patch in our end until we drop support for old MinGW.

Or we may want to disable GCS on Windows with R build like we did for macOS: #49068

@thisisnic@jonkeane Do you have any opinion for GCS on Windows?

CRAN uses MinGW-w64 version 11.0.1, so we will end up with that issue there. Hard to say as it's unclear how many users use GCS on Windows - what do you think @jonkeane ?

[raulcd]

@github-actions crossbow submit -g cpp

[raulcd]

@github-actions crossbow submit -g cpp

[github-actions]

Revision: fc3cb96

Submitted crossbow builds: ursacomputing/crossbow @ actions-1bc4f30025

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-debian-13-cpp-amd64
test-debian-13-cpp-i386
test-debian-experimental-cpp-gcc-15
test-fedora-42-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

[raulcd]

I've rebased and re-triggered CI. As RC0 looks good and CI looks good here. I plan to merge tomorrow targeting 25.0.0

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit b95645d.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 2 possible false positives for unstable benchmarks that are known to sometimes produce them.