GH-48317: [C++] Use FetchContent for bundled google-cloud-cpp #48333

raulcd · 2025-12-04T12:53:30Z

Rationale for this change

As a follow up of requiring a minimum CMake version >= 3.25 we discussed moving our dependencies from ExternalProject to FetchContent. This can simplify our third party dependency management.

What changes are included in this PR?

The general change is moving google-cloud-cpp from ExternalProject to FetchContent and some cleanup to required dependencies that don't require double-installation anymore like Absl and CRC32C

Are these changes tested?

Yes, the changes are tested locally and on CI.

Are there any user-facing changes?

No

GitHub Issue: [C++] Use FetchContent for bundled google-cloud-cpp #48317

raulcd · 2025-12-05T14:00:32Z

@github-actions crossbow submit -g cpp

github-actions · 2025-12-05T14:03:05Z

Revision: 70b632a

Submitted crossbow builds: ursacomputing/crossbow @ actions-67f847b9c4

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1
test-cuda-cpp-ubuntu-24.04-cuda-13.0.2
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-42-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

raulcd · 2025-12-05T14:18:32Z

@github-actions crossbow submit -g r

github-actions · 2025-12-05T14:21:18Z

Revision: 70b632a

Submitted crossbow builds: ursacomputing/crossbow @ actions-83a12e40e9

Task	Status
r-binary-packages
r-recheck-most
test-r-arrow-backwards-compatibility
test-r-depsource-bundled
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-extra-packages
test-r-gcc-11
test-r-gcc-12
test-r-install-local
test-r-install-local-minsizerel
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-sanitizers
test-r-linux-valgrind
test-r-m1-san
test-r-macos-as-cran
test-r-minimal-build
test-r-offline-maximal
test-r-offline-minimal
test-r-rhub-debian-gcc-devel-lto-latest
test-r-rhub-ubuntu-gcc12-custom-ccache
test-r-rhub-ubuntu-release-latest
test-r-rocker-r-ver-latest
test-r-rstudio-r-base-4.1-focal
test-r-rstudio-r-base-4.2-focal
test-r-ubuntu-22.04
test-r-versions

raulcd

Google Cloud seems to unconditionally install when being built. There's no way of disabling installation as far as I know. This means that both google cloud headers, *.pc files and some of its dependencies seem to be pulled on the RPM packages.
The only way I've found to bypass that is to patch Google cloud so it's not installed via adding a new flag GOOGLE_CLOUD_CPP_ENABLE_INSTALL and set it to OFF to disable installation.
@kou any idea of something I could try?
The existing Linux-packaging CI failures are unrelated are happening on main.

raulcd · 2025-12-09T16:18:22Z

@github-actions crossbow submit -g cpp -g r

github-actions · 2025-12-09T16:21:31Z

Revision: 67bcadf

Submitted crossbow builds: ursacomputing/crossbow @ actions-fc1a19f010

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
r-binary-packages
r-recheck-most
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1
test-cuda-cpp-ubuntu-24.04-cuda-13.0.2
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-42-cpp
test-r-arrow-backwards-compatibility
test-r-depsource-bundled
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-extra-packages
test-r-gcc-11
test-r-gcc-12
test-r-install-local
test-r-install-local-minsizerel
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-sanitizers
test-r-linux-valgrind
test-r-m1-san
test-r-macos-as-cran
test-r-minimal-build
test-r-offline-maximal
test-r-offline-minimal
test-r-rhub-debian-gcc-devel-lto-latest
test-r-rhub-ubuntu-gcc12-custom-ccache
test-r-rhub-ubuntu-release-latest
test-r-rocker-r-ver-latest
test-r-rstudio-r-base-4.1-focal
test-r-rstudio-r-base-4.2-focal
test-r-ubuntu-22.04
test-r-versions
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

raulcd · 2025-12-09T18:15:13Z

@github-actions crossbow submit -g r

github-actions · 2025-12-09T18:17:56Z

Revision: 4ed4bbb

Submitted crossbow builds: ursacomputing/crossbow @ actions-b5803cb2be

Task	Status
r-binary-packages
r-recheck-most
test-r-arrow-backwards-compatibility
test-r-depsource-bundled
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-extra-packages
test-r-gcc-11
test-r-gcc-12
test-r-install-local
test-r-install-local-minsizerel
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-sanitizers
test-r-linux-valgrind
test-r-m1-san
test-r-macos-as-cran
test-r-minimal-build
test-r-offline-maximal
test-r-offline-minimal
test-r-rhub-debian-gcc-devel-lto-latest
test-r-rhub-ubuntu-gcc12-custom-ccache
test-r-rhub-ubuntu-release-latest
test-r-rocker-r-ver-latest
test-r-rstudio-r-base-4.1-focal
test-r-rstudio-r-base-4.2-focal
test-r-ubuntu-22.04
test-r-versions

kou · 2025-12-10T00:09:54Z

Could you try this?

diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake b/cpp/cmake_modules/ThirdpartyToolchain.cmake
index 28e984f95d..ffc0e17f75 100644
--- a/cpp/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake
@@ -3293,9 +3293,12 @@ function(build_crc32c_once)
   set(CRC32C_BUILD_TESTS OFF)
   set(CRC32C_BUILD_BENCHMARKS OFF)
   set(CRC32C_USE_GLOG OFF)
-  set(CRC32C_INSTALL OFF)
   fetchcontent_makeavailable(crc32c)
 
+  if(CMAKE_VERSION VERSION_LESS 3.28)
+    set_property(DIRECTORY ${crc32_SOURCE_DIR} PROPERTY EXCLUDE_FROM_ALL TRUE)
+  endif()
+
   # Create alias target for consistency (crc32c exports as Crc32c::crc32c when installed)
   if(NOT TARGET Crc32c::crc32c)
     add_library(Crc32c::crc32c ALIAS crc32c)
@@ -3429,13 +3432,17 @@ function(build_google_cloud_cpp_storage)
   set(GOOGLE_CLOUD_CPP_ENABLE_WERROR OFF)
   set(GOOGLE_CLOUD_CPP_WITH_MOCKS OFF)
   # Disable installation when embedded via FetchContent
-  set(GOOGLE_CLOUD_CPP_ENABLE_INSTALL OFF)
+  # set(GOOGLE_CLOUD_CPP_ENABLE_INSTALL OFF)
   set(BUILD_TESTING OFF)
   # Unity build causes some build errors.
   set(CMAKE_UNITY_BUILD FALSE)
 
   fetchcontent_makeavailable(google_cloud_cpp)
 
+  if(CMAKE_VERSION VERSION_LESS 3.28)
+    set_property(DIRECTORY ${google_cloud_cpp_SOURCE_DIR} PROPERTY EXCLUDE_FROM_ALL TRUE)
+  endif()
+
   # Remove unused directories to save build directory storage.
   # 141MB -> 79MB
   file(REMOVE_RECURSE "${google_cloud_cpp_SOURCE_DIR}/ci")

raulcd · 2025-12-10T10:55:30Z

The problem is that it stills tries to install them and as we are not installing absl those can't be found.

-- Configuring done (56.2s)
CMake Error: install(EXPORT "google_cloud_cpp_common-targets" ...) includes target "google_cloud_cpp_common" which requires target "absl_base" that is not in any export set.
CMake Error: install(EXPORT "google_cloud_cpp_common-targets" ...) includes target "google_cloud_cpp_common" which requires target "absl_memory" that is not in any export set.
CMake Error: install(EXPORT "google_cloud_cpp_common-targets" ...) includes target "google_cloud_cpp_common" which requires target "absl_optional" that is not in any export set.
CMake Error: install(EXPORT "google_cloud_cpp_common-targets" ...) includes target "google_cloud_cpp_common" which requires target "absl_span" that is not in any export set.
CMake Error: install(EXPORT "google_cloud_cpp_common-targets" ...) includes target "google_cloud_cpp_common" which requires target "absl_str_format" that is not in any export set.
CMake Error: install(EXPORT "google_cloud_cpp_common-targets" ...) includes target "google_cloud_cpp_common" which requires target "absl_time" that is not in any export set.
CMake Error: install(EXPORT "google_cloud_cpp_common-targets" ...) includes target "google_cloud_cpp_common" which requires target "absl_variant" that is not in any export set.
CMake Error: install(EXPORT "google_cloud_cpp_rest_internal-targets" ...) includes target "google_cloud_cpp_rest_internal" which requires target "absl_span" that is not in any export set.
CMake Error: install(EXPORT "storage-targets" ...) includes target "google_cloud_cpp_storage" which requires target "absl_cord" that is not in any export set.
CMake Error: install(EXPORT "storage-targets" ...) includes target "google_cloud_cpp_storage" which requires target "absl_memory" that is not in any export set.
CMake Error: install(EXPORT "storage-targets" ...) includes target "google_cloud_cpp_storage" which requires target "absl_strings" that is not in any export set.
CMake Error: install(EXPORT "storage-targets" ...) includes target "google_cloud_cpp_storage" which requires target "absl_time" that is not in any export set.
CMake Error: install(EXPORT "storage-targets" ...) includes target "google_cloud_cpp_storage" which requires target "absl_variant" that is not in any export set.

If I install abseil the problem is that those are bundled (along with crc32c and google cloud cpp) on the libarrow_bundled libraries.

cpp/cmake_modules/ThirdpartyToolchain.cmake

kou · 2025-12-10T21:36:56Z

We need to execute install(...) in Abseil but not install Abseil by EXCLUDE_FROM_ALL TRUE like we did for CRC32C.

cpp/cmake_modules/ThirdpartyToolchain.cmake

raulcd · 2025-12-11T09:47:44Z

@github-actions crossbow submit -g r -g cpp

github-actions · 2025-12-11T09:50:48Z

Revision: 08c3294

Submitted crossbow builds: ursacomputing/crossbow @ actions-b3910c6580

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
r-binary-packages
r-recheck-most
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1
test-cuda-cpp-ubuntu-24.04-cuda-13.0.2
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-42-cpp
test-r-arrow-backwards-compatibility
test-r-depsource-bundled
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-extra-packages
test-r-gcc-11
test-r-gcc-12
test-r-install-local
test-r-install-local-minsizerel
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-sanitizers
test-r-linux-valgrind
test-r-m1-san
test-r-macos-as-cran
test-r-minimal-build
test-r-offline-maximal
test-r-offline-minimal
test-r-rhub-debian-gcc-devel-lto-latest
test-r-rhub-ubuntu-gcc12-custom-ccache
test-r-rhub-ubuntu-release-latest
test-r-rocker-r-ver-latest
test-r-rstudio-r-base-4.1-focal
test-r-rstudio-r-base-4.2-focal
test-r-ubuntu-22.04
test-r-versions
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

…google cloud

…ent scope

…NS for google-cloud

raulcd · 2025-12-11T10:52:32Z

@github-actions crossbow submit -g r

github-actions · 2025-12-11T10:55:24Z

Revision: eab1602

Submitted crossbow builds: ursacomputing/crossbow @ actions-f854cdd15a

Task	Status
r-binary-packages
r-recheck-most
test-r-arrow-backwards-compatibility
test-r-depsource-bundled
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-extra-packages
test-r-gcc-11
test-r-gcc-12
test-r-install-local
test-r-install-local-minsizerel
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-sanitizers
test-r-linux-valgrind
test-r-m1-san
test-r-macos-as-cran
test-r-minimal-build
test-r-offline-maximal
test-r-offline-minimal
test-r-rhub-debian-gcc-devel-lto-latest
test-r-rhub-ubuntu-gcc12-custom-ccache
test-r-rhub-ubuntu-release-latest
test-r-rocker-r-ver-latest
test-r-rstudio-r-base-4.1-focal
test-r-rstudio-r-base-4.2-focal
test-r-ubuntu-22.04
test-r-versions

raulcd · 2025-12-11T12:43:54Z

@github-actions crossbow submit -g r -g cpp

github-actions · 2025-12-11T12:47:16Z

Revision: 736f045

Submitted crossbow builds: ursacomputing/crossbow @ actions-551409b403

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
r-binary-packages
r-recheck-most
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1
test-cuda-cpp-ubuntu-24.04-cuda-13.0.2
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-42-cpp
test-r-arrow-backwards-compatibility
test-r-depsource-bundled
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-extra-packages
test-r-gcc-11
test-r-gcc-12
test-r-install-local
test-r-install-local-minsizerel
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-sanitizers
test-r-linux-valgrind
test-r-m1-san
test-r-macos-as-cran
test-r-minimal-build
test-r-offline-maximal
test-r-offline-minimal
test-r-rhub-debian-gcc-devel-lto-latest
test-r-rhub-ubuntu-gcc12-custom-ccache
test-r-rhub-ubuntu-release-latest
test-r-rocker-r-ver-latest
test-r-rstudio-r-base-4.1-focal
test-r-rstudio-r-base-4.2-focal
test-r-ubuntu-22.04
test-r-versions
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

github-actions bot added Component: C++ awaiting committer review Awaiting committer review labels Dec 4, 2025

raulcd mentioned this pull request Dec 4, 2025

GH-48315: [C++] Use FetchContent for bundled CRC32C #48318

Merged

raulcd force-pushed the GH-48317 branch 4 times, most recently from 0fd9cd1 to 85620d2 Compare December 5, 2025 11:30

raulcd force-pushed the GH-48317 branch from 70b632a to 6febeaa Compare December 5, 2025 14:39

raulcd added the CI: Extra Run extra CI label Dec 5, 2025

raulcd commented Dec 9, 2025

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Dec 9, 2025

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Dec 9, 2025

kou reviewed Dec 10, 2025

View reviewed changes

cpp/cmake_modules/ThirdpartyToolchain.cmake Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting change review Awaiting change review awaiting changes Awaiting changes labels Dec 10, 2025

kou reviewed Dec 10, 2025

View reviewed changes

cpp/cmake_modules/ThirdpartyToolchain.cmake Outdated Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Dec 10, 2025

kou reviewed Dec 11, 2025

View reviewed changes

cpp/cmake_modules/ThirdpartyToolchain.cmake Outdated Show resolved Hide resolved

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Dec 11, 2025

raulcd and others added 14 commits December 11, 2025 11:31

apacheGH-48315: [C++] Use FetchContent for bundled Crc32c

b109fc8

apacheGH-48317: [C++] Use FetchContent for bundled google-cloud-cpp

50d1fed

Add absl libraries to ARROW_BUNDLED_STATIC_LIBS if ABSL_VENDORED for …

8720d2b

…google cloud

Try different approach of defining ARROW_BUNDLED_STATIC_LIBS with par…

3482732

…ent scope

Set install off for crc32c and abseil and add FC_DECLARE_COMMON_OPTIO…

cd3bf7b

…NS for google-cloud

Try something that is probably wrong

f161d16

Fix patch to not install header files nor config files

ca527d9

add google_cloud_cpp_storage_grpc to the patch

74f352b

Try installing patch/git for R bundled

f45fdcc

Apply suggestion from review

8b9512d

Set EXCLUDE_FROM_ALL for Abseil

1327ea4

Enable Abseil install

f6304d7

Fix a typo

58cccdd

Use time instead of absl_time due to ABSL_ENABLE_INSTALL=ON

eab1602

raulcd force-pushed the GH-48317 branch from 08c3294 to eab1602 Compare December 11, 2025 10:31

Remove all remnants from unnecessary patch

736f045

raulcd marked this pull request as ready for review December 11, 2025 21:43

raulcd requested a review from kou December 11, 2025 21:44

GH-48317: [C++] Use FetchContent for bundled google-cloud-cpp #48333

Are you sure you want to change the base?

GH-48317: [C++] Use FetchContent for bundled google-cloud-cpp #48333

Conversation

raulcd commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

raulcd commented Dec 5, 2025

Uh oh!

github-actions bot commented Dec 5, 2025

Uh oh!

raulcd commented Dec 5, 2025

Uh oh!

github-actions bot commented Dec 5, 2025

Uh oh!

raulcd left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

raulcd commented Dec 9, 2025

Uh oh!

github-actions bot commented Dec 9, 2025

Uh oh!

raulcd commented Dec 9, 2025

Uh oh!

github-actions bot commented Dec 9, 2025

Uh oh!

kou commented Dec 10, 2025

Uh oh!

raulcd commented Dec 10, 2025

Uh oh!

Uh oh!

Uh oh!

kou commented Dec 10, 2025

Uh oh!

Uh oh!

raulcd commented Dec 11, 2025

Uh oh!

github-actions bot commented Dec 11, 2025

Uh oh!

raulcd commented Dec 11, 2025

Uh oh!

github-actions bot commented Dec 11, 2025

Uh oh!

raulcd commented Dec 11, 2025

Uh oh!

github-actions bot commented Dec 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

raulcd commented Dec 4, 2025 •

edited

Loading

raulcd left a comment •

edited

Loading