[C++] Allow building against system opentelemetry

[h-vetinari]

Describe the enhancement requested

I've been wanting to add opentelemetry support in conda-forge for a long time, but given the amount of mutation that happens in packaging there, I wasn't comfortable in doing that before landing conda-forge/arrow-cpp-feedstock#1058.

Now that this is in, I've rebased the PR, but am running into problems pretty quickly. For one, arrow hardcodes

arrow/cpp/cmake_modules/ThirdpartyToolchain.cmake

Line 4974 in 8428c47

set(opentelemetry-cpp_SOURCE "AUTO")

even if the config explicitly asks for -Dopentelemetry-cpp_SOURCE="SYSTEM". With the following patch

diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake b/cpp/cmake_modules/ThirdpartyToolchain.cmake
index 0b215b5b2..b9cc05953 100644
--- a/cpp/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake
@@ -4977,7 +4977,6 @@ if(ARROW_WITH_OPENTELEMETRY)
   # cURL is required whether we build from source or use an existing installation
   # (OTel's cmake files do not call find_curl for you)
   find_curl()
-  set(opentelemetry-cpp_SOURCE "AUTO")
   resolve_dependency(opentelemetry-cpp)
   set(ARROW_OPENTELEMETRY_LIBS
       opentelemetry-cpp::trace

I then run into further problems in ThirdpartyToolchain.cmake

CMake Error at cmake_modules/ThirdpartyToolchain.cmake:313 (find_package):
  By not providing "Findopentelemetry-cpp.cmake" in CMAKE_MODULE_PATH this
  project has asked CMake to find a package configuration file provided by
  "opentelemetry-cpp", but CMake did not find one.

  Could not find a package configuration file provided by "opentelemetry-cpp"
  with any of the following names:

    opentelemetry-cppConfig.cmake
    opentelemetry-cpp-config.cmake

  Add the installation prefix of "opentelemetry-cpp" to CMAKE_PREFIX_PATH or
  set "opentelemetry-cpp_DIR" to a directory containing one of the above
  files.  If "opentelemetry-cpp" provides a separate development package or
  SDK, be sure it has been installed.
Call Stack (most recent call first):
  cmake_modules/ThirdpartyToolchain.cmake:4980 (resolve_dependency)
  CMakeLists.txt:546 (include)

despite the fact that our otel packages clearly contain the CMake metadata.

CC @lidavidm @kou

Component(s)

C++, Packaging

[assignUser]

Hm you explicitly set SYSTEM vs CONDA. Can you try setting adding export CMAKE_PREFIX_PATH="$CONDA_PREFIX" in build.sh?

[assignUser]

Looking at the code that area of the cmake could probably be substantially improved when we move to newer cmake as discussed in #44950

[kou]

It seems that we can remove the set(opentelemetry-cpp_SOURCE "AUTO").

Could you share full error log URL?

[h-vetinari]

Could you share full error log URL?

I restarted a new build, here are the logs

Hm you explicitly set SYSTEM vs CONDA. Can you try setting adding export CMAKE_PREFIX_PATH="$CONDA_PREFIX" in build.sh?

Done, logs here (no change)

[assignUser]

I don't think we modify the cmake prefix path in anyway that should prevent otel from being found. I can confirm that a normal find_package(opentelemetry-cpp) does indeed find it within a conda env without any special additions.
Could you run a build with:

diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake b/cpp/cmake_modules/ThirdpartyToolchain.cmake
index 35ad4089e..c12d6c7af 100644
--- a/cpp/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake
@@ -4966,8 +4966,9 @@ if(ARROW_WITH_OPENTELEMETRY)
   # cURL is required whether we build from source or use an existing installation
   # (OTel's cmake files do not call find_curl for you)
   find_curl()
-  set(opentelemetry-cpp_SOURCE "AUTO")
+  set(CMAKE_FIND_DEBUG_MODE TRUE)
   resolve_dependency(opentelemetry-cpp)
+  set(CMAKE_FIND_DEBUG_MODE FLASE)
   set(ARROW_OPENTELEMETRY_LIBS
       opentelemetry-cpp::trace
       opentelemetry-cpp::logs

[h-vetinari]

@assignUser, here are the logs for your suggestion, lightly cleaned up by resubstituting common variables.

I really don't know ThirdpartyToolchain.cmake, but ISTM that the problem begins by looking for Findopentelemetry-cpp.cmake, rather than just going directly to $PREFIX/lib/cmake/opentelemetry-cpp/opentelemetry-cpp-config.cmake (which exists as I pointed out in the OP; it appears in a kind of no-man's-land in the log below, no idea what to make of that).

CMake Debug Log at cmake_modules/ThirdpartyToolchain.cmake:313 (find_package):
  find_package considered the following paths for
  Findopentelemetry-cpp.cmake:

    $SRC_DIR/cpp/cmake_modules/Findopentelemetry-cpp.cmake
    $PREFIX/lib/cmake/glog/../../../share/glog/cmake/Findopentelemetry-cpp.cmake
    $PREFIX/lib/cmake/grpc/modules/Findopentelemetry-cpp.cmake
    $BUILD_PREFIX/share/cmake-3.31/Modules/Findopentelemetry-cpp.cmake

  The file was not found.

    $PREFIX/lib/cmake/opentelemetry-cpp/opentelemetry-cppConfig.cmake
    $PREFIX/lib/cmake/opentelemetry-cpp/opentelemetry-cpp-config.cmake              # <- see here

  The internally managed CMAKE_FIND_PACKAGE_REDIRECTS_DIR.

    $SRC_DIR/cpp/build/CMakeFiles/pkgRedirects

  <PackageName>_ROOT CMake variable [CMAKE_FIND_USE_PACKAGE_ROOT_PATH].

    $PREFIX

  CMAKE_PREFIX_PATH variable [CMAKE_FIND_USE_CMAKE_PATH].

    none

  CMAKE_FRAMEWORK_PATH and CMAKE_APPBUNDLE_PATH variables
  [CMAKE_FIND_USE_CMAKE_PATH].

    none

  Env variable opentelemetry-cpp_DIR [CMAKE_FIND_USE_CMAKE_ENVIRONMENT_PATH].

    none

  CMAKE_PREFIX_PATH env variable [CMAKE_FIND_USE_CMAKE_ENVIRONMENT_PATH].

    $BUILD_PREFIX/x86_64-conda-linux-gnu/sysroot/usr

  CMAKE_FRAMEWORK_PATH and CMAKE_APPBUNDLE_PATH env variables
  [CMAKE_FIND_USE_CMAKE_ENVIRONMENT_PATH].

  Paths specified by the find_package HINTS option.

    none

  Standard system environment variables
  [CMAKE_FIND_USE_SYSTEM_ENVIRONMENT_PATH].

    $BUILD_PREFIX
    /opt/conda/condabin
    /opt/conda
    /usr/local
    /usr
    /
    /home/conda

  CMake User Package Registry [CMAKE_FIND_USE_PACKAGE_REGISTRY].

    none

  CMake variables defined in the Platform file
  [CMAKE_FIND_USE_CMAKE_SYSTEM_PATH].

    /usr/X11R6
    /usr/pkg
    /opt

  CMake System Package Registry
  [CMAKE_FIND_PACKAGE_NO_SYSTEM_PACKAGE_REGISTRY].

    none

  Paths specified by the find_package PATHS option.

    none

  Prepending the following roots to each prefix:

  CMAKE_FIND_ROOT_PATH

    $PREFIX
    $BUILD_PREFIX/x86_64-conda-linux-gnu/sysroot

  CMAKE_SYSROOT_COMPILE

    none

  CMAKE_SYSROOT_LINK

    none

  CMAKE_SYSROOT

    none

  find_package considered the following locations for opentelemetry-cpp's
  Config module:

    $PREFIX/opentelemetry-cppConfig.cmake
    $PREFIX/opentelemetry-cpp-config.cmake
    $PREFIX/opentelemetry-cppConfig.cmake
    $PREFIX/opentelemetry-cpp-config.cmake
    $BUILD_PREFIX/x86_64-conda-linux-gnu/sysroot/usr/opentelemetry-cppConfig.cmake
    $BUILD_PREFIX/x86_64-conda-linux-gnu/sysroot/usr/opentelemetry-cpp-config.cmake
    $BUILD_PREFIX/x86_64-conda-linux-gnu/sysroot/usr/opentelemetry-cppConfig.cmake
    $BUILD_PREFIX/x86_64-conda-linux-gnu/sysroot/usr/opentelemetry-cpp-config.cmake
    $BUILD_PREFIX/x86_64-conda-linux-gnu/sysroot/opentelemetry-cppConfig.cmake
    $BUILD_PREFIX/x86_64-conda-linux-gnu/sysroot/opentelemetry-cpp-config.cmake
    $SRC_DIR/cpp/build/CMakeFiles/pkgRedirects/opentelemetry-cppConfig.cmake
    $SRC_DIR/cpp/build/CMakeFiles/pkgRedirects/opentelemetry-cpp-config.cmake
    $PREFIX/opentelemetry-cppConfig.cmake
    $PREFIX/opentelemetry-cpp-config.cmake
    $BUILD_PREFIX/x86_64-conda-linux-gnu/sysroot/usr/opentelemetry-cppConfig.cmake
    $BUILD_PREFIX/x86_64-conda-linux-gnu/sysroot/usr/opentelemetry-cpp-config.cmake
    $BUILD_PREFIX/opentelemetry-cppConfig.cmake
    $BUILD_PREFIX/opentelemetry-cpp-config.cmake
    /opt/conda/condabin/opentelemetry-cppConfig.cmake
    /opt/conda/condabin/opentelemetry-cpp-config.cmake
    /opt/conda/opentelemetry-cppConfig.cmake
    /opt/conda/opentelemetry-cpp-config.cmake
    /usr/local/opentelemetry-cppConfig.cmake
    /usr/local/opentelemetry-cpp-config.cmake
    /usr/opentelemetry-cppConfig.cmake
    /usr/opentelemetry-cpp-config.cmake
    /opentelemetry-cppConfig.cmake
    /opentelemetry-cpp-config.cmake
    /home/conda/opentelemetry-cppConfig.cmake
    /home/conda/opentelemetry-cpp-config.cmake
    /opt/opentelemetry-cppConfig.cmake
    /opt/opentelemetry-cpp-config.cmake

  The file was not found.

Call Stack (most recent call first):
  cmake_modules/ThirdpartyToolchain.cmake:4970 (resolve_dependency)
  CMakeLists.txt:546 (include)

[kou]

Findopentelemetry-cpp.cmake must not be a problem.
If Findopentelemetry-cpp.cmake doesn't exist, opentelemetry-cpp-config.cmake is searched automatically.

Does /home/conda/feedstock_root/build_artifacts/apache-arrow_1733797696439/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/cmake/opentelemetry-cpp/opentelemetry-cpp-config.cmake really exist?