forked from dremio/arrow
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
apacheGH-39898: [C++] Add support for OpenTelemetry logging (apache#3…
…9905) <!-- Thanks for opening a pull request! If this is your first pull request you can find detailed information on how to contribute here: * [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request) * [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html) If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project. Then could you also rename the pull request title in the following format? GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY} or MINOR: [${COMPONENT}] ${SUMMARY} In the case of PARQUET issues on JIRA the title also supports: PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY} --> ### Rationale for this change Supporting OTel logs will help us improve diagnostics/debugging where OTel tracing is utilized. ### What changes are included in this PR? Primary changes: - Bumps `opentelemetry-cpp` version to 1.13.0 to access the stable logs SDK - Introduces a new `ARROW_TELEMETRY` module for these additions (and in anticipation of future OpenMetrics support) - Integrates Otel logging facilities, provides `telemetry::Logger` class that wraps an OTel logger and an API for creating loggers via OTel's global logger provider - Adds developer-friendly log record exporters, mimicking the already-existing span exporters Some auxiliary, but significant additions: - Adds an extended/re-imagined version of the current `ArrowLog` APIs that aims to be more flexible and configurable - which the OTel loggers currently utilize. Notable details: - Adds an abstract `util::Logger` class that enables creating custom loggers. - Adds a `LoggerRegistry` for global access to an arbitrary number of loggers - Adds new `ARROW_LOG_*` macros that take individual loggers as a parameter. Additionally, they can be stripped at compile time based on a minimum log level - Definitely looking for opinions on this part of the PR, specifically... - Adds `ArrowLogLevel::ARROW_TRACE` as the lowest log level to mirror the equivalent OTel enums NOTE: I've added some log statements that utilize the OTel facilities to Flight/FlightSQL - which are driven by the FlightSQL server tests. These are for demonstrative purposes only and I intend to remove them prior to merge ### Are these changes tested? Yes (although the OTel-specific tests are currently light) ### Are there any user-facing changes? This will introduce new APIs that are likely to be public. <!-- If there are any breaking changes to public APIs, please uncomment the line below and explain which changes are breaking. --> <!-- **This PR includes breaking changes to public APIs.** --> <!-- Please uncomment the line below (and provide explanation) if the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld). We use this to highlight fixes to issues that may affect users without their knowledge. For this reason, fixing bugs that cause errors don't count, since those are usually obvious. --> <!-- **This PR contains a "Critical Fix".** --> * Closes: apache#39898 * GitHub Issue: apache#39898
- Loading branch information
Showing
27 changed files
with
1,410 additions
and
10 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| // Licensed to the Apache Software Foundation (ASF) under one | ||
| // or more contributor license agreements. See the NOTICE file | ||
| // distributed with this work for additional information | ||
| // regarding copyright ownership. The ASF licenses this file | ||
| // to you under the Apache License, Version 2.0 (the | ||
| // "License"); you may not use this file except in compliance | ||
| // with the License. You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, | ||
| // software distributed under the License is distributed on an | ||
| // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| // KIND, either express or implied. See the License for the | ||
| // specific language governing permissions and limitations | ||
| // under the License. | ||
|
|
||
| #include <string_view> | ||
|
|
||
| #include "arrow/flight/otel_logging.h" | ||
| #include "arrow/flight/otel_logging_internal.h" | ||
| #include "arrow/result.h" | ||
| #include "arrow/util/logger.h" | ||
| #include "arrow/util/logging.h" | ||
|
|
||
| namespace arrow::flight { | ||
|
|
||
| namespace { | ||
| constexpr std::string_view kGrpcClientName = "arrow-flight-grpc-client-otel"; | ||
| constexpr std::string_view kGrpcServerName = "arrow-flight-grpc-server-otel"; | ||
| constexpr std::string_view kSqlClientName = "arrow-flight-sql-client-otel"; | ||
| constexpr std::string_view kSqlServerName = "arrow-flight-sql-server-otel"; | ||
| } // namespace | ||
|
|
||
| Status RegisterFlightOtelLoggers(const telemetry::OtelLoggingOptions& options) { | ||
| for (auto name : {kGrpcClientName, kGrpcServerName, kSqlClientName, kSqlServerName}) { | ||
| ARROW_ASSIGN_OR_RAISE(auto logger, | ||
| telemetry::OtelLoggerProvider::MakeLogger(name, options)); | ||
| DCHECK_NE(logger, nullptr); | ||
| ARROW_RETURN_NOT_OK(util::LoggerRegistry::RegisterLogger(name, std::move(logger))); | ||
| } | ||
| return Status::OK(); | ||
| } | ||
|
|
||
| namespace internal { | ||
|
|
||
| std::shared_ptr<util::Logger> GetOtelGrpcClientLogger() { | ||
| return util::LoggerRegistry::GetLogger(kGrpcClientName); | ||
| } | ||
| std::shared_ptr<util::Logger> GetOtelGrpcServerLogger() { | ||
| return util::LoggerRegistry::GetLogger(kGrpcServerName); | ||
| } | ||
| std::shared_ptr<util::Logger> GetOtelSqlClientLogger() { | ||
| return util::LoggerRegistry::GetLogger(kSqlClientName); | ||
| } | ||
| std::shared_ptr<util::Logger> GetOtelSqlServerLogger() { | ||
| return util::LoggerRegistry::GetLogger(kSqlServerName); | ||
| } | ||
|
|
||
| } // namespace internal | ||
|
|
||
| } // namespace arrow::flight |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| // Licensed to the Apache Software Foundation (ASF) under one | ||
| // or more contributor license agreements. See the NOTICE file | ||
| // distributed with this work for additional information | ||
| // regarding copyright ownership. The ASF licenses this file | ||
| // to you under the Apache License, Version 2.0 (the | ||
| // "License"); you may not use this file except in compliance | ||
| // with the License. You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, | ||
| // software distributed under the License is distributed on an | ||
| // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| // KIND, either express or implied. See the License for the | ||
| // specific language governing permissions and limitations | ||
| // under the License. | ||
|
|
||
| #pragma once | ||
|
|
||
| #include "arrow/util/config.h" | ||
|
|
||
| #ifdef ARROW_WITH_OPENTELEMETRY | ||
| #include "arrow/status.h" | ||
| #include "arrow/telemetry/logging.h" | ||
| #include "arrow/util/macros.h" | ||
|
|
||
| namespace arrow::flight { | ||
|
|
||
| ARROW_EXPORT Status | ||
| RegisterFlightOtelLoggers(const telemetry::OtelLoggingOptions& options); | ||
|
|
||
| } // namespace arrow::flight | ||
| #endif |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| // Licensed to the Apache Software Foundation (ASF) under one | ||
| // or more contributor license agreements. See the NOTICE file | ||
| // distributed with this work for additional information | ||
| // regarding copyright ownership. The ASF licenses this file | ||
| // to you under the Apache License, Version 2.0 (the | ||
| // "License"); you may not use this file except in compliance | ||
| // with the License. You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, | ||
| // software distributed under the License is distributed on an | ||
| // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| // KIND, either express or implied. See the License for the | ||
| // specific language governing permissions and limitations | ||
| // under the License. | ||
|
|
||
| #pragma once | ||
|
|
||
| #include "arrow/util/config.h" | ||
|
|
||
| #include "arrow/util/macros.h" | ||
| #ifdef ARROW_WITH_OPENTELEMETRY | ||
| #include "arrow/flight/otel_logging.h" | ||
| #include "arrow/util/logger.h" | ||
|
|
||
| namespace arrow::flight::internal { | ||
|
|
||
| ARROW_EXPORT std::shared_ptr<util::Logger> GetOtelGrpcClientLogger(); | ||
| ARROW_EXPORT std::shared_ptr<util::Logger> GetOtelGrpcServerLogger(); | ||
| ARROW_EXPORT std::shared_ptr<util::Logger> GetOtelSqlClientLogger(); | ||
| ARROW_EXPORT std::shared_ptr<util::Logger> GetOtelSqlServerLogger(); | ||
|
|
||
| } // namespace arrow::flight::internal | ||
|
|
||
| #define ARROW_FLIGHT_OTELLOG_CLIENT(LEVEL, ...) \ | ||
| ARROW_LOGGER_CALL(::arrow::flight::internal::GetOtelGrpcClientLogger(), LEVEL, \ | ||
| __VA_ARGS__) | ||
| #define ARROW_FLIGHT_OTELLOG_SERVER(LEVEL, ...) \ | ||
| ARROW_LOGGER_CALL(::arrow::flight::internal::GetOtelGrpcServerLogger(), LEVEL, \ | ||
| __VA_ARGS__) | ||
| #define ARROW_FLIGHT_OTELLOG_SQL_CLIENT(LEVEL, ...) \ | ||
| ARROW_LOGGER_CALL(::arrow::flight::internal::GetOtelSqlClientLogger(), LEVEL, \ | ||
| __VA_ARGS__) | ||
| #define ARROW_FLIGHT_OTELLOG_SQL_SERVER(LEVEL, ...) \ | ||
| ARROW_LOGGER_CALL(::arrow::flight::internal::GetOtelSqlServerLogger(), LEVEL, \ | ||
| __VA_ARGS__) | ||
|
|
||
| #else | ||
|
|
||
| #define ARROW_FLIGHT_OTELLOG_CLIENT(LEVEL, ...) ARROW_UNUSED(0) | ||
| #define ARROW_FLIGHT_OTELLOG_SERVER(LEVEL, ...) ARROW_UNUSED(0) | ||
| #define ARROW_FLIGHT_OTELLOG_SQL_CLIENT(LEVEL, ...) ARROW_UNUSED(0) | ||
| #define ARROW_FLIGHT_OTELLOG_SQL_SERVER(LEVEL, ...) ARROW_UNUSED(0) | ||
|
|
||
| #endif |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.