Decoupling Event API from Logs API

[scheler]

Fixes #3086

Changes

Decouples Event API from Logs API. Now that the EventLogger cannot created from a Logger, it has be to created from a Provider, hence the Event API spec now includes EventLoggerProvider as well.

Related issues # #3086

[reyang]

I feel this is going to introduce lots of confusion, and will bring issues to programming languages which have established logging APIs (and many of these established logging APIs can also cover "events").

My suggestion:

• stick with the 3 pillars of observability concept, do not introduce a 4th pillar
• we should design logging API, this is critical for programming languages which do not have established logging API (e.g. C++)
• make it clear that the logging API is only intended to be implemented by SIGs whose programming languages do not have well established logging APIs (such as Log4J and ILogger)
• work with other communities (e.g. owner of Log4J/ILogger) if there are some great logging feature OpenTelemetry discovered which should be nice additions to those libraries

[scheler]

@reyang not sure why this is coming up now. It was accepted long ago that we need an Events API in OpenTelemetry and that the existing Logging libraries in various languages do not have sufficient support for creating Events. Please see the reasoning in the OTEP here.

[reyang]

@reyang not sure why this is coming up now. It was accepted long ago that we need an Events API in OpenTelemetry and that the existing Logging libraries in various languages do not have sufficient support for creating Events. Please see the reasoning in the OTEP here.

@scheler sorry I wasn't involved in the logging SIG and the OTEP. I'll take a read to get more context.

I was mainly raising the questions from the implementation side, for example:

1. In OpenTelemetry C++, there is no logging API spec and we do need something there.
2. In OpenTelemetry .NET, there is an established API which exists for more than a decade, and seems it has covered the scenario which we're trying to achieve here.
3. In Java, log4j is well established, is there overlap between the proposed event api and log api, and if yes, how should the user decide which one to use.

[scheler]

@reyang the push for the Events API is mostly coming from the RUM / Client Instrumentation SIG currently. Web JavaScript, iOS and Android all do not have an Event API along with an Appender like mechanism to ship the events to any destination behind the scenes. For languages and situations where this is feasible, it is still recommended that Instrumentation authors use more popular existing APIs and the SDK authors build the corresponding Appenders using the Event SDK.

[tigrannajaryan]

I feel this is going to introduce lots of confusion, and will bring issues to programming languages which have established logging APIs (and many of these established logging APIs can also cover "events").

@reyang thanks for raising the concern. I'll try to address them below.

My suggestion:

• stick with the 3 pillars of observability concept, do not introduce a 4th pillar

We are not introducing a new pillar. The current understanding is that events are specially shaped log records (i.e. the log records have certain attributes that are defined in semantic conventions). We also want to provide helper API to create events easily.

This is somewhat analogous to span exceptions. There are semantic conventions for recording exceptions as span events and there is a helper API to record exceptions on a span easily.

• we should design logging API, this is critical for programming languages which do not have established logging API (e.g. C++)

The currently specified Logs API is a low-level API for creating log records. It is not intended to be used by end users for as a general-purpose logging API. The purpose of the Logs API is to allow building appenders and bridges from existing well-known logging libraries to Otel. Sometimes APIs like this are also called "log backends", "log handlers" in some languages.

For languages where there is no established logging API for end users we can in the future add "log frontend" API to Otel spec. We do not have this yet, but nothing prevents us from adding it in the future.

• make it clear that the logging API is only intended to be implemented by SIGs whose programming languages do not have well established logging APIs (such as Log4J and ILogger)

I think the spec does this. Please see the first paragraph in bold here: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/api.md

If this is not clear enough please suggest some additional wording.

• work with other communities (e.g. owner of Log4J/ILogger) if there are some great logging feature OpenTelemetry discovered which should be nice additions to those libraries

We believe we have a good solution for Java, .Net and Python ecosystems (we have prototypes for these languages). We are working with Go community to make sure the newly proposed slog package works nicely with Otel (it doesn't currently, we are trying to fix it).

[reyang]

I feel this is going to introduce lots of confusion, and will bring issues to programming languages which have established logging APIs (and many of these established logging APIs can also cover "events").

@reyang thanks for raising the concern. I'll try to address them below.

My suggestion:

• stick with the 3 pillars of observability concept, do not introduce a 4th pillar

We are not introducing a new pillar. The current understanding is that events are specially shaped log records (i.e. the log records have certain attributes that are defined in semantic conventions). We also want to provide helper API to create events easily.

This is somewhat analogous to span exceptions. There are semantic conventions for recording exceptions as span events and there is a helper API to record exceptions on a span easily.

I have two concerns:

• For OpenTelemetry C++, there is clear demand for user-facing logging API, it might be confusing to have both logging and "event" API.
• For other programming languages which already have some established logging APIs, some of them have already provided all the necessary features to describe an "event". For example, https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.logging.eventid?view=dotnet-plat-ext-7.0

logger.LogInformation(eventId: new EventId(123, "Sample.Hello"), "Hello from {name} {price}.", "tomato", 2.99);

It makes less sense for OpenTelemetry .NET to introduce yet another "event API" - users would be confused if that happened.

• we should design logging API, this is critical for programming languages which do not have established logging API (e.g. C++)

The currently specified Logs API is a low-level API for creating log records. It is not intended to be used by end users for as a general-purpose logging API. The purpose of the Logs API is to allow building appenders and bridges from existing well-known logging libraries to Otel. Sometimes APIs like this are also called "log backends", "log handlers" in some languages.

For languages where there is no established logging API for end users we can in the future add "log frontend" API to Otel spec. We do not have this yet, but nothing prevents us from adding it in the future.

+1 on we can, and this is exactly what we need to clarify, please find the discussion here #2911 (comment).

• make it clear that the logging API is only intended to be implemented by SIGs whose programming languages do not have well established logging APIs (such as Log4J and ILogger)

I think the spec does this. Please see the first paragraph in bold here: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/api.md

If this is not clear enough please suggest some additional wording.

I like what @trask proposed here #2911 (comment).

• work with other communities (e.g. owner of Log4J/ILogger) if there are some great logging feature OpenTelemetry discovered which should be nice additions to those libraries

We believe we have a good solution for Java, .Net and Python ecosystems (we have prototypes for these languages). We are working with Go community to make sure the newly proposed slog package works nicely with Otel (it doesn't currently, we are trying to fix it).

Thanks for sharing, this is really nice! I can share that OpenTelemetry .NET and the .NET runtime folks are working closely, and the OpenTelemetry C++ group had several discussions with the ISO C++ standard committee folks.

[tigrannajaryan]

• For OpenTelemetry C++, there is clear demand for user-facing logging API, it might be confusing to have both logging and "event" API.
• For other programming languages which already have some established logging APIs, some of them have already provided all the necessary features to describe an "event". For example, https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.logging.eventid?view=dotnet-plat-ext-7.0

Thanks @reyang. This is a good feedback for Events API folks @scheler @MSNev.

We already decided that Events API will not be stabilized together with Log API/SDK. It will be a separate effort later.

@scheler @MSNev it would be useful to show whether C++ is the outlier here that does not need an Events API or that JS is the outlier that needs a separate Events API. Depending on which of these 2 things is true we may make a different decision about making Events API the part of the Otel Spec.

[scheler]

@tigrannajaryan @reyang a reminder that browser/javascript instrumentation is not the only one asking for Events API. It is for android/java and ios/swift too. None of these environments have built-in Events API or one with Appender like mechanism. Do we need to revisit the OTEP that introduced Events API?

[tigrannajaryan]

@tigrannajaryan @reyang a reminder that browser/javascript instrumentation is not the only one asking for Events API. It is for android/java and ios/swift too. None of these environments have built-in Events API or one with Appender like mechanism. Do we need to revisit the OTEP that introduced Events API?

One possible way to resolve this is to make it clear that the Events API as a whole is optional. Some languages may implement it, many others may decide that it is not necessary.

[MSNev]

@tigrannajaryan @reyang @scheler

I believe that ALL Client Side focused SDK's that are looking to capture User / Application level usage telemetry WILL require what we are describing as the EventAPI. Which is a different use-case from what is described as

some of them have already provided all the necessary features to describe an "event". For example, https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.logging.eventid?view=dotnet-plat-ext-7.0

As the above EventId appears to be more akin to general (Server/System style) logging where you can provide additional context around the "Event" that caused the produced logging entry (ie. Windows Event viewer).

Yes, they could be made to operate in the same manner to provide the same level of detail, but then you end up talking about convenience methods / helpers to construct / use the language level provided logging mechanism.

For example, I see that for C++ / .Net it is very likely that a possible SDK implementation of the "EventAPI" implementation internally uses the Microsoft.Extensions.Logging rather than the OpenTelemetry Log API (because the OpenTelemetry Log API is probably using it anyway), thus it becomes the "defined" convenience methods to construct the User / Application level usage telemetry (which could also be viewed as a Structured log).

And in the case of the "EventId" this would an OpenTelemetry defined ID (name / domain) where the implementation to "decode" the Log record would be provided by an OpenTelemetry registered DLL (or automatically provided framework version) to see the detail of the event with all of it's attributes including the event.data which is the strutured form for these Events.

[CodeBlanch]

I'm with @reyang on this one. #3086 demonstrated events can be implemented using the Log API. I think OpenTelemetry Specification should define Log Appender API (borrowing @jack-berg's name) and nothing else. We can make a dedicated specification for OpenTelemetry Events as an addon thing which uses LogAppender API. Language SIGs may or may not ship it. We could also make a dedicated specification for OpenTelemetry Logging (targeting languages like C++) which also just uses the main spec LogAppender API and is optional. Doesn't seem to me to be a strong need for this stuff in the main OpenTelemetry Specification.

[reyang]

I believe that ALL Client Side focused SDK's that are looking to capture User / Application level usage telemetry WILL require what we are describing as the EventAPI. Which is a different use-case from what is described as

I think we should revisit the scope of the client instrumentation #2734 (comment) - I believe that client specific semantic conventions do have value, the instrumentation APIs should be designed in a fairly agnostic way, rather than limiting its functionality to clients (then later we'll have to invent similar things for libraries, servers, operating systems, devices, etc.).

[lalitb]

Just to summarize requirements from C++ prospective

• Event API could be made as optional, to be implemented by SIGs/languages needing it.
• Log API could be renamed to Log SPI/ Log backend API / Log Appender API to make it clear this is not user-facing API.
• (In future) - OpenTelemetry defines a user-facing Log API, or else let languages define their own API ( if C++ is the only one without any logging framework).

[scheler]

For OpenTelemetry C++, there is clear demand for user-facing logging API, it might be confusing to have both logging and "event" API.

@lalitb @reyang can you tell a bit more about this - is the user-facing logging API meant for use by the instrumentation authors or application authors? If it is application authors, will they always be using the API along with the SDK, as the API only has no-op implementation of the logging API.

[reyang]

For OpenTelemetry C++, there is clear demand for user-facing logging API, it might be confusing to have both logging and "event" API.

@lalitb @reyang can you tell a bit more about this - is the user-facing logging API meant for use by the instrumentation authors or application authors? If it is application authors, will they always be using the API along with the SDK, as the API only has no-op implementation of the logging API.

@scheler it is for both of them.

[jkwatson]

I do not think that the OpenTelemetry logging "API/SPI/whatever" spec should include an end-user facing logging API. If C++ needs this, it should be a separate, independent effort that does not impact the rest of the languages that are already overflowing with existing, established logging APIs.

[scheler]

yes, since end-user facing logging API requires rich configuration capabilities too.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[MSNev]

not stale

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[github-actions]

Closed as inactive. Feel free to reopen if this PR is still being worked on.

[scheler]

@tigrannajaryan can you please reopen this PR?

In #3086 (comment) we discussed that the Event API should not depend on logger which is what this PR did.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[github-actions]

Closed as inactive. Feel free to reopen if this PR is still being worked on.

[mattmccleary]

I received some insightful feedback from a customer who wants a way to generate business events like "Invoice Downloaded". They see this as an altogether different category of events than those that have to do with "the working of the application". The customer is considering building their own mechanism for "business event logging", distinct from their monitoring/tracing solution. At any rate, what is required to reach a decision re how the community plans to proceed?