Default APM tracing configuration wastes CPU on exception throwing

[original-brownbear]

Since the APM tracing was merged I see lots of exception/error throwing in profiles for nodes that have no configuration for APM set up.
I think something needs to be fixed here to make the APM calls noops without configuration or so.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[ChrisHegarty]

Summarising an offline conservation.

From a quick initial investigation of PIT searches, there is quite the slowdown end-to-end for scenarios that runs many tasks/transport-messages.

The Otel API call, Span::fromContextOrNull, when passed a null ends up logging the fact that null was passed to it, which leads to the creation of a new AssertionError() [1] - which we can see in the flame graph. This assertion error is triggered solely to capture the current stack trace. The issue is with the overhead of filling in the stack trace information that is being using for logging. It's costly and the likely majority of the new overhead.

The Otel code appears to be using java.util.logging for its logging purposes. The API usage requires the creation of the Exception a priori, even if the logger is not configured or the level not reached!

Is there a way through the Otel API that avoids this log new AssertionError route? We may not even be configuring or enabling the Otel logging, but will still pay the price here regardless.

It would also be prudent to file an issue with Otel about this.

[1] https://github.com/open-telemetry/opentelemetry-java/blob/4bfe17fb76c291f774de30e53cd3698e89e50adc/api/all/src/main/java/io/opentelemetry/api/internal/ValidationUtil.java#L54

[ChrisHegarty]

The underlying root cause of this issue has been fixed upstream (in Otel), and we have a good workaround in ES code.

If there is nothing remaining, then this issue can be closed.

Caveat, we can likely remove the workaround in ES once the upstream fix gets into a release which ES can leverage. This can be tracked as a separate ES issue.

[original-brownbear]

Alright then, closing here for now.