Add log appender for tracing

[cijothomas]

Opening a PR here to get the initial appender working. Once this repo releases a new api/sdk, this can be shifted to the tracing-opentelemetry repo. Starting here temporarily, as releasing from this repo (+ updating tracing-opentelemetry to pick up the new release) need some time (especially since Metrics has breaking changes). This will allow us to continue polishing the log bridge api/sdk.

Changes

Adds log-appender for the tracing crate, which converts the Events from tracing into LogRecords.
Adds basic example
Add a stress test for testing logs. (Primarily using this to test what is the overhead introduced by OpenTelemetry, so OTel API/SDK log bridge can be optimized before release).

Merge requirement checklist

• CONTRIBUTING guidelines followed
• Unit tests added/updated (if applicable)
• Appropriate CHANGELOG.md files updated for non-trivial, user-facing changes
• Changes in public API reviewed (if applicable)

[codecov]

Codecov Report

Patch coverage: 2.3% and project coverage change: -0.3 ⚠️

Comparison is base (e773f92) 50.7% compared to head (86a583f) 50.5%.

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #1085     +/-   ##
=======================================
- Coverage   50.7%   50.5%   -0.3%     
=======================================
  Files        165     168      +3     
  Lines      19808   19893     +85     
=======================================
+ Hits       10059   10061      +2     
- Misses      9749    9832     +83     

Impacted Files	Coverage Δ
opentelemetry-appender-tracing/src/layer.rs	0.0% <0.0%> (ø)
stress/src/logs.rs	2.3% <2.3%> (ø)
opentelemetry-appender-tracing/src/lib.rs	100.0% <100.0%> (ø)

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[cijothomas]

From SIG meeting notes 05/30/2023:

[Aaron] qn. Why use tracing for logging at all?
log, slog, tracing (events API) - 3 populate logs.

tracing-otel already converts events into SpanEvents, so why is that no sufficient already, and why we need another "bridge" to convert to Logs.?
[Cijo]

1. SpanEvents require Span, and Span's sampling affects SpanEvents.
2. OTel has limits/throttle on the number of SpanEvents per Span
3. SpanEvents are given to Exporters, only when the Span itself is ended/dropped, as opposed to Logs(events) being sent to Exporters immediately (depending on Export Processor.)

[Lalit]
tracing allows events to be created outside Span, but SpanEvents in OTel API does not allow it outside tracing API.

The log facade could be used for pure logging., but does not allow structured logs.

Tracing crate already has ability to emit Logs Always (need to explore - https://docs.rs/tracing/latest/tracing/#emitting-log-records)

event!(Level::TRACE, greeting = ?my_struct);
How do we user decide if this should go as span_event or log?

1. We need to investigate/explore this more. (there may be "Filter" that can be leverages Metadata)
2. We need end-user feedback.

Python
Log API
Otel Tracing API (Spans, SpanEvents)

Rust
Log Crate -- the answer is similar to Python/other languages
Tracing crate - that has "events" and "Span", so the distinction is not easy to make.
Otel Tracing API (Spans, SpanEvents)

[davidbarsky]

Tracing crate already has ability to emit Logs Always (need to explore - https://docs.rs/tracing/latest/tracing/#emitting-log-records)

You shouldn't enable this feature for opentelemetry-rust. it's meant for libraries like hyper or tokio to smoothly transition to tracing in a minor semver release, allowing users to not immediately lose logging when then update their dependencies.

How do we user decide if this should go as span_event or log? [...] We need to investigate/explore this more. (there may be "Filter" that can be leverages Metadata)

Using dynamic_filter_fn is an option, but be sure to benchmark this since it might be a little too costly. We can always improve it on the tracing side of things though, especially since there is some low-hanging, performance-related fruit in tracing-subscriber that we've been meaning to improve. Y'all saying that it's a little too slow would sufficient for me to justify the positive impact of me working on fixing it.

[valkum]

Moving over here from the Discord discussion @djc and I started.

Is there a downside to supporting basically all configured paths? i.e. if the user has log events and tracing-log feature enabled then bridge those, if an otel tracer provider is configured then emit span events, and if a log provider is configured then emit logs.

This would not work properly with our use case.
As mentioned by djc, we want to export our traces and logs to Datadog.
Datadog only considers LogRecords as Logs. They interpret SpanEvents as opaque structured tags, and will not show them in their Logging UI or index them.

So for now, it would be sufficient for us to be able to just emit all tracing events as LogRecords.

But for an overall better usability of tracing-openetelemtry I guess being able to use both would be good.
E.g. during setup I would expect to be able to tell tracing-openetelemtry what to do with tracing-log's tracing events, and I should be able to mark tracing events so that tracing-openetelemtry can route them as desired (LogRecords or SpanEvents).

One way to route them would be to add another special field.
If we add another field, similar to otel.name, et al.,tracing-openetelemtry certainly also should provide a new set of macros which extend/overwrite the tracing macros.
E.g. info!, warn!, debug!, and trace! which will call the tracing macros of the same name but will inject the discriminator for logging.

I also like the idea to utilize the Layer system of tracing to write custom filter rules that will allow the user to decide which events should end up in SpanEvents and LogRecords. tracing-opentelemetry would need to provide two sinks which only handle on_event and a top layer which will handle the current span relation.

I also would like to keep the current behavior of marking a span as failed if an error event was emitted inside the span.

[cijothomas]

@djc @jtescher @davidbarsky @valkum I marked this PR as non-draft, to unblock progress. Once we release Log Bridge API/SDK, this will be moved to tracing-opentelemetry repo (with some configs to let user decide how to export events). We can continue the discussion about how exactly that config looks like in tracing-opentelemetry repo. I'll need some helps there, will open an issue there.

For now, this PR (and follow ups here), can help validate the Log Bridge API/SDK., help test its perf etc.

[cijothomas]

@open-telemetry/rust-approvers Could you help review?
Also see : #1085 (comment)