Logs do not get attached to trace_id when using `#[instrument]

[ewoolsey]

Please see this issue for more context

When looking at the output of the logging pipeline, the Trace ID field is blank for all logs emitted inside a function marked by #[instrument]

This does not emit a trace id

#[instrument]
pub async fn my_func()  {
    trace!("fun log");
}

But this does

#[instrument]
pub async fn my_func() {
    let tracer = opentelemetry::global::tracer("ex.com/basic");
    tracer.in_span("cool span", |_| {
        trace!("fun log");
    });
}

I'm not sure exactly what's going wrong under the hood.

Here is my initial setup

pub fn init_tracing() -> Result<()> {
    let stdout = tracing_subscriber::fmt::layer()
        .with_writer(std::io::stdout)
        .compact()
        .without_time()
        .with_target(false)
        .with_line_number(true)
        .with_file(true)
        .with_filter(EnvFilter::from_default_env());

    let subscriber;
    #[cfg(debug_assertions)]
    {
        let console = console_subscriber::spawn();
        subscriber = Registry::default().with(stdout).with(console)
    }
    #[cfg(not(debug_assertions))]
    {
        use crate::metrics::init_meter_provider;
        use opentelemetry::KeyValue;
        use opentelemetry_appender_tracing::layer::OpenTelemetryTracingBridge;
        use opentelemetry_sdk::{trace, Resource};
        use tracing_opentelemetry::MetricsLayer;

        let (sender, receiver) = tokio::sync::mpsc::unbounded_channel();
        tokio::spawn(log_handler(receiver));

        let paper_trail = tracing_subscriber::fmt::layer()
            .with_writer(MakePaperTrailWriter { sender })
            .compact()
            .without_time()
            .with_target(false)
            .with_line_number(true)
            .with_file(true)
            .with_filter(EnvFilter::from_default_env());

        let tracing = tracing_opentelemetry::layer()
            .with_tracer(
                opentelemetry_otlp::new_pipeline()
                    .tracing()
                    .with_exporter(opentelemetry_otlp::new_exporter().http())
                    .with_trace_config(trace::config().with_resource(Resource::new(vec![
                        KeyValue::new("service.name", "liquidation-bot-rs"),
                    ])))
                    .install_batch(opentelemetry_sdk::runtime::Tokio)?,
            )
            .with_filter(EnvFilter::from_default_env());

        let logs = opentelemetry_otlp::new_pipeline()
            .logging()
            .with_exporter(opentelemetry_otlp::new_exporter().http())
            .with_log_config(
                opentelemetry_sdk::logs::config().with_resource(Resource::new(vec![
                    KeyValue::new("service.name", "liquidation-bot-rs"),
                ])),
            )
            .install_batch(opentelemetry_sdk::runtime::Tokio)?;

        let logging = OpenTelemetryTracingBridge::new(&logs.provider().unwrap())
            .with_filter(EnvFilter::from_default_env());

        let metrics = MetricsLayer::new(init_meter_provider()?);

        subscriber = Registry::default()
            .with(stdout)
            .with(logging)
            .with(paper_trail)
            .with(metrics)
            .with(tracing)
    }

    tracing::subscriber::set_global_default(subscriber).expect("setting tracing default failed");
    Ok(())
}

[ewoolsey]

Anyone have any ideas for a potential solution here?

[davidbarsky]

Sorry for the late response, but looking over the linked issue on opentelemetry-rust, it seems like the short-term solution might be open-telemetry/opentelemetry-rust#1394, but I think a longer-term solution would require some sort of API on tracing_subscriber::fmt's end.

(I'll close this for now, but let me know if this should be re-opened.)

[nastynaz]

@ewoolsey did you ever find a workaround for this? I see you're collecting both logs and traces, how are you correlating them? I've spend a while trying to figure it out to no avail, any help would be much appreciated.