Trace sampling

[Geal]

related:

• Tracing: What do we want to measure and how #29
• Opentelemetry-jaeger scalability #50
• test trace sampling #193
• Tracing fixes #237
• implement trace sampling #224
• Tail sampling tokio-rs/tracing#1768

request tracing can be expensive, we should have a way to select which traces should be sent to jaeger or opentelemetry.

[Geal]

Following up on the tests in #224 and #237, and after discussing it, the idea of adapting the sampling rate on the result (example: aggressive sampling for successful queries, no sampling for failed ones so we can receive them all) has tradeoffs we might not want to make yet: this requires that we do tail sampling, where we store all of the spans until the end of the query, and then decide if we want to send the trace.

This can result in high memory usage, especially with long running queries. This will also be difficult to get right.
So as a first phase, I'll set up tracing-opentelemetry's head-based sampling, with a rate defined in the configuration, where the sampling decision is done once at the beginning of the trace.
If needed, users will be able to change the configuration and reload it to adapt the rate for debugging

[jwilm]

Hi @Geal,

I saw your issue link and thought I'd drop a note here. We ended up implementing a proof-of-concept tail sampling for tracing-opentelemetry. The code is available publicly, but we still consider it experimental. That said, we have it deployed as part of a high volume application, and it has been running without issue for about 1 week. Actually, we've had better memory utilization than without it due to fewer spans being sent out of the system.

To utilize the tail sampling layer, we internally added an extension method to the tracing::Span type that looks like this:

    fn drop_this_trace(&self) {
        dispatcher::get_default(|d| {
            let registry = d
                .downcast_ref::<Registry>()
                .expect("Subscriber is not Registry; this is a bug.");

            let span = d.current_span();
            let spanref = registry.span(&span.id().unwrap()).unwrap();
            let extensions = spanref.extensions();
            let trace_context = extensions.get::<TraceContext>().unwrap();
            let mut trace_ext = trace_context.trace.extensions_mut();
            trace_ext.insert(SampleDecision {
                record_trace: false,
            });
        });
    }

For now we are just calling this in our application code, but it should be possible to write a tracing layer to be inserted before the tracing-opentelemetry layer to look for span attributes having a particular status code (if I'm understanding your use case correctly) and adding the SamplingDecision to the trace context extensions as in the code snippet.

If you want to try it out, I'd be happy to answer any questions or take feedback you may have.

Cheers!