Possibility to suppress traces in certain scopes like background services

[plachor]

Hi I'm enabling traces in my framework used for building microservices.

In my scenario I'm looking for best option to enable traces on web API requests. So I have added AddAspNetCoreInstrumentation and setup filtering rules within that instrumentation so it is enabling telemetry conditionally upon HTTP request.

Unfortunately my microservices also have multiple background service for which I do not wish to collect traces. Those services are aggressively polling db, Redis and other storage's. They are optimized and do not requires traces yet pollute a lot.

Is there any correct way to mark a scope as not traceable. I could not find similar question nor example but perhaps I missed something.

I found that:

1. starting new activity which is not registered in TracerProviderBuilder will suppress such traces in such background service
2. adding using var suppressedScope = SuppressInstrumentationScope.Begin(); also does the trick

Perhaps there is something build in for that that has minimal performance impact and will suite this case. Or any of my workarounds is valid and can be used with long living background services (application lifetime)

[TimothyMothra]

Hi @plachor,
Sorry I don't fully understand your scenario.

For the sake of conversation, lets assume you have multiple microservices: A -> B -> C -> D

• It sounds like you want to collect Http requests for your first microservice (A), but not for any of the downsteam services.
  If I'm understanding you correctly, in this scenario you would not add OpenTelemetry to B, C, or D.

• You already found two work arounds above 1 & 2, and you were asking about performance impact.
  For 1 there should be no impact. When you call the API to create an Activity, if there is no subscribed listener, no Activity is created. These APIs should return null. For 2, this is the recommended way to suppress calls that you don't want instrumented. For example, my Exporter sends telemetry over HTTP. I don't want traces created for these HTTP calls, so I use the SuppressInstrumentationScope to ignore those calls.

[plachor]

Hi @TimothyMothra,

For 2, this is the recommended way to suppress calls that you don't want instrumented. For example, my Exporter sends telemetry over HTTP. I don't want traces created for these HTTP calls, so I use the SuppressInstrumentationScope to ignore those calls.

Looks like an answer however lets clarify use case, my single microservice has dependencies on many storage's and is also involved in HTTP communication with other microservices.

It can consume request from end user which will trigger traces across multiple microservices as we instrument HTTP calls and storage requests.

Apart from serving API requests which I would like to trace we have several background services for which I would love to suppress traces. They have tight loops that would produce allot of useless traces.

As mentioned I found I can achieve that by:

    public abstract class BackgroundServiceBase : BackgroundService
    {
        protected abstract Task ExecuteWithoutTracesAsync(CancellationToken stoppingToken);
        
        protected sealed override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            // Not sure it was designed for this usecase
            using var suppressedScope = SuppressInstrumentationScope.Begin();

            await ExecuteWithoutTracesAsync(stoppingToken);
        }
    }

I'm just not sure if that is valid way to approach that. Nor if I hit any performance issues doing that in background thread that can work for weeks.

The other working solution I found was:

    public abstract class BackgroundServiceBase : BackgroundService
    {
        protected abstract Task ExecuteWithoutTracesAsync(CancellationToken stoppingToken);
        
        protected sealed override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            // Is there any performance hit in such approach ?
            using var activity = new Activity("SuppressInstrumentationActivity").Start();

            await ExecuteWithoutTracesAsync(stoppingToken);
        }
    }