[QUERY] Health check filtering results in parentless issues

[hansmbakker]

Library name and version

Azure.Monitor.OpenTelemetry.Exporter 1.2.0

Query/Question

In #46056 (comment), it was suggested to filter out health check requests.

However, when implementing

builder.Services.Configure<AspNetCoreTraceInstrumentationOptions>(options =>
            {
                options.Filter = (httpContext) =>
                {
                    return httpContext.Request.Path != "/api/health";
                };
            });

you might end up seeing parentless errors and traces when an issue happens in the health check or in a dependency. The root activity (health check API call) is not sent, but the exceptions and dependency errors are still sent. This might be confusing - so be aware of that. For this reason, we removed the filtering again until we found a better way to do this.

Environment

No response

[cijothomas]

Looks like this is an issue specific to Azure Monitor, as it uses a sampler that does not follow the parent's sampling decision. The Filter option in Asp.Net core propagates sampling decision to child spans, so when using samplers that respect parent sampling, it ensures that all child spans are also sampled out, when the Health request is filtered.

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @cijothomas @rajkumar-rangaraj @reyang @TimothyMothra.

[TimothyMothra]

Hi @hansmbakker, I'm sorry you're still having issues with this.
You can write your own sampler to drop the health check requests, and any downstream dependencies should also be dropped.
However, if you use a custom sampler, this will break our logic that sets the ItemCount in telemetry.

Just for my learning, why would the healthcheck api have dependencies?

[hansmbakker]

Just for my learning, why would the healthcheck api have dependencies?

In general it is possible to not only check for the health of your own app but also for its database or storage for example.

The idea is that if you have an application that cannot function without a working connection to (insert dependency X), it could be considered unhealthy itself.

Aspire.NET does this out of the box for some of its components.

[TimothyMothra]

I'm closing this because it's not a bug in the Azure Monitor exporter.
This issue is more broadly related to OpenTelemetry and completely filtering requests and their dependencies.

The suggested workaround is to use OpenTelemetry's available customizations. If that's not sufficient, we should have this conversation in the OpenTelemetry repo.