Metadata to exclude request from http.server.request.duration metric

[JamesNK]

Is there an existing issue for this?

• I have searched the existing issues

Is your feature request related to a problem? Please describe the problem.

Every request that comes into ASP.NET Core is recorded to the http.server.request.duration metric. There are some situations where its desirable to opt out of that behavior:

• Prometheus /metrics scraping endpoint
• Healthchecks /heathz endpoint

Excluding system endpoints that are regularly requested is useful because they distort results.

Describe the solution you'd like

Add metadata that can be used to skip recording the request. Then, an app can configure endpoints with the metadata to exclude them from recording the HTTP metric.

It could be an SuppressHttpMetric() extension method:

app.MapPrometheusScrapingEndpoint().SuppressHttpMetric();

Or attribute:

[HttpGet]
[SuppressHttpMetric]
public IActionResult GetHealth()
{
     // ...
}

Some libraries might automatically include the metadata, e.g. OTel's PrometheusScapingEndpoint internally adds the metadata when it creates the endpoint.

Additional context

No response

[Ne4to]

Is there any workaround?
I tried to find filter options in HostingMetrics with no success.
MapPrometheusScrapingEndpoint allows to pass custom MeterProvider but it has no methods to override and internally MeterProvider is converted to internal MeterProviderSdk in MeterProviderExtensions

[JamesNK]

Can you provide why you want to disable the metric? It is useful for us to understand why and when a new feature would be used.

[Ne4to]

My top three requests now are /metrics for Prometheus and two endpoints for Kubernetes liveness and readiness probes (based on ASP.NET HealthChecks). When I open these dashboards https://grafana.com/orgs/dotnetteam the first thing I see is not helpful. Some summary graph is 99% misleading for a service on a test environment.

UPD: #53322 created
BTW: official dashboards use different units than published by ASP.NET 8 and
http_server_request_duration_seconds_bucket should be changed to http_server_request_duration_s_bucket in the dashboard to start it working (s instead of seconds)

[WeihanLi]

I have similar issues with @Ne4to , some endpoints should be excluded like the Prometheus scraping endpoint and probe endpoints. This should be a high priority I think.

And I tried with the ShortCircuit() extension introduced in .NET 8, which would not work for this case, should the metrics be excluded when ShortCircuit() is specified?

[JamesNK]

ShortCircuit() skips middleware, but the counter is recorded at the hosting layer. It won't be skipped.

[JamesNK]

To come up with the right improvement for this situation I'm going to reach out to OpenTelemetry team for their advice.

One approach might be to skip recording system endpoints. Another approach would be to still record a value but add a tag to indicate that it's a system endpoint. Then people can choose to use the tag to filter results out.

[leandrobattochio]

Just another thumbs up here. I ended up changing asp net core default grafana dashboard to not include endpoints matching "/metrics" from the results. (not the best approach. I would like to exclude the endpoint entirely, which in my searchs lead me here)
But then I also noticed some requests from my SignalR are also being recorded and it is messing with my graphs. I think we should be able to choose to exclude some endpoint from metrics. I don't wan't to record calls to "/metrics" endpoint.

I also wouldn't want SignalR handshakes too. I mean, I expect some connections to be opened for a long time. What I'm interested in is if any of my business API is taking too long to execute.

Have any of you experienced something similiar when it comes to metrics and signalr hubs ?

[vchirikov]

Same here, it will be nice to have more control on metrics, looks like in net8.0 we can only enrich tags from middleware with IHttpMetricsTagsFeature, but not to filter them as it was in net6.0 with otel (AddAspNetCoreInstrumentation had an Filter option).

@JamesNK do you have any news about this?

[JamesNK]

Not in the last 6 days.

.NET 9 won't come out until November so there is lots of time to improve this scenario.

If you're looking for an immediate solution then you could add an "ignore" tag to requests you want to filter out, and then write queries that ignore requests with that tag.