[Prometheus.HttpListener] Fix shutdown hang by martincostello · Pull Request #7587 · open-telemetry/opentelemetry-dotnet

martincostello · 2026-08-04T13:58:26Z

Changes

Avoid disposal race in PrometheusHttpListener that can cause shutdown to hang indefinitely and flaky tests (1, 2, 3).

Merge requirement checklist

CONTRIBUTING guidelines followed (license requirements, nullable enabled, static analysis, etc.)
Unit tests added/updated
Appropriate CHANGELOG.md files updated for non-trivial changes
~~Changes in public API reviewed (if applicable)~~

Avoid disposal race in `PrometheusHttpListener` that can cause shutdown to hang indefinitely.

opentelemetry-pr-dashboard · 2026-08-04T14:02:24Z

Pull request dashboard status

Merged · refreshed 2026-08-04 21:27 UTC

Status above doesn't look right?

Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

codecov · 2026-08-04T14:04:20Z

Codecov Report

❌ Patch coverage is 69.23077% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.49%. Comparing base (2ae52ae) to head (5d86411).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
....Prometheus.HttpListener/PrometheusHttpListener.cs	69.23%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7587      +/-   ##
==========================================
- Coverage   90.51%   90.49%   -0.02%     
==========================================
  Files         290      290              
  Lines       16029    16039      +10     
==========================================
+ Hits        14509    14515       +6     
- Misses       1520     1524       +4

Flag	Coverage Δ
unittests-Project-Experimental	`90.49% <69.23%> (-0.07%)`	⬇️
unittests-Project-Stable	`90.53% <69.23%> (-0.08%)`	⬇️
unittests-UnstableCoreLibraries-Experimental	`51.35% <69.23%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
....Prometheus.HttpListener/PrometheusHttpListener.cs	`88.30% <69.23%> (-1.76%)`	⬇️

... and 2 files with indirect coverage changes

martincostello · 2026-08-04T15:56:23Z

6 runs with no Prometheus related failures.

There was one failure here on attempt 5, but that has no specific cause logged but appears to be within OpenTelemetry.Tests.

Copilot

Pull request overview

This PR addresses a shutdown hang in the Prometheus HttpListener exporter by bounding shutdown waits and ensuring in-flight scrapes fail fast (HTTP 503) once disposal begins, with accompanying test hardening to avoid race-driven flakiness.

Changes:

Add bounded timeouts to listener shutdown/drain logic and log when the processing loop fails to stop promptly.
Ensure scrapes that are still collecting when disposal starts return HTTP 503 instead of completing with a successful response.
Harden integration/unit tests by adding explicit scrape/promtool timeouts and avoiding thread-pool scheduling races during disposal.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
test/OpenTelemetry.Exporter.Prometheus.HttpListener.Tests/PromToolFixture.cs	Adds scrape and overall promtool execution timeouts to prevent hangs in container-based validation.
test/OpenTelemetry.Exporter.Prometheus.HttpListener.Tests/PrometheusHttpListenerTests.cs	Makes the disposal-vs-scrape race test deterministic (thread-pool availability + dedicated disposal thread).
src/OpenTelemetry.Exporter.Prometheus.HttpListener/PrometheusHttpListener.cs	Implements bounded shutdown waits and returns 503 when disposal begins mid-collection.
src/OpenTelemetry.Exporter.Prometheus.HttpListener/CHANGELOG.md	Adds Unreleased entries describing the behavioral changes (needs PR link fixups).

Suppressed comments (1)

src/OpenTelemetry.Exporter.Prometheus.HttpListener/CHANGELOG.md:24

This CHANGELOG entry still uses a placeholder PR reference (#TODO). Please update it to the real PR link before merging.

  ([#TODO](https://github.com/open-telemetry/opentelemetry-dotnet/pull/TODO))

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Add PR number.

….0 (#1316) Updated [OpenTelemetry.Exporter.OpenTelemetryProtocol](https://github.com/open-telemetry/opentelemetry-dotnet) from 1.17.0 to 1.18.0. <details> <summary>Release notes</summary> _Sourced from [OpenTelemetry.Exporter.OpenTelemetryProtocol's releases](https://github.com/open-telemetry/opentelemetry-dotnet/releases)._ ## 1.18.0 For highlights and announcements pertaining to this release see: [Release Notes > 1.18.0](https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/RELEASENOTES.md#1180). The following changes are from the previous release [1.17.0](https://github.com/open-telemetry/opentelemetry-dotnet/releases/tag/core-1.17.0). * NuGet: [OpenTelemetry v1.18.0](https://www.nuget.org/packages/OpenTelemetry/1.18.0) * Fixed self-diagnostics log lines being silently dropped when an event message or parameter contained enough 3-byte UTF-8 characters to overflow the internal buffer estimate. Such content is now truncated. ([#7543](open-telemetry/opentelemetry-dotnet#7543)) * Fixed activity creation throwing when multiple tracer providers return a sampler attribute with the same key. ([#7558](open-telemetry/opentelemetry-dotnet#7558)) * Added the `otel.sdk.processor.log.processed` SDK self-observability metric. ([#7486](open-telemetry/opentelemetry-dotnet#7486)) * Added the `otel.sdk.processor.span.processed` SDK self-observability metric. ([#7598](open-telemetry/opentelemetry-dotnet#7598)) * `BatchActivityExportProcessor` and `SimpleActivityExportProcessor` no longer forward spans to the exporter once `Shutdown` has been called, and `BatchActivityExportProcessor.Shutdown` now waits for in-flight `OnEnd` calls to finish enqueueing before flushing. ([#7598](open-telemetry/opentelemetry-dotnet#7598)) * Fix logger, meter and tracer providers leaking background threads if an exception is thrown by their constructor after resource creation. ([#7615](open-telemetry/opentelemetry-dotnet#7615)) * `CircularBufferBuckets.Copy` optimized to use bulk array copies. ([#7670](open-telemetry/opentelemetry-dotnet#7670)) * Restored configured `MaxScale` after delta exponential histogram collection. ([#7671](open-telemetry/opentelemetry-dotnet#7671)) See [CHANGELOG](https://github.com/open-telemetry/opentelemetry-dotnet/blob/core-1.18.0/src/OpenTelemetry/CHANGELOG.md) for details. * NuGet: [OpenTelemetry.Api v1.18.0](https://www.nuget.org/packages/OpenTelemetry.Api/1.18.0) * Avoid formatting exceptions and creating exception attributes when `RecordException` is called on a span that is not recorded. ([#7669](open-telemetry/opentelemetry-dotnet#7669)) See [CHANGELOG](https://github.com/open-telemetry/opentelemetry-dotnet/blob/core-1.18.0/src/OpenTelemetry.Api/CHANGELOG.md) for details. * NuGet: [OpenTelemetry.Api.ProviderBuilderExtensions v1.18.0](https://www.nuget.org/packages/OpenTelemetry.Api.ProviderBuilderExtensions/1.18.0) No notable changes. See [CHANGELOG](https://github.com/open-telemetry/opentelemetry-dotnet/blob/core-1.18.0/src/OpenTelemetry.Api.ProviderBuilderExtensions/CHANGELOG.md) for details. * NuGet: [OpenTelemetry.Exporter.Console v1.18.0](https://www.nuget.org/packages/OpenTelemetry.Exporter.Console/1.18.0) * Added support for serializing attribute values that are key/value lists (`IEnumerable<KeyValuePair<string, object?>>`). These attributes will be serialized as JSON objects. ([#7015](open-telemetry/opentelemetry-dotnet#7015)) ... (truncated) ## 1.18.0-rc.1 The following changes are from the previous release [1.17.0](https://github.com/open-telemetry/opentelemetry-dotnet/releases/tag/core-1.17.0). * NuGet: [OpenTelemetry v1.18.0-rc.1](https://www.nuget.org/packages/OpenTelemetry/1.18.0-rc.1) * Fixed self-diagnostics log lines being silently dropped when an event message or parameter contained enough 3-byte UTF-8 characters to overflow the internal buffer estimate. Such content is now truncated. ([#7543](open-telemetry/opentelemetry-dotnet#7543)) * Fixed activity creation throwing when multiple tracer providers return a sampler attribute with the same key. ([#7558](open-telemetry/opentelemetry-dotnet#7558)) * Added the `otel.sdk.processor.log.processed` SDK self-observability metric. ([#7486](open-telemetry/opentelemetry-dotnet#7486)) * Added the `otel.sdk.processor.span.processed` SDK self-observability metric. ([#7598](open-telemetry/opentelemetry-dotnet#7598)) * `BatchActivityExportProcessor` and `SimpleActivityExportProcessor` no longer forward spans to the exporter once `Shutdown` has been called, and `BatchActivityExportProcessor.Shutdown` now waits for in-flight `OnEnd` calls to finish enqueueing before flushing. ([#7598](open-telemetry/opentelemetry-dotnet#7598)) * Fix logger, meter and tracer providers leaking background threads if an exception is thrown by their constructor after resource creation. ([#7615](open-telemetry/opentelemetry-dotnet#7615)) * `CircularBufferBuckets.Copy` optimized to use bulk array copies. ([#7670](open-telemetry/opentelemetry-dotnet#7670)) * Restored configured `MaxScale` after delta exponential histogram collection. ([#7671](open-telemetry/opentelemetry-dotnet#7671)) See [CHANGELOG](https://github.com/open-telemetry/opentelemetry-dotnet/blob/core-1.18.0-rc.1/src/OpenTelemetry/CHANGELOG.md) for details. * NuGet: [OpenTelemetry.Api v1.18.0-rc.1](https://www.nuget.org/packages/OpenTelemetry.Api/1.18.0-rc.1) * Avoid formatting exceptions and creating exception attributes when `RecordException` is called on a span that is not recorded. ([#7669](open-telemetry/opentelemetry-dotnet#7669)) See [CHANGELOG](https://github.com/open-telemetry/opentelemetry-dotnet/blob/core-1.18.0-rc.1/src/OpenTelemetry.Api/CHANGELOG.md) for details. * NuGet: [OpenTelemetry.Api.ProviderBuilderExtensions v1.18.0-rc.1](https://www.nuget.org/packages/OpenTelemetry.Api.ProviderBuilderExtensions/1.18.0-rc.1) No notable changes. See [CHANGELOG](https://github.com/open-telemetry/opentelemetry-dotnet/blob/core-1.18.0-rc.1/src/OpenTelemetry.Api.ProviderBuilderExtensions/CHANGELOG.md) for details. * NuGet: [OpenTelemetry.Exporter.Console v1.18.0-rc.1](https://www.nuget.org/packages/OpenTelemetry.Exporter.Console/1.18.0-rc.1) * Added support for serializing attribute values that are key/value lists (`IEnumerable<KeyValuePair<string, object?>>`). These attributes will be serialized as JSON objects. ([#7015](open-telemetry/opentelemetry-dotnet#7015)) See [CHANGELOG](https://github.com/open-telemetry/opentelemetry-dotnet/blob/core-1.18.0-rc.1/src/OpenTelemetry.Exporter.Console/CHANGELOG.md) for details. ... (truncated) ## 1.18.0-beta.1 The following changes are from the previous release [1.17.0-beta.1](https://github.com/open-telemetry/opentelemetry-dotnet/releases/tag/coreunstable-1.17.0-beta.1). * NuGet: [OpenTelemetry.Exporter.Prometheus.AspNetCore v1.18.0-beta.1](https://www.nuget.org/packages/OpenTelemetry.Exporter.Prometheus.AspNetCore/1.18.0-beta.1) * Fix concurrent scrapes returning an empty response under contention. Now the exporter will return an HTTP 500 error instead. ([#7571](open-telemetry/opentelemetry-dotnet#7571)) * Waiting for concurrent scrapes to finish before collecting no longer blocks, which could stall concurrent scrapes being waited on. ([#7571](open-telemetry/opentelemetry-dotnet#7571)) * Fixed the interaction between `PrometheusAspNetCoreOptions.TranslationStrategy` and content negotiation. The configured strategy is now applied before content negotiation, instead of the negotiated escaping scheme replacing the strategy's, and the `Content-Type` header now reports the escaping scheme that was applied rather than the one that was negotiated. ([#7610](open-telemetry/opentelemetry-dotnet#7610)) * Fixed metric values and histogram bucket bounds being written with 17 significant digits instead of their shortest round-trippable representation. ([#7589](open-telemetry/opentelemetry-dotnet#7589)) * Fixed the canonical representation used for histogram `le` and summary `quantile` label values falling back to 17 significant digits incorrectly. ([#7589](open-telemetry/opentelemetry-dotnet#7589)) * Fixed a race where a slow scrape could return an HTTP 200 instead of 408. ([#7615](open-telemetry/opentelemetry-dotnet#7615)) * Updated OpenTelemetry core component version(s) to `1.18.0`. ([#7674](open-telemetry/opentelemetry-dotnet#7674)) See [CHANGELOG](https://github.com/open-telemetry/opentelemetry-dotnet/blob/coreunstable-1.18.0-beta.1/src/OpenTelemetry.Exporter.Prometheus.AspNetCore/CHANGELOG.md) for details. * NuGet: [OpenTelemetry.Exporter.Prometheus.HttpListener v1.18.0-beta.1](https://www.nuget.org/packages/OpenTelemetry.Exporter.Prometheus.HttpListener/1.18.0-beta.1) * Fix concurrent scrapes returning an empty response under contention. Now the exporter will return an HTTP 500 error instead. ([#7571](open-telemetry/opentelemetry-dotnet#7571)) * Waiting for concurrent scrapes to finish before collecting no longer blocks, which could stall concurrent scrapes being waited on. ([#7571](open-telemetry/opentelemetry-dotnet#7571)) * A scrape which is still collecting when the listener is disposed now returns an HTTP 503 response. ([#7587](open-telemetry/opentelemetry-dotnet#7587)) * Shutting down the listener no longer waits indefinitely for its request processing loop to stop, and no longer throws if the loop faulted. ([#7587](open-telemetry/opentelemetry-dotnet#7587)) * Fixed the interaction between `PrometheusHttpListenerOptions.TranslationStrategy` and content negotiation. The configured strategy is now applied before content negotiation, instead of the negotiated escaping scheme replacing the strategy's, and the `Content-Type` header now reports the escaping scheme that was applied rather than the one that was negotiated. ([#7610](open-telemetry/opentelemetry-dotnet#7610)) * Fixed metric values and histogram bucket bounds being written with 17 significant digits instead of their shortest round-trippable representation. ([#7589](open-telemetry/opentelemetry-dotnet#7589)) * Fixed the canonical representation used for histogram `le` and summary `quantile` label values falling back to 17 significant digits incorrectly. ([#7589](open-telemetry/opentelemetry-dotnet#7589)) ... (truncated) Commits viewable in [compare view](open-telemetry/opentelemetry-dotnet@core-1.17.0...core-1.18.0). </details> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=OpenTelemetry.Exporter.OpenTelemetryProtocol&package-manager=nuget&previous-version=1.17.0&new-version=1.18.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

[Prometheus.HttpListener] Fix shutdown hang

ac3abdb

Avoid disposal race in `PrometheusHttpListener` that can cause shutdown to hang indefinitely.

github-actions Bot added the pkg:OpenTelemetry.Exporter.Prometheus.HttpListener Issues related to OpenTelemetry.Exporter.Prometheus.HttpListener NuGet package label Aug 4, 2026

opentelemetry-pr-dashboard Bot mentioned this pull request Aug 4, 2026

Pull Request Dashboard #7539

Open

martincostello marked this pull request as ready for review August 4, 2026 15:56

martincostello requested a review from a team as a code owner August 4, 2026 15:56

Copilot AI lite review requested due to automatic review settings August 4, 2026 15:56

Copilot started reviewing on behalf of martincostello August 4, 2026 15:57 View session

Copilot AI reviewed Aug 4, 2026

View reviewed changes

Comment thread src/OpenTelemetry.Exporter.Prometheus.HttpListener/CHANGELOG.md Outdated

martincostello commented Aug 4, 2026

View reviewed changes

Comment thread src/OpenTelemetry.Exporter.Prometheus.HttpListener/CHANGELOG.md Outdated

[Prometheus.HttpListener] Update CHANGELOG

5d86411

Add PR number.

martincostello enabled auto-merge August 4, 2026 16:03

rajkumar-rangaraj approved these changes Aug 4, 2026

View reviewed changes

martincostello added this pull request to the merge queue Aug 4, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Aug 4, 2026

martincostello added this pull request to the merge queue Aug 4, 2026

Merged via the queue into open-telemetry:main with commit 3837c59 Aug 4, 2026
80 checks passed

martincostello deleted the fix-prometheus-flakiness branch August 4, 2026 21:26

martincostello added this to the v1.18.0 milestone Aug 13, 2026

Navigation Menu

Uh oh!

[Prometheus.HttpListener] Fix shutdown hang - #7587

[Prometheus.HttpListener] Fix shutdown hang#7587
martincostello merged 2 commits into
open-telemetry:mainfrom
martincostello:fix-prometheus-flakiness

martincostello commented Aug 4, 2026

Uh oh!

opentelemetry-pr-dashboard Bot commented Aug 4, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Aug 4, 2026 •

edited

Loading

Uh oh!

martincostello commented Aug 4, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

martincostello commented Aug 4, 2026

Changes

Merge requirement checklist

Uh oh!

opentelemetry-pr-dashboard Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull request dashboard status

Uh oh!

codecov Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

martincostello commented Aug 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

opentelemetry-pr-dashboard Bot commented Aug 4, 2026 •

edited

Loading

codecov Bot commented Aug 4, 2026 •

edited

Loading