Skip to content

[Exporter.Prometheus] Add default request timeout and improve timeout behaviour - #7757

Merged
martincostello merged 9 commits into
open-telemetry:mainfrom
martincostello:add-prometheus-httplistener-timeout
Sep 18, 2026
Merged

martincostello merged 9 commits into
open-telemetry:mainfrom
martincostello:add-prometheus-httplistener-timeout

Conversation

@martincostello

Copy link
Copy Markdown
Member

Changes

Add a configurable maximum request timeout for scrape request to avoid a single scrape request being able to stall all scrape requests indefinitely.

Merge requirement checklist

  • CONTRIBUTING guidelines followed (license requirements, nullable enabled, static analysis, etc.)
  • Unit tests added/updated
  • Appropriate CHANGELOG.md files updated for non-trivial changes
  • Changes in public API reviewed (if applicable)

Add a configurable maximum request timeout for scrape request to avoid a request being able to stall all scrape requests indefinitely.
@martincostello martincostello added this to the v1.19.0 milestone Sep 15, 2026
@codecov

codecov Bot commented Sep 15, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.40659% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.88%. Comparing base (f86d6b1) to head (782ef8f).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...ner/Internal/Shared/PrometheusCollectionManager.cs 93.61% 3 Missing ⚠️
....Prometheus.HttpListener/PrometheusHttpListener.cs 91.89% 3 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #7757      +/-   ##
==========================================
+ Coverage   91.85%   91.88%   +0.03%     
==========================================
  Files         339      339              
  Lines       18471    18527      +56     
==========================================
+ Hits        16967    17024      +57     
+ Misses       1504     1503       -1     
Flag Coverage Δ
unittests-Project-Experimental 91.88% <93.40%> (-0.11%) ⬇️
unittests-Project-Stable 91.98% <93.40%> (+<0.01%) ⬆️
unittests-UnstableCoreLibraries-Experimental 50.79% <93.40%> (+0.31%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...metheus.AspNetCore/PrometheusExporterMiddleware.cs 99.24% <100.00%> (+<0.01%) ⬆️
...heus.HttpListener/PrometheusHttpListenerOptions.cs 100.00% <100.00%> (ø)
...ner/Internal/Shared/PrometheusCollectionManager.cs 90.36% <93.61%> (-0.01%) ⬇️
....Prometheus.HttpListener/PrometheusHttpListener.cs 87.50% <91.89%> (+0.27%) ⬆️

... and 5 files with indirect coverage changes

Comment thread src/OpenTelemetry.Exporter.Prometheus.HttpListener/CHANGELOG.md Outdated
@github-actions github-actions Bot added the pkg:OpenTelemetry.Exporter.Prometheus.HttpListener Issues related to OpenTelemetry.Exporter.Prometheus.HttpListener NuGet package label Sep 15, 2026
Fix test failing on Linux when not using the exact loopback host.
Speculative fix to failing test on Linux.
@martincostello
martincostello marked this pull request as ready for review September 16, 2026 15:14
Copilot AI lite review requested due to automatic review settings September 16, 2026 15:14
@martincostello
martincostello requested a review from a team as a code owner September 16, 2026 15:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The collection path may remain blocked indefinitely, and timeout enforcement lacks two required regression tests.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds a configurable default timeout for Prometheus HttpListener scrape responses to limit stalled requests.

Changes:

  • Adds a validated 60-second timeout option.
  • Applies timeout cancellation and response-abort handling.
  • Adds regression tests, changelog documentation, and public API tracking.
File summaries
File Summary Review findings
test/OpenTelemetry.Exporter.Prometheus.HttpListener.Tests/PrometheusHttpListenerTests.cs Tests stalled-client recovery. Moderate (1 vote): The test does not deterministically prove the first response is stalled; add controlled back-pressure and verify it remains incomplete before the deadline.
src/OpenTelemetry.Exporter.Prometheus.HttpListener/PrometheusHttpListenerOptions.cs Defines and validates the timeout option. No final findings.
src/OpenTelemetry.Exporter.Prometheus.HttpListener/PrometheusHttpListener.cs Enforces scrape deadlines and aborts stalled writes. Critical (3 votes): Collection is not cancellation-aware, so a hanging collect/export can prevent timeout handling and release state. Moderate (1 vote): Add coverage proving a longer client timeout header cannot extend the server timeout.
src/OpenTelemetry.Exporter.Prometheus.HttpListener/CHANGELOG.md Documents the new behavior. No final findings.
src/OpenTelemetry.Exporter.Prometheus.HttpListener/.publicApi/PublicAPI.Unshipped.txt Tracks the new public property. No final findings.
Review details

Suppressed comments (2)

src/OpenTelemetry.Exporter.Prometheus.HttpListener/PrometheusHttpListener.cs:273

  • The new contract that X-Prometheus-Scrape-Timeout-Seconds may not extend the server limit is not covered: the existing timeout tests use the default 60-second server limit, and this new test sends no header. Please add a regression case with a short ScrapeResponseTimeoutMilliseconds and a longer header, then block the scrape long enough to assert that the server returns 408 at its own deadline.
            if (TryGetScrapeTimeout(context.Request.Headers, out var clientTimeout) &&
                clientTimeout.GetValueOrDefault() < scrapeTimeout)
            {
                scrapeTimeout = clientTimeout.GetValueOrDefault();

test/OpenTelemetry.Exporter.Prometheus.HttpListener.Tests/PrometheusHttpListenerTests.cs:624

  • This regression test does not prove that the first response is actually stalled. It only sends roughly 12 MB and later checks that a second scrape succeeds; on a platform where that payload fits in the effective send/receive buffers, the pre-change implementation would also release the reader slot and this test would pass without exercising the timeout. Make back-pressure deterministic (for example by constraining the stalled socket's receive buffer) and assert the first request remains incomplete before the deadline.
        // Emit a payload far larger than any socket / HTTP.sys send buffer (including autotuned
        // ones) so the response write cannot complete until the client actually drains the body.
        // Stay under the SDK's default 1000-metric-stream cap and use a large label value on each
        // stream instead; this produces roughly 12 MB of exposition text.
  • Files reviewed: 5/5 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@martincostello
martincostello marked this pull request as draft September 16, 2026 15:50
@opentelemetry-pr-dashboard

opentelemetry-pr-dashboard Bot commented Sep 17, 2026 •

Copy link
Copy Markdown

Pull request dashboard status

Merged · refreshed 2026-09-18 11:01 UTC

Status above doesn't look right?
  • Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

Allow collect to timeout to avoid another potential source of deadlock.
@github-actions github-actions Bot added the pkg:OpenTelemetry.Exporter.Prometheus.AspNetCore Issues related to OpenTelemetry.Exporter.Prometheus.AspNetCore NuGet package label Sep 17, 2026
@martincostello
martincostello requested a balanced review from Copilot September 17, 2026 10:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Background collection introduces disposal races and exception paths that can permanently leak reader slots.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (1)

src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/Shared/PrometheusCollectionManager.cs:298

  • Faulting this task leaks the reader slot acquired by TryEnterCollect. Both production callers await EnterCollect before entering the try/finally that calls ExitCollect (PrometheusHttpListener.cs:285-382 and PrometheusExporterMiddleware.cs:87-146), so an exception skips the decrement and leaves subsequent collections waiting for readers indefinitely. Publish a failed collection result instead so callers return 500 and still reach ExitCollect.
        catch (Exception ex)
        {
            collectionContext.SetException(ex);
  • Files reviewed: 8/8 changed files
  • Comments generated: 4
  • Review effort level: Balanced

Comment thread src/OpenTelemetry.Exporter.Prometheus.HttpListener/CHANGELOG.md Outdated
- Avoid leaked task.
- Avoid flaky test.
- Improve test coverage.
- Update CHANGELOGs.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Expected cancellations emit erroneous failure telemetry, and public documentation misstates mid-response timeout behavior.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/Shared/PrometheusCollectionManager.cs:272

  • Task.Run queues work to the thread pool; it does not create a dedicated thread. This distinction matters here because a hung observable callback still consumes a pool worker, so please describe the actual isolation rather than implying dedicated-thread semantics.
  • Files reviewed: 9/9 changed files
  • Comments generated: 3
  • Review effort level: Balanced

Comment thread src/OpenTelemetry.Exporter.Prometheus.HttpListener/CHANGELOG.md Outdated
Address the latest review feedback.
@martincostello martincostello changed the title [Prometheus.HttpListener] Add default request timeout [Exporter.Prometheus] Add default request timeout and improve timeout behaviour Sep 17, 2026
@martincostello
martincostello requested a balanced review from Copilot September 17, 2026 12:39
This was referenced Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pkg:OpenTelemetry.Exporter.Prometheus.AspNetCore Issues related to OpenTelemetry.Exporter.Prometheus.AspNetCore NuGet package pkg:OpenTelemetry.Exporter.Prometheus.HttpListener Issues related to OpenTelemetry.Exporter.Prometheus.HttpListener NuGet package

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants