Repository navigation
[Exporter.Prometheus] Add default request timeout and improve timeout behaviour - #7757
Conversation
Add a configurable maximum request timeout for scrape request to avoid a request being able to stall all scrape requests indefinitely.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #7757 +/- ##
==========================================
+ Coverage 91.85% 91.88% +0.03%
==========================================
Files 339 339
Lines 18471 18527 +56
==========================================
+ Hits 16967 17024 +57
+ Misses 1504 1503 -1
Flags with carried forward coverage won't be shown. Click here to find out more.
|
Fix test failing on Linux when not using the exact loopback host.
Speculative fix to failing test on Linux.
There was a problem hiding this comment.
🟡 Changes recommended
The collection path may remain blocked indefinitely, and timeout enforcement lacks two required regression tests.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Adds a configurable default timeout for Prometheus HttpListener scrape responses to limit stalled requests.
Changes:
- Adds a validated 60-second timeout option.
- Applies timeout cancellation and response-abort handling.
- Adds regression tests, changelog documentation, and public API tracking.
File summaries
| File | Summary | Review findings |
|---|---|---|
test/OpenTelemetry.Exporter.Prometheus.HttpListener.Tests/PrometheusHttpListenerTests.cs |
Tests stalled-client recovery. | Moderate (1 vote): The test does not deterministically prove the first response is stalled; add controlled back-pressure and verify it remains incomplete before the deadline. |
src/OpenTelemetry.Exporter.Prometheus.HttpListener/PrometheusHttpListenerOptions.cs |
Defines and validates the timeout option. | No final findings. |
src/OpenTelemetry.Exporter.Prometheus.HttpListener/PrometheusHttpListener.cs |
Enforces scrape deadlines and aborts stalled writes. | Critical (3 votes): Collection is not cancellation-aware, so a hanging collect/export can prevent timeout handling and release state. Moderate (1 vote): Add coverage proving a longer client timeout header cannot extend the server timeout. |
src/OpenTelemetry.Exporter.Prometheus.HttpListener/CHANGELOG.md |
Documents the new behavior. | No final findings. |
src/OpenTelemetry.Exporter.Prometheus.HttpListener/.publicApi/PublicAPI.Unshipped.txt |
Tracks the new public property. | No final findings. |
Review details
Suppressed comments (2)
src/OpenTelemetry.Exporter.Prometheus.HttpListener/PrometheusHttpListener.cs:273
- The new contract that
X-Prometheus-Scrape-Timeout-Secondsmay not extend the server limit is not covered: the existing timeout tests use the default 60-second server limit, and this new test sends no header. Please add a regression case with a shortScrapeResponseTimeoutMillisecondsand a longer header, then block the scrape long enough to assert that the server returns 408 at its own deadline.
if (TryGetScrapeTimeout(context.Request.Headers, out var clientTimeout) &&
clientTimeout.GetValueOrDefault() < scrapeTimeout)
{
scrapeTimeout = clientTimeout.GetValueOrDefault();
test/OpenTelemetry.Exporter.Prometheus.HttpListener.Tests/PrometheusHttpListenerTests.cs:624
- This regression test does not prove that the first response is actually stalled. It only sends roughly 12 MB and later checks that a second scrape succeeds; on a platform where that payload fits in the effective send/receive buffers, the pre-change implementation would also release the reader slot and this test would pass without exercising the timeout. Make back-pressure deterministic (for example by constraining the stalled socket's receive buffer) and assert the first request remains incomplete before the deadline.
// Emit a payload far larger than any socket / HTTP.sys send buffer (including autotuned
// ones) so the response write cannot complete until the client actually drains the body.
// Stay under the SDK's default 1000-metric-stream cap and use a large label value on each
// stream instead; this produces roughly 12 MB of exposition text.
- Files reviewed: 5/5 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Pull request dashboard statusMerged · refreshed 2026-09-18 11:01 UTC Status above doesn't look right?
|
Allow collect to timeout to avoid another potential source of deadlock.
There was a problem hiding this comment.
🟡 Changes recommended
Background collection introduces disposal races and exception paths that can permanently leak reader slots.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (1)
src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/Shared/PrometheusCollectionManager.cs:298
- Faulting this task leaks the reader slot acquired by
TryEnterCollect. Both production callers awaitEnterCollectbefore entering thetry/finallythat callsExitCollect(PrometheusHttpListener.cs:285-382andPrometheusExporterMiddleware.cs:87-146), so an exception skips the decrement and leaves subsequent collections waiting for readers indefinitely. Publish a failed collection result instead so callers return 500 and still reachExitCollect.
catch (Exception ex)
{
collectionContext.SetException(ex);
- Files reviewed: 8/8 changed files
- Comments generated: 4
- Review effort level: Balanced
- Avoid leaked task. - Avoid flaky test. - Improve test coverage. - Update CHANGELOGs.
There was a problem hiding this comment.
🟡 Changes recommended
Expected cancellations emit erroneous failure telemetry, and public documentation misstates mid-response timeout behavior.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/Shared/PrometheusCollectionManager.cs:272
Task.Runqueues work to the thread pool; it does not create a dedicated thread. This distinction matters here because a hung observable callback still consumes a pool worker, so please describe the actual isolation rather than implying dedicated-thread semantics.
- Files reviewed: 9/9 changed files
- Comments generated: 3
- Review effort level: Balanced
Address the latest review feedback.
Changes
Add a configurable maximum request timeout for scrape request to avoid a single scrape request being able to stall all scrape requests indefinitely.
Merge requirement checklist
CHANGELOG.mdfiles updated for non-trivial changes