Flaky: FakeOtlpCollector HttpListener disposed / address in use in OtlpSubprocessExporterTests and FakeOtlpCollectorTests

[mforce]

Seen twice on 2026-09-02 while shipping #562 (PR #671), both in the FakeOtlpCollector test scaffolding, neither reproducible on a clean re-run, and neither touching files the PR changed:

1. Locally (implementer's first foreground full-suite run, Release): FakeOtlpCollectorTests.Predicate_wait_throws_a_terminal_error_completed_at_the_timeout_catch_boundary — System.Net.HttpListenerException: Address already in use. Immediate re-run green (1654/1654).
2. CI (run 33704636108, job Build and test, head 667a4242): OtlpSubprocessExporterTests.Production_plaintext_loopback_endpoint_with_acknowledgement_boots — System.ObjectDisposedException: Cannot access a disposed object. Object name: 'System.Net.HttpListener' at FakeOtlpCollector..ctor() (tests/Cluckwork.Api.IntegrationTests/Infrastructure/FakeOtlpCollector.cs:35, HttpListener.get_Prefixes). Re-run: see the PR's checks.

For context, main's own CI run on the same base commit (80b53f4b, run 33695280424) failed on a third timing-bound test, DurableJobWorkerLeaderGateTests.Follower_NeverPolls_ButStampsHeartbeat (Assert.NotNull() Failure on the heartbeat), so the runner class is already marginal for timing-sensitive integration tests.

What to look at

• FakeOtlpCollector.cs:35 reads Prefixes on a listener that has been disposed — most likely the port-probe/retry loop disposing a listener it then reuses, or a static/shared listener across parallel test classes.
• Port selection: two tests binding loopback ports concurrently (Address already in use) suggests the collector picks a port without holding it, or two fixtures share a port range.

Verify

A fix must come with a loop that runs the two named tests N times in parallel with the rest of the integration collection and stays green; a single green run proves nothing about a flake.

[mforce]

Fixed and merged as 965c7374 (squash of PR #677, head cdf43f49).

Both root causes were in the fixture, not in what it was watching.

1. The constructor reused a single HttpListener across bind attempts, and HttpListener.Start closes the instance on any failure — verified in the .NET 10 source and by a local probe. So the retry's first statement, Prefixes.Clear(), threw ObjectDisposedException: sighting 2's exact stack. The retry loop had therefore never once retried. It now builds a fresh listener per attempt, disposes the failed one, and allows ten attempts. Probe-then-bind stays because HttpListener cannot bind port 0 (Invalid port in prefix, observed).
2. AssertNoRequestAsync asserted that no TCP client had spoken to the collector, not that no OTLP export had arrived: ServeAsync published every request it accepted. The stray request in Flaky under CPU load: OtlpSubprocessExporterTests "an OTLP request arrived while export was expected to be disabled" (reproduces 5/5 pinned to one core, on base and head alike) #676 was GET /, empty body, User-Agent: Go-http-client/1.1 — a local port scanner (moshi-hook serve, installed on this box 2026-09-01), identified by mapping the peer socket with ss -tanp, not by inference. That is why continuous integration never saw that shape. The collector now answers 404 and drops anything that is not a POST under /v1/.

Sighting 1 (Address already in use escaping to the test) is recorded as not reproducible, and closed-form impossible from that constructor: attempt 1's exception was caught and attempt 2 could never reach Start. The mutation that clamps the retry budget to 1 does produce exactly that text, so the fix removes the path regardless.

Review found three further defects in the fix itself, all now fixed: the absorbing catch also covered the accept, so a listener-level failure would have spun the loop forever instead of faulting; an export that died mid-transfer was absorbed silently, so "no export arrived" could pass while one had; and an export that merely stalled past the observation window was invisible to every check. The collector now counts aborted and in-flight exports and refuses to conclude while either is non-zero. Absorbing is right; forgetting is not.

The verification #672 asked for: the two named tests ran 5/5 green under three CPU-pinned busy loops, the condition that reproduced the failure 5/5 on the unmodified base; a 60-run repeat pass of the whole collector class under the same load was clean; and eight mutation rows were run on the shipped bytes — six red on their named symptom, one control green, and one recorded as surviving: no test in this harness can synthesise a listener-level accept failure while the listener still reports listening, so that guard is an open coverage item rather than a claim.

Left alone deliberately: the three sibling probe-then-bind sites (OtlpSubprocessExporterTests.GetFreeTcpPort:334, MultiInstanceIdempotencyTests.cs:100, MultiInstanceRateLimitTests.cs:107). Each hands its port to a child process, so a lost race there fails the child's own bind and surfaces as a readiness timeout — a different shape with no sighting on record. Recorded, not filed.

DurableJobWorkerLeaderGateTests (named here for runner-class context) is untouched: it is a no-database timing test sharing no code with this fixture.