Repository navigation
Flaky under CPU load: OtlpSubprocessExporterTests "an OTLP request arrived while export was expected to be disabled" (reproduces 5/5 pinned to one core, on base and head alike) #676
Description
Activity
- added a commit that references this issue
on Sep 3, 2026 Fixed and merged as
965c7374(PR #677). Closed by the same slice as #672.The sender was identified, not inferred. The stray request was
GET /, zero-byte body,User-Agent: Go-http-client/1.1, carrying none of the debug headers a child process was given. A bare loopback listener drew the identical request, andss -tanpmapped the peer socket tomoshi-hook serve— a local daemon installed on this box on 2026-09-01 whose own help text describes probing local TCP listeners for SSH preflight. It sweeps in bursts, not on a steady poll.The load correlation is explained rather than assumed.
AssertNoRequestAsyncfails on an already-completed_anyRequest, whichPublishsets on the first request the collector ever receives. So the exposed window is the collector's whole lifetime, not the 2-3 second assertion: it is constructed before the child API process spawns and stays open through a readiness wait of up to 60 seconds. CPU load reproduces the failure because it lengthens that boot, which is also why the two-collector test reddened on a quiet box.The fix makes the fixture answer 404 to anything that is not a
POSTunder/v1/and never publish it, so a port scanner cannot make a disabled-export assertion fail. Review then found two more holes in that fix, both now closed: an export that died mid-transfer was absorbed silently, and one that merely stalled past the window was invisible; the collector counts both and refuses to conclude "no export arrived" while either is non-zero.Verification, as this issue asked: the two named tests 5/5 green under three
taskset -c 0busy loops against 5/5 red on the base, plus a 60-run pass of the whole class under the same load.- added a commit that references this issue
on Sep 12, 2026
Sighting recorded while verifying #670 / PR #675 (2026-09-03). Same scaffolding family as #672, with the load condition characterised this time.
What failed
OtlpSubprocessExporterTests.Blank_canonical_endpoint_disables_an_ambient_standard_exporterandOtlpSubprocessExporterTests.Canonical_endpoint_never_receives_the_ambient_standard_header, both with:Both tests assert that a fresh
FakeOtlpCollector(random free port,FreePort()atFakeOtlpCollector.cs:33) receives NO request inside a 2–3 s observation window (OtlpSubprocessExporterTests.cs:96,:137) after spawning the API subprocess.The load condition, and the base comparison
taskset -c 0busy loops pinned to one core for ~20 min, on a box that also had another test suite running): 5/5 runs red on the PR head (8a4f2518's product bytes) and 3/3 runs red on the unmodified base4d1dfa37, identical two tests, identical message. So it is not the diff; it is the scaffolding under load.dotnet teststill running).What this adds to #672
#672's two sightings were
HttpListenerException: Address already in useandObjectDisposedExceptionin the collector's constructor. This is a third shape in the same fixture: a request arriving at a collector that expects none, reproducible on demand under CPU pressure and independent of the diff under test. Candidate mechanisms worth checking together with #672's port-reuse hypothesis: a child API process from an earlier test in the class outliving its test under load and exporting to a port a laterFreePort()re-picked; or the "disabled" child's own startup export racing the observation window when the process is starved.Verify
Whatever the fix: the repeat runner #672 asks for should run this class N times under a pinned-CPU busy loop (the condition above reproduces it 5/5) and stay green; a single quiet green proves nothing here.