api: client-disconnect body-read failures are misclassified as auth errors (400 WARN noise)

[AdaAibaby]

Problem

Under high concurrency (e.g. 1 000 simultaneous sandbox creates after a period of idleness), the orchestration-api emits a flood of WARN log lines and 400 responses that look like authentication failures:

WARN  error in openapi3filter.SecurityRequirementsError: security requirements failed: reading failed: read tcp 192.168.0.146:3000->100.64.x.x:port: i/o timeout
WARN  error in openapi3filter.SecurityRequirementsError: security requirements failed: reading failed: read tcp 192.168.0.146:3000->100.64.x.x:port: connection reset by peer

These are not auth failures. They are network-level events where either:

• the server's ReadTimeout: 10s expired while the goroutine was queued waiting to be scheduled (under heavy load), or
• the client SDK's own timeout fired and closed the connection before the server read the body.

Root cause

kin-openapi@v0.139.0 (openapi3filter/validate_request.go:448) reads the entire request body via io.ReadAll before calling any AuthenticationFunc. When that TCP read fails it returns:

&RequestError{Reason: "reading failed", Err: <net.Error>}

validateSecurityRequirements then wraps this in SecurityRequirementsError. processCustomErrors in packages/api/internal/utils/error.go has no special handling for this case, so it falls through to the generic securityErrPrefix path. ErrorHandler then calls telemetry.ReportError (sets OTel span error status → WARN log line), adds to c.Errors, and returns HTTP 400 — all of which suggest an auth failure to every downstream observer: dashboards, alerts, on-call.

Impact

• Auth-failure dashboards and alerts fire on client-side network noise.
• Incident responders waste time investigating fake 401/400 spikes.
• Real auth failures are diluted by the noise, increasing MTTD.

Fix

In processCustomErrors, detect RequestError{Reason:"reading failed"} with an underlying net.Error and return it under a dedicated clientDisconnectPrefix. ErrorHandler handles this prefix early: records an OTel span event (informational, not error status) via telemetry.ReportEvent, and responds 499 without adding to c.Errors.

Result:

• No more fake SecurityRequirementsError WARN logs for client disconnects.
• 499 responses are queryable in metrics separately from 400/401/403.
• Auth-failure alerts stay clean.