DNS health debounce is counted in queries, not time, so resolver failures flap sub-second and are never observable

[thedancingdeveloper]

DnsHealthState debounces the DNS resolution verdict by counting consecutive
queries: failure_threshold 3 to go degraded, success_threshold 2 to come
back (src/dns.rs:204). Those are counts of queries, not durations. On a
gateway serving ~31 qps, 3 queries is about 100ms and 2 queries is about 65ms,
so the debounce does not debounce anything — it converts a burst of upstream
failures into a state change and back before anything can observe it.

Observed

dns.upstream_udp transitions from one gateway's status API, 2h39m uptime:

02:11:52 healthy -> degraded  dns.upstream_udp_failures
02:11:52 degraded -> healthy  dns.udp_healthy
02:31:08 healthy -> degraded  dns.upstream_udp_failures
02:31:08 degraded -> healthy  dns.udp_healthy
02:31:41 healthy -> degraded  dns.upstream_udp_failures
02:31:41 degraded -> healthy  dns.udp_healthy
...
04:20:29 healthy -> degraded  dns.upstream_udp_failures
04:20:29 degraded -> healthy  dns.udp_healthy

23 degraded episodes. Every one returns to healthy in the same second, and 15 of
the 23 return within the same timestamp. Meanwhile:

egressy_dns_resolution_healthy 1

Two consequences

1. Nothing can observe the check. egressy_dns_resolution_healthy is a
gauge, read at scrape time. A state that exists for under a second between
scrapes 15–60s apart is never sampled. The gauge added in #12 was specifically
meant to make resolver trouble alertable, and in this failure mode it reads 1
throughout — not because the check is wrong, but because the check is never
degraded at any moment a scrape lands on. The same applies to any consumer
polling /api/v2/status.

2. The flapping is evicting the transition history. These self-cancelling
pairs occupy 46 of the 200 transition slots — 23% of the bounded history spent
recording state changes that immediately undid themselves. The window now only
reaches back to 02:11, so transitions from the first 40 minutes of uptime have
already been pushed out by the noise.

Also: the dominant failure never reaches the health model at all

observe_udp_health is only called from the worker, after admission. A query
refused by the concurrency limit returns at src/dns.rs:406 before any worker
is spawned, so it contributes nothing to the verdict. On the gateway above that
is 14,343 refused queries — the largest failure population by two orders of
magnitude — that the resolution health check cannot see by construction.

A client whose queries are all being refused has no working DNS, and every DNS
health signal the gateway publishes says healthy.

Suggested fix

• Debounce over a time window rather than a query count, or require the failure
  condition to hold for a minimum duration before clearing. At 31 qps the
  current thresholds describe roughly a tenth of a second of trouble.
• Include admission refusals in the resolution verdict. From the client's point
  of view a refused query and a failed query are the same event, and the check
  is meant to report whether clients can resolve names.
• Consider not recording a transition pair that cancels within the debounce
  interval, so the bounded history is spent on state that meant something.