Skip to content

fix: evict the busiest component from the transition window, not the oldest entry - #33

Merged
thedancingdeveloper merged 1 commit into
mainfrom
fix/transition-ring-share
Aug 19, 2026
Merged

fix: evict the busiest component from the transition window, not the oldest entry#33
thedancingdeveloper merged 1 commit into
mainfrom
fix/transition-ring-share

Conversation

@thedancingdeveloper

Copy link
Copy Markdown
Collaborator

Closes #29.

What was wrong

CanonicalSnapshot.transitions is a 200-entry ring (src/domain.rs) evicted
oldest-first. It is shared across every component with no per-component share,
so whichever checks change most often decide how far back the whole window
reaches. Live snapshot from a gateway at 18h uptime — full, and reaching back
2.02 hours:

port_forward.verification    88   44%
dns.upstream_udp             78   39%
external_probe               30   15%
topology.route_intent         2    1%
topology.client_lifecycle     2    1%

Three high-frequency checks held 98% of it. The two describing what the gateway
actually did to client routing — exactly what an operator goes looking for when
a container has lost egress — held two slots each and were minutes from being
pushed out, with nothing in the response to say they had been dropped.

What changed

Eviction now removes the oldest entry of whichever component holds the most of
the window, so a chatty component churns against itself and a quiet one keeps
its place. Ties go to whichever has been in the window longest, so it is
deterministic.

Raising TRANSITION_CAPACITY instead would only move the boundary — it does not
stop one component owning the window.

Scope, honestly stated

Nothing was ever lost durably. Transitions are persisted and served from
/api/v2/history/events, which still returns rows from well outside this
window. This is about what the at-a-glance view is worth: /api/v2/status is
what the dashboard renders and what an operator reads first.

This is also not fallout from #22port_forward.verification is the larger
contributor at 44%. #22 raised the DNS share to 39% by making its episodes real
rather than sub-second, which is what made an existing limit visible. #28
reduces the DNS share further, and this change stops any single component owning
the window regardless.

Operational impact

The status snapshot reaches further back for low-frequency components. No API or
schema change — same field, same shape, same capacity. Behaviour with a single
active component is unchanged (still oldest-first), which is covered by a test.

Rollback: revert the commit.

Validation

cargo fmt --all --check, cargo clippy --all-targets -- -D warnings,
cargo test --all-targets — clean, 208 tests. Four new tests: a chatty
component not evicting a quiet one, capacity still respected, a single component
still evicting oldest-first, and two components at very different rates
converging on a share rather than the faster one owning the window.

@thedancingdeveloper
thedancingdeveloper force-pushed the fix/transition-ring-share branch from 8178b8a to 3ee6d2d Compare August 19, 2026 02:56
…oldest entry

The 200-entry snapshot window is shared by every component and evicted
oldest-first, so the checks that change most often decide how far back it
reaches for everyone. On a gateway at 18h uptime it went back barely two
hours:

  port_forward.verification    88   44%
  dns.upstream_udp             78   39%
  external_probe               30   15%
  topology.route_intent         2    1%
  topology.client_lifecycle     2    1%

Three high-frequency checks held 98% of it. The two describing what the
gateway did to client routing — what an operator goes looking for when a
container has lost egress — held two slots each and were minutes from
being pushed out, with nothing in the response to say they had been.

Evict the oldest entry of whichever component holds the most of the
window instead, so a chatty component churns against itself and a quiet
one keeps its place. Raising the capacity alone would only move the
boundary; it does not stop one component owning the window.

Nothing was ever lost durably: transitions are persisted and served from
`/api/v2/history/events`. This is about what the at-a-glance view in
`/api/v2/status`, which the dashboard renders, is worth on a busy
gateway.

Refs #29
@thedancingdeveloper
thedancingdeveloper force-pushed the fix/transition-ring-share branch from 3ee6d2d to 336f505 Compare August 19, 2026 03:55
@thedancingdeveloper
thedancingdeveloper merged commit 69432b4 into main Aug 19, 2026
6 checks passed
@thedancingdeveloper
thedancingdeveloper deleted the fix/transition-ring-share branch August 19, 2026 04:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The bounded transition snapshot fills in two hours, so low-frequency components are evicted by chatty ones

1 participant