Performance: web client useSession refetch storm dominates hub access logs (~95% of syslog volume on busy installs)

[heavygee]

Performance: web client useSession refetch storm dominates hub access logs (~95% of syslog volume on busy installs)

Summary

On a HAPI install with a non-trivial fleet (130+ sessions), the hub access log is dominated by GET /api/sessions/<uuid> requests originating from the React web client. Sustained rate ~7-15 req/sec at idle, with bursts to several hundred req/sec when an SSE event invalidates per-session detail caches across all rendered rows. On the reporter's box this works out to ~9.3 GB/day of syslog written by hapi-hub.service, ~95% of which is the GET /api/sessions/<uuid> access-log line emitted by Hono's logger() middleware (hub/src/web/server.ts).

This is a performance issue (CPU, network, log volume, SSD wear), not a correctness issue. Nothing user-visible breaks, but the system scales linearly with fleet size in places it should not.

Operator expectation

"I should be able to have as many sessions as I wish and it still not produce problematic log growth."

Fleet size should not determine log volume. Per-session detail data already arrives via SSE; the REST refetch is redundant work plus access-log noise.

Receipts (reporter's box)

• Sampled 5,000 syslog lines covering 6m 19s.
• 131 distinct session UUIDs, each hit ~22-30 times in that window. Roughly one GET per session every 13-20 seconds.
• Three GETs land within 50-100 microseconds of each other for different UUIDs: classic single-client fan-out, not many clients each fetching one.
• Connections to :3006 from cursor-agent subprocesses use Socket.IO (long-lived) - exonerated. The REST hits are short-lived TCP, indicating a browser-side client.
• glance-hapi-hub-summary.py reads ~/.hapi/hapi.db directly - exonerated.

Root cause

Three reinforcing problems in web/src/hooks/:

1. useSession hook has no staleTime

web/src/hooks/queries/useSession.ts calls useQuery with no staleTime and no refetchInterval. TanStack Query defaults: staleTime: 0, refetchOnWindowFocus: true, refetchOnMount: 'always'. Any window focus, any mount, any invalidation refetches immediately.

2. SSE handler falls back to per-session invalidation when patch path misses

web/src/hooks/useSSE.ts has a sensible patch-the-cache primary path (patchSessionDetail + patchSessionSummary) but two unconditional fallback branches:

// Line 509-512
} else {
    queueSessionDetailInvalidation(event.sessionId)
    queueSessionListInvalidation()
}

This else fires whenever getSessionPatch(event.data) returns falsy. Every such SSE event invalidates the detail cache, which causes any active useSession(id) observer for that ID to refetch from REST. With a session-list page rendered, every row likely has an active useSession observer, so every SSE session-touched event triggers a per-session GET.

Line 503-507 has a similar trap: when patchSessionDetail returns false (because the detail cache entry doesn't exist - e.g. session never opened), the code invalidates rather than accepting the no-op.

3. Session list page likely uses N+1 fetching (needs verification)

The pattern of "131 distinct UUIDs hit in parallel from one client" is the shape of a list page where each row independently calls useSession(id) instead of relying on the bulk /api/sessions list endpoint. Each row becomes a query observer, so each SSE invalidation cascades to a refetch storm.

Worth verifying which page mounts these observers and whether row-level data can come from the bulk list response (SessionsResponse already carries SessionSummary per session).

Proposed fix

Layered, low-risk:

Fix A (1-line, ships first): Add staleTime: 30_000 (or higher) to useSession. Cuts focus-refetch and mount-refetch volume immediately, with no behavior change because SSE drives the freshness.

Fix B (architectural): In useSSE.ts, audit the two queueSessionDetailInvalidation fallbacks. The right behavior when getSessionPatch returns falsy is to either:

• Synthesize a patch from event.data if it carries enough fields, or
• Do nothing (let SSE deliver a follow-up patch event), or
• Only invalidate if a session-detail page is currently mounted for that ID (i.e., useSession is the active query observer), not unconditionally.

The "only invalidate if a detail observer is active" rule is the cleanest: list rows pull their data from the bulk sessions query (which setQueryData patches in real time), and only the currently-viewed session detail triggers a REST refetch.

Fix C (verify and possibly cut): Confirm the session list page is not mounting N+1 useSession observers. If it is, refactor to consume the SessionSummary from the bulk list query, eliminating per-row REST hits entirely.

Out of scope for this issue

• Changes to the Hono logger() middleware itself (path-skipping). That's a separate workaround if root cause can't be fixed quickly.
• Archiving/garbage-collecting old sessions to shrink fleet size. Real fix is to make fleet size irrelevant to log volume.
• Server-side throttling. The client should not need server-side throttling; it should not be hammering REST in the first place.

Definition of done

• useSession hook respects a sensible staleTime, with a unit test asserting the value.
• SSE invalidation fallback no longer triggers per-session REST refetch when a detail observer is not active. Verified by:
  • Unit test that mounts a useSession-using component, fires a synthetic SSE event with non-patchable data, and asserts no refetch occurs unless the detail observer is active.
  • Manual test: open session-list page in browser, wait 60s, count GET /api/sessions/<uuid> lines in hub log. Should be zero (or one per opened detail page) over that minute.
• Session list page documented as either:
  • Already using SessionSummary from bulk list (no per-row useSession), or
  • Refactored to do so.
• Performance receipts attached to the PR: before/after GET /api/sessions/<uuid> rate over a 5-minute window with 100+ sessions in fleet.

Files touched (expected)

• web/src/hooks/queries/useSession.ts (Fix A)
• web/src/hooks/useSSE.ts (Fix B)
• web/src/components/<session-list> (Fix C - path TBD by peer)
• Test coverage: existing useSession.test.ts, possibly new useSSE.test.ts

Reproduction (reporter's environment)

• Fleet: 131 sessions in ~/.hapi/hapi.db (mix of active / idle / dead).
• Browser: web app open in at least one tab on the session list page.
• Symptom: GET /api/sessions/<uuid> arrives at hub at ~7-15 req/sec idle, bursts to several hundred req/sec on SSE events. ~9.3 GB/day syslog from hapi-hub.service.
• Workaround applied locally: rsyslog stop-rule /etc/rsyslog.d/30-hapi-hub-quiet.conf dropping bun GET /api/sessions/<uuid> lines. Stops disk bleed, does not address CPU/network/cache cost.

[github-actions]

Thanks for the detailed receipts. I checked the current web/hub code and can confirm the core client-side refetch risk is real.

useSession currently has no explicit staleTime or refetchOnWindowFocus override, so it uses TanStack Query defaults while calling api.getSession(sessionId). SSE also has fallback paths that enqueue invalidateQueries() for individual session-detail query keys when a session patch cannot be applied or when the detail cache is missing. With mounted useSession observers, that can turn SSE activity into GET /api/sessions/:id traffic.

One point from the report that I could not confirm in the current tree: SessionList itself does not appear to mount useSession per row. The sessions page fetches the bulk list through useSessions, passes SessionSummary[] into SessionList, and SessionItem renders from that summary object. So the N+1 row-fetch path is not found in the current SessionList code. The active detail routes do mount useSession, including the chat/detail route wrapper.

The low-risk fix direction looks valid: add a sensible freshness window to useSession, then narrow the SSE fallback so a non-patchable session event does not force REST refetches for session-detail queries unless that detail data actually needs to be recovered.

Relevant code:

• web/src/hooks/queries/useSession.ts:19 - useQuery for queryKeys.session(...); no staleTime/refetch options set.
• web/src/hooks/queries/useSession.ts:25 - session detail query calls api.getSession(sessionId).
• web/src/hooks/useSSE.ts:247 - invalidation flush reads queued session IDs.
• web/src/hooks/useSSE.ts:260 - queued session IDs invalidate queryKeys.session(sessionId).
• web/src/hooks/useSSE.ts:359 - patchSessionDetail returns false when no detail cache entry exists.
• web/src/hooks/useSSE.ts:503 - failed detail patch queues session-detail invalidation.
• web/src/hooks/useSSE.ts:509 - non-patchable session update queues session-detail and list invalidation.
• web/src/hooks/queries/useSessions.ts:12 - bulk sessions query used for session summaries.
• web/src/router.tsx:162 - sessions page uses useSessions(api).
• web/src/router.tsx:514 - sessions page passes bulk sessions into SessionList.
• web/src/components/SessionList.tsx:590 - SessionItem accepts a SessionSummary; no per-row useSession found here.
• hub/src/web/server.ts:215 - Hono logger() is applied to all web routes.

Need more info:

• Is the high request rate observed on /sessions only, or with a selected /sessions/<id> detail route open in the same tab/layout?
• Does it reproduce with the session groups collapsed, expanded, or both?

---

HAPI Bot