Replication W8: Observability & metrics pipeline #437

Workstream W8 of #430 · observability & metrics pipeline

Summary

Today there are two half-pipes that don't meet: cluster_status (an ephemeral poll snapshot of health gauges) and hdb_analytics (durable time-series, but only byte counters). Neither gives an operator a divergence alarm — which is why #426 (silent ~39% loss) and #386 (344 lost attachments) went undetected with connected:true. Connect the pipes and add the missing signals.

Tier 1 — extend cluster_status (cheap, high diagnostic value, mostly already filed)

• Last connection error {message, code} per socket (cluster_status: include last connection error message and code #214) — store the error the close/error handlers already see. Biggest "why is it down" win.
• Resume-cursor + applied-vs-head lag per socket — surface lastDurableSequenceId/persisted seqId and a computed lag; fix lastReceivedVersion not populating on the relay/data path (it's a non-signal exactly when you need it — see rapid-reconnect stress: follower permanently loses replication backlog after restart churn (data loss; fast-skip ruled out) #426).
• Removed-node truth + replace the hardcoded is_enabled:true with real membership (cluster_status on a removed node should clearly indicate removal and close sockets #217).
• Shard count (Expose shard count via cluster status API #204).
• Blob in-flight depth + "cursor held by blob gap" flag (complements the already-shipped blobReplicationFailures).
• Audit/transaction-log backlog depth (pending between cursor and head). (Reframes the abandoned-NATS Add NATS message stream backlog counts to cluster_status #261 onto the live audit log.)
• Make the op cheap — drop the full getSystemInfo (cluster_status: avoid full getSystemInfo call for performance #260).

Tier 2 — a real metrics pipeline (trends + alerts + divergence)

• Promote replication-health gauges into hdb_analytics (it already ingests replication bytes — extend to lag, back-pressure %, blob-failure rate, connection state, cursor position).
• Prometheus / OpenTelemetry export over that series (none exists today).
• Divergence detection — periodic cross-peer record/blob-count comparison emitting an alertable metric. This is the connective tissue with W2: it converts every "silent divergence" into a loud, actionable signal before data loss.

Tier 3 — log hygiene

• Rate-limit/dedup repeated log lines with run-counts (Rate-limit / deduplicate repeated identical log entries in replication #262) — defuses the replication: no-backoff subscription-setup retry storm on transient boot-time DNS failure ends in OOM #327/Replication: 5.0.31 wedge-reconcile spins at 100% CPU re-subscribing an unreachable peer (forceResubscribe leaks into persistent listeners) #349 storm floods and makes logs alert-parseable.

Retires / advances

• cluster_status: include last connection error message and code #214, cluster_status on a removed node should clearly indicate removal and close sockets #217, Expose shard count via cluster status API #204, cluster_status: avoid full getSystemInfo call for performance #260, Add NATS message stream backlog counts to cluster_status #261 (reframed), Rate-limit / deduplicate repeated identical log entries in replication #262
• Closes the loop W2 opens (divergence alarm)

Dependencies

W1 (registry/last-error surfacing), W2 (gap signal), W4 (per-origin lag). Tier 1 can largely proceed in Phase 1 as quick wins; Tier 2 is the strategic piece.

Effort / risk

M / low.

Acceptance criteria

• An operator can see per-subscription lag, last connection error, and cursor position.
• A diverging cluster raises an alert before data loss.
• Storm-condition logs are bounded and parseable.

🤖 Filed by Claude on behalf of Kris.