Read-authority HTTP 503: preserve sanitized dependency reasons in server-side diagnostics

[Zigoljube]

Summary

Retryable read-authority failures correctly return HTTP 503, but the internal authority source and reason are not available in the captured server diagnostics. Operators cannot reliably distinguish a local store/metadata failure from a chain-authority lookup failure.

Please preserve sanitized, structured server-side attribution and request correlation while keeping the public response generic and authorization fail-closed.

Environment

• DKG 10.0.19, Node.js 22.22.0, macOS ARM64
• Base mainnet (8453)
• Managed external Oxigraph backend
• Authenticated POST /api/query, scoped Context Graph, view: shared-working-memory

Observed response

{
  "code": "CONTEXT_GRAPH_READ_AUTHORITY_UNAVAILABLE",
  "message": "Context Graph read authority is temporarily unavailable; retry once chain and metadata access recover."
}

HTTP status: 503. The runtime handler also provides retryability semantics. The generic wording does not establish which dependency failed.

Correlated evidence — 26 September 2026, UTC

• 14:16:06.375: store normal lane reported eight queued and two active operations approximately 21 seconds old.
• 14:16:06.376: managed SPARQL client deadline exceeded; server termination for supervised recovery initiated.
• 14:16:06.420: Oxigraph terminated with SIGKILL for recovery; dependent operations reported Managed Oxigraph is recovering; query was not started.
• 14:16:06.463504: client captured the explicit authority 503.
• Successful scoped queries occurred immediately before this window.

A complete programmatic scan of the daemon log through 223,353 lines, together with the saved client errors, did not reveal the internal authority reason for this request. The local-store outage strongly supports a store-dependent authority failure, but the exact caught branch and request-level causal attribution remain unproven. Historical RPC quorum failures are separate evidence and do not establish this request's cause.

Source trace in installed 10.0.19

Paths below refer to distributed JavaScript, not assumed TypeScript line numbers:

• dkg-agent/dist/dkg-agent-query.js:76–95: scoped queries resolve authority before executing the requested SPARQL; unavailable decisions throw a typed exception.
• dkg-agent/dist/context-graph-read-authority.js:9–22: internal exception retains source and reason.
• CLI dist/daemon/routes/query.js:355–371: authority errors are handled before the generic store-unavailable branch.
• CLI dist/daemon/http-utils.js:99–106: renders the generic authority 503 without those internal details.
• dkg-agent/dist/dkg-agent-cg-registry.js:540–559 and local-context-graph-registration-status.js:21–36: even the locally created/unregistered shortcut requires a fresh store-backed registration-status read. Its failure can become local-existence-unavailable.
• Metadata projection rebuilds provide other store-dependent failure paths. A registered-chain source label alone does not prove an RPC was called.

Requested change

1. Emit a structured server-side diagnostic when an unavailable authority decision is rendered: safe reason enum, authority source, dependency category when known, operation/request correlation ID, and bounded timing where available.
2. Preserve the same HTTP 503, retryability, and authorization behavior. Do not return stale authorization, suppress the failure, or convert it to an empty success.
3. Keep sensitive graph/caller details, RPC URLs, credentials and raw dependency errors out of public responses. Restrict/sanitize server diagnostics and rate-limit repeated events.
4. Add regression tests injecting a registration-status store error and a chain-authority failure. Assert identical safe public behavior but distinguishable internal attribution. Preserve existing denied-query behavior.

Related

• canReadContextGraph collapses an 'unavailable' read-authority decision into 'false', and /api/query returns HTTP 200 with zero bindings #2641 concerns unavailable authority being rendered as HTTP 200 with zero bindings. This report is different: 503 is correct; operators need internal dependency attribution.
• Investigate recurring main-thread pauses across mainnet cores, including 10.0.19 #2812 tracks main-thread pause investigation. Workload performance may contribute to outages, but this diagnostics change is useful independently of their cause.

No permission bypass or unverified root-cause fix is requested. Raw private logs and profiles are not attached.