Design: complete Harper's MCP protocol surface

Status: Planned — roadmap committed (see §5/§6 decisions) · Spec: MCP 2025-06-18 (backcompat 2025-03-26) · Scope: components/mcp/

Design + tracking doc for the MCP protocol features Harper's server does not yet implement. This revision grounds every feature in existing Harper primitives (verified against the code) so the work is mostly protocol-adaptation rather than new infrastructure, corrects several inaccuracies from the first draft (flagged in cross-model review), and re-orders phasing by the now-clearer effort.

Revision note. ping, logging/setLevel, and notifications/message shipped in #1350 (merged) and are removed from scope here. Corrections vs. the first draft are called out inline as [corrected].

1. Current state (verified)

Implemented (components/mcp/transport.ts dispatch):

• Lifecycle: initialize, notifications/initialized; sessions persisted in system.mcp_session (session.ts) with TTL idle-eviction; DELETE termination (gated by mcp.session.allowClientDelete).
• ping; logging/setLevel + notifications/message (feat(mcp): implement ping + logging/setLevel + notifications/message #1350).
• tools/list, tools/call; resources/list, resources/read, resources/templates/list.
• Server→client push over the GET SSE channel: notifications/tools/list_changed, notifications/resources/list_changed (listChanged.ts), framed + streamed by components/mcp/sse.ts — see the GET-SSE delivery correction below.
• Advertised capabilities (lifecycle.ts SERVER_CAPABILITIES): tools.listChanged, resources.listChanged, logging.

[corrected] Pagination: tools/list and resources/list use opaque cursors (pagination.ts). resources/templates/list does not — it ignores request params and returns all templates with no nextCursor (transport.ts dispatchResourceTemplatesList, resources.ts listResourceTemplates). The 2025-06-18 spec says resource templates support pagination, so this is a real gap (see §3.7). tools/call and resources/read are not paginated (correct — they aren't list methods).

[corrected] GET-SSE delivery — was broken on main, fixed by #1386: an earlier revision of this doc listed GET-SSE push as simply "Implemented." It was not working end-to-end on main:

• Application profile: the GET never established — Node defers header transmission until the first body byte, and the SSE queue yields nothing until a push, so the GET hung with headers unsent (and the raw queue objects were never serialized to SSE text).
• Operations profile: the GET established headers but never streamed pushed frames — reply.send(IterableEventQueue) on Fastify sends headers then stalls after one chunk. So list_changed was advertised + dispatched but undelivered.

The listChanged dispatcher itself was correct (it fired and called queue.send — including same-worker, via signalUserChange's local handler call, and cross-worker via ITC utility/signalling.ts / server/threads/itc.js); the gap was purely the SSE transport. Fixed in components/mcp/sse.ts (toSseStream — a primed, framed SSE Readable) wired into both adapters (Harper-HTTP pipes it; Fastify hijack()s the reply and pipes to the raw socket). Bun: the operations profile is bridged to Fastify via fastify.inject(), which buffers and resolves only on response end — so the SSE GET hung forever. #1386 also fixes that in server/http.ts (bunDelegateToNodeServer uses payloadAsStream to stream text/event-stream responses, and propagates client disconnect back to the hijacked reply for session cleanup). The fix is therefore solid on all runtimes, so the §3.4 / §3.6 / §3.7 features that build on the SSE channel can rely on it on both Node and Bun.

Per-worker note: the dispatcher is cross-worker-correct (ITC fan-out). The genuine per-worker limitation applies only to push that is not ITC- or audit-log-backed (e.g. the logging setLevel level-propagation caveat in #1350, and any future per-session push) — see §4.

2. Foundational Harper primitives (the leverage)

Four existing primitives carry most of this work. Each feature below builds on them.

• P1 — Resource.subscribe() + resources/transactionBroadcast.ts + IterableEventQueue. Durable, cross-worker, reconnect-safe change events: the audit log (memory-mapped LMDB/RocksDB) fires a 'committed' event on every worker; each worker's local subscriptions catch up from the audit log and push to their connections. This is how MQTT delivers subscription updates today (server/DurableSubscriptionsSession.ts).
• P2 — AbortSignal-in-ALS. server/serverHelpers/Request.ts owns an AbortController (auto-aborts on client disconnect); the signal rides the ALS Context (resources/ResourceInterface.ts signal?: AbortSignal) and is already consumed inside Resource/model calls (resources/models/Models.ts resolveCallContext). Cancellation's hard half (propagating into in-flight work) exists.
• P3 — Streaming/progress. server/serverHelpers/progressEmitter.ts (ProgressEmitter + createSSEResponseStream), the text/event-stream serializer (server/serverHelpers/contentTypes.ts), and IterableEventQueue — proven on the GET channel; NormResponse.sseIterable already exists. Note — two distinct SSE serializers: the MCP GET/POST channel frames via components/mcp/sse.ts (serializeSseFrame / toSseStream, added in fix(mcp): deliver server-push SSE on both profiles (GET establish + list_changed) #1386); server/serverHelpers/contentTypes.ts:128-162 is Harper's general text/event-stream serializer (subscriptions / EventSource). New MCP streaming work should extend components/mcp/sse.ts, not contentTypes.ts.
• P4 — Durable system-table pattern. system.mcp_session (session.ts, already holds logLevel) and the MQTT analog hdb_durable_session (DurableSubscriptionsSession.ts, persists subscriptions[] + awaitingAcks[]). The home for subscription state, client capabilities, and pending-request metadata.

3. Features

Severity legend: 🟡 commonly-expected · 🔵 advanced/optional. Reuse legend: High (primitive does the hard part) · Partial (wiring) · Net-new.

3.1 🟡 resources/templates/list pagination — High reuse (quickest win)

• Gap: returns all templates, no cursor.
• Primitive: the cursor helpers already used by the other list methods (pagination.ts encodeCursor/decodeCursor).
• Net-new: slice + nextCursor in dispatchResourceTemplatesList, mirroring dispatchToolsList. Trivial.

3.2 🟡 completion/complete — High reuse (data source) [corrected scope]

• Spec: completes resource-template variables (ref/resource) and prompt arguments (ref/prompt) — not tool arguments. (First draft wrongly proposed deriving from deriveSearchSchema's tool-input enum.)
• Primitive: template variables map to schema introspection — {database}/{table} from getDatabases() (resources/databases.ts) and the RBAC-aware describeAll (dataLayer/schemaDescribe.ts); {resourcePath} from the Resources registry (components/mcp/resources.ts getResources() with the exportTypes.mcp filter). All RBAC-filterable via user.role.permission[db].tables[t].{read,describe} (the userTablePermissions helper already in resources.ts).
• Net-new: the dispatch handler, ref/resource vs ref/prompt routing, and per-variable candidate filtering (cap 100). Prompt-arg completion depends on §3.5.
• [decided] Data source: ref/resource (template-variable) candidates are schema-derived (the schema is the candidate set — {database}/{table}/{resourcePath}); ref/prompt (prompt-argument) candidates are author-declared via the §3.5 prompt definitions. No auto-derived prompts.

3.3 🟡 Progress + cancellation — Partial reuse [corrected]

• Current: notifications/progress not emitted; notifications/cancelled accepted-but-ignored (in-flight tools/call not cancelled).
• Primitive: P2 (AbortSignal already consumed downstream) + P3 (ProgressEmitter/SSE). The first draft said transactional() needs a new abort hook — not so; the signal mechanism exists, plus an operations SSE progressEmitter precedent.
• Net-new: (a) create an AbortController per tools/call, put its signal on the ALS Context when invoking the handler, and route inbound notifications/cancelled → .abort(); (b) thread _meta.progressToken and expose a progress emitter to tool handlers, delivering notifications/progress. Pairs naturally with §3.4.

3.4 🔵 Per-request SSE streaming on POST — Partial reuse

• Current: POST always returns a single JSON object; sseIterable is used only by the GET channel.
• Primitive: P3 — the serializer + IterableEventQueue + the existing NormResponse.sseIterable field.
• Net-new: allow dispatchToolsCall to return sseIterable, and have the Fastify + Harper-HTTP adapters stream it on POST. Prereq for streaming tool results and for in-band progress/server-requests during a call.
• [corrected] Bun transport already in place (fix(mcp): deliver server-push SSE on both profiles (GET establish + list_changed) #1386): the adapters' SSE branch (res.sseIterable !== undefined) and the Bun inject() bridge (payloadAsStream, streams any text/event-stream response) are method-agnostic — a POST returning sseIterable already streams on both profiles and on Bun. So the operations-on-Bun transport half of this feature is done; what remains is the Node-side wiring (dispatchToolsCall returning sseIterable). Without fix(mcp): deliver server-push SSE on both profiles (GET establish + list_changed) #1386 this would have hit the same Bun inject() hang and needed its own fix.

3.5 🟡 Prompts — prompts/list, prompts/get (+ notifications/prompts/list_changed) — Partial reuse

• Primitive: the component-author opt-in pattern (static mcpTools → registerCustomMcpTools in tools/application.ts); Resource static metadata (Resource.ts static fields) is surfaced at registration. Add a parallel static mcpPrompts + a promptRegistry (mirroring toolRegistry.ts); reuse listChanged.ts for list_changed (export a registry-iteration API rather than reaching into its internals).
• Net-new: the registry, dispatch handlers, pagination, title, argument + rendered-message validation. Prompt content is net-new author-declared (Harper has no existing template/canned-message primitive — verified across GraphQL/openApi/component config).
• [decided] Source: component-author opt-in only (static mcpPrompts, mirroring mcpTools) — no schema auto-derivation of prompt content.

3.6 🔵 Resource subscriptions — resources/subscribe/unsubscribe (+ notifications/resources/updated) — High reuse [corrected design]

• First draft was a blocker: it stored subscriptions on the in-memory RegisteredSession (per-worker, lost on reconnect) and assumed user/schema ITC events are data-change events. Wrong on both counts.
• Primitive: P1 + P4. Build on Resource.subscribe() → transactionBroadcast (durable, cross-worker, reconnect-safe), with subscription state persisted in a durable table (extend mcp_session, à la hdb_durable_session). Register/restore the subscription on the worker holding the client's SSE stream (exactly what MQTT's resume() does on connect), so the audit-log broadcast delivers updates locally.
• Net-new: resource-URI ↔ resource mapping; notifications/resources/updated framing; durable subscription state + restore-on-connect.
• [decided] Subscribable scope: row-backed resources only (URIs whose changes ride the audit-log 'committed' broadcast). Synthetic harper://* URIs have no row-change source and are list_changed-only (not subscribable). Still open: the push-volume coalescing window for notifications/resources/updated.

3.7 🔵 Server-initiated requests — sampling/createMessage, elicitation/create, roots/list — Net-new (the genuinely hard cluster) · [decided] in scope (build it)

• Spec: the server sends a JSON-RPC request to the client and awaits a response. Today inbound client responses are classified fire-and-forget and dropped (jsonrpc.ts isClientFireAndForget, transport.ts → 202).
• Closest primitive: MQTT's QoS awaitingAcks Map<id, …> correlation (DurableSubscriptionsSession.ts) — a model for id→pending, with backpressure, but no timeout/error-routing.
• Net-new: (1) client-capability tracking (prereq) — handleInitialize ignores params.capabilities; read + store on McpSessionRecord (P4) and gate sends on it. McpSessionRecord (session.ts:38-53) has no capabilities field today, so this is a new persisted field (e.g. clientCapabilities). (2) A real Map<id,{resolve,reject,timeout}> pending-request registry on the session, with response routing added to handlePost before the fire-and-forget check, plus error/timeout/cleanup. (3) Delivery + cross-worker correlation: resolved in §7.1 — v1 is in-band only (request rides the tools/call POST's own SSE stream, §3.4), with cross-worker response correlation via the existing crossThread ITC pattern.
• [decided] In scope — to be built (was previously "defer unless a use case lands"). It remains the heaviest, genuinely net-new cluster, so it is sequenced last (Phase 4) behind its prereq (client-capability tracking) and the §3.4 POST-stream delivery path.

3.8 🔵 SSE resumability (Last-Event-ID) — Partial reuse (optional)

• Primitive: the MCP SSE serializer (components/mcp/sse.ts serializeSseFrame) already emits an id: line when frame.id is set; IterableEventQueue.
• Net-new: assign per-stream unique event ids, a bounded replay buffer, and Last-Event-ID handling (replay must not cross streams). Optional, not a conformance prerequisite.

4. Cross-worker push (general note)

listChanged works cross-worker because it's driven by ITC-broadcast user/schema events (§1). Subscriptions (§3.6) work cross-worker because they ride the audit-log 'committed' broadcast (P1). Any new push type that is not backed by ITC or the audit log (e.g. logging level-propagation, or an out-of-band server→client request targeting a session whose stream lives on another worker) needs its own cross-worker delivery story — track per feature; don't assume the GET registry alone suffices.

5. Phasing (full roadmap — committed)

The whole surface is in scope (incl. §3.7). Re-sequenced by effort + dependencies, on top of the now-solid SSE channel + Bun bridge (#1386). Dependency spine: §3.4 → (§3.3 progress, §3.7 in-band delivery); §3.5 → §3.2 prompt-arg completion; §3.7 → client-capability tracking.

Phase 1 — Quick wins (wiring over existing primitives)

• §3.1 resources/templates/list pagination — slice + nextCursor, mirroring dispatchToolsList. Standalone, trivial.
• §3.4 POST per-request SSE streaming — Node-side wiring only (Bun transport already done in fix(mcp): deliver server-push SSE on both profiles (GET establish + list_changed) #1386). Unblocks streaming results + in-band progress/server-requests.
• §3.3 progress + cancellation — AbortController per tools/call → ALS Context.signal; route notifications/cancelled → .abort(); thread _meta.progressToken → notifications/progress over the §3.4 stream.

Phase 2 — Completion + Prompts

• §3.5 prompts — promptRegistry + static mcpPrompts opt-in; prompts/list/get + pagination + list_changed.
• §3.2 completion — schema-derived for ref/resource; author-declared for ref/prompt (depends on §3.5).

Phase 3 — Subscriptions (row-backed)

• §3.6 resources/subscribe/unsubscribe + notifications/resources/updated on Resource.subscribe() → transactionBroadcast, durable state in mcp_session, restore-on-connect. harper://* = list_changed-only.

Phase 4 — Server-initiated requests

• §3.7 prereq client-capability tracking → pending-request registry Map<id,{resolve,reject,timeout}> + response routing in handlePost before the fire-and-forget check + error/timeout/cleanup; in-band delivery on the POST stream (§3.4), out-of-band on the GET queue (mind §4 cross-worker).

Phase 5 — Optional

• §3.8 SSE resumability (Last-Event-ID) — per-stream event ids + bounded replay buffer. Not a conformance prereq; do only if needed.

6. Decisions (resolved) + residual questions

Resolved:

• Scope: full roadmap committed (§5), including §3.7.
• Prompts/completion: component-author opt-in for prompt content (static mcpPrompts); completion is schema-derived for resource-template variables and author-declared for prompt arguments. No auto-derived prompts. (§3.2/§3.5)
• Subscriptions: row-backed resources only; harper://* synthetic URIs are list_changed-only, not subscribable. (§3.6)
• §3.7 server-initiated requests: in scope, sequenced last (Phase 4).

Previously residual — now resolved in §7 (Phase 3–4 design resolutions): server-initiated cross-worker delivery, pending-request timeout/bounding, and the resources/updated coalescing question.

7. Phase 3–4 design resolutions

The Phase 1–2 features are wiring over confirmed primitives. The two heavy phases (§3.6 subscriptions, §3.7 server-initiated) had open design points; resolved below so they can be built without a mid-implementation design wall. All grounded against current code.

7.1 §3.7 — cross-worker delivery + response correlation

Problem. The session registry is per-worker (components/mcp/sessionRegistry.ts) and there is no session affinity (SO_REUSEPORT spreads requests; HTTP_SESSIONAFFINITY is explicitly unsupported), so (A) a session's GET stream lives on one worker, and (B) the client's response to a server→client request is a fresh POST that can land on any worker. The durable mcp_session record does not track the owning worker.

Decision — v1 is in-band only. Server-initiated requests are issued only during a tools/call and stream on that call's own POST SSE response (§3.4). That request is produced and consumed on the same worker handling the POST, so delivery (problem A) is local — no cross-worker push needed. Spontaneous/out-of-band server→client requests (no active call, would need to reach a session's GET stream on another worker) are deferred (speculative; "likely never needed for operations").

Response correlation (problem B). The awaiting worker holds the pending Map<id,…> and stamps originator: threadId. The inbound client-response POST resolves locally if the id is in its map; otherwise it fans the response to the originator using the proven cross-worker request/response pattern already in the codebase — components/status/crossThread.ts (requestId + originator + ITC broadcast, responder replies via server/threads/manageThreads.js sendToThread(originator, …); precedents COMPONENT_STATUS_REQUEST/RESPONSE, RESOURCE_OPENAPI_REQUEST/RESPONSE). Add an MCP_CLIENT_RESPONSE entry to the enumerated hdbTerms.ITC_EVENT_TYPES + a handler in server/itc/serverHandlers.js carrying { jsonrpcId, sessionId, result | error }.

7.2 §3.7 — pending-request timeout + map bounding

• Self-cleaning timeout: each pending entry gets a setTimeout that rejects the promise and deletes itself on expiry — no periodic sweep (Harper has none; mirrors crossThread.ts's timeout-with-graceful-degradation). Default 30s, config mcp.serverRequest.timeoutMs (convention per existing HTTP_TIMEOUT 120s / headers 60s / lock 10s).
• Session teardown: clear the session's pending map when the registry drops the session (disconnect / DELETE / idle-prune already remove the entry — clear pending there, rejecting outstanding promises).
• Bound the map: high-water-mark cap mirroring MQTT's AWAITING_ACKS_HIGH_WATER_MARK = 100 (server/DurableSubscriptionsSession.ts) — reject new server→client requests past the cap rather than grow unbounded.

7.3 §3.6 — resources/updated coalescing

• Ride the existing coalescing. resources/transactionBroadcast.ts already collapses 'committed' bursts into one notify pass per event-loop turn (setImmediate + NOTIFY_BATCH_SIZE = 256); subscribers do not get one event per write today. MCP subscriptions ride this same substrate.
• Lossless URI collapse. notifications/resources/updated carries only the URI (no payload), so multiple changes to one subscribed URI within a notify pass naturally collapse to a single notification.
• Decision: v1 adds no new debounce window — the existing turn-coalescing suffices. If high write volume still over-notifies, add an optional per-subscription debounce mcp.subscriptions.coalesceMs (default 0/off) later. Not a v1 requirement.

7.4 §3.6 — subscription restore

Mirror server/DurableSubscriptionsSession.resume(): persist { uri, startTime } per subscription on the durable mcp_session record; on GET reconnect, re-register each on the worker now holding the stream and catch up from startTime (the audit-log startTime ≥ timestamp filter prevents replaying already-delivered changes).

_{🤖 Design drafted + revised by an LLM (Claude), grounded in the linked Harper code + MCP 2025-06-18 spec and cross-model (Codex) review. §7 grounded against ITC/subscription internals. Reviewed by Kyle.}