[MCP/OpenAPI] Return envelope for search and list verbs #1107

Context

Spun out of #1095 (MCP/OpenAPI shared descriptions). That issue ships outputSchema for the record-shaped verbs (get/create/update/patch/delete) where the return shape is trivially derivable from the table attributes. The search_* verb is intentionally deferred because its return shape is a wrapped envelope around the records, and the wrapper shape is an open design question.

This issue picks the wire-protocol envelope for search_* (and any future list/page verb) so that:

• MCP clients get a real outputSchema they can validate against
• OpenAPI emits a non-empty responses[200] schema
• Both consumers share the same shape, same source

Why this is non-trivial

outputSchema/response.schema are compatibility commitments. Once shipped, MCP clients validate against the declared shape; renaming cursor → nextCursor is a breaking change. The envelope decision shouldn't be made inside a description-quality issue — it deserves its own design discussion with explicit review.

Today Harper's search_* returns an iterable that the MCP layer wraps before sending. The exact wire shape isn't widely documented; this issue should both decide the shape and write down what shipped.

The decision

Pick one envelope (or design a new one). Candidate shapes:

Option A — {records, cursor} (concise)

{
  "records": [/* Product records */],
  "cursor": "opaque-pagination-token-or-null"
}

JSON Schema:

{
  "type": "object",
  "properties": {
    "records": { "type": "array", "items": <ProductSchema> },
    "cursor": { "type": ["string", "null"], "description": "Opaque pagination cursor; null when no more pages." }
  },
  "required": ["records", "cursor"],
  "additionalProperties": false
}

Pros: minimal, matches existing internal conventions if records/cursor is what we already use.
Cons: no place for total counts, page metadata, or warnings.

Option B — {data, nextCursor} (REST API convention)

Matches common REST API patterns (GitHub, Stripe-ish).

{
  "data": [/* records */],
  "nextCursor": "opaque-token-or-null"
}

Pros: familiar shape to API consumers; data is the most common modern convention.
Cons: less concise; deviates from Harper's internal records vocabulary if that's what's there today.

Option C — Richer envelope with optional metadata

{
  "records": [/* records */],
  "pagination": {
    "cursor": "opaque-token-or-null",
    "hasMore": true,
    "estimatedTotal": null
  },
  "warnings": []
}

Pros: extensible; supports per-query metadata; clear separation of data vs. meta.
Cons: more complex; over-engineered for v1.

Option D — Match existing internal shape verbatim

Check what Harper's search_* actually returns today (the wire format that REST clients already consume), commit to that shape unchanged. Zero migration risk.

Decisions needed

• Envelope shape (A / B / C / D / something else)
• Cursor naming (cursor / nextCursor / pageToken)
• Cursor nullability (null when no more pages, or omitted)
• Inclusion of hasMore / total / estimatedTotal metadata
• Error semantics — does the envelope carry errors, or do they go through MCP's isError mechanism on the tool result?
• Whether the same shape applies to list_*-style operations (currently list_users, list_roles, etc. in operations profile)

Investigation needed

• What does Harper's current search_* actually return on the wire? (Check resources/Resource.ts search implementation + the REST middleware that serializes it.)
• Are there existing customer/integration assumptions about the response shape? (Anything in harper/documentation REST reference?)
• Does the OpenAPI generator emit anything for search responses today? (Likely no per [MCP] Quality descriptions and richer tool-payload metadata for LLM tool selection #1095 audit, but worth checking.)
• How does pagination work today — query params, headers, response body field? Whatever already exists should drive the response envelope.

Scope

In:

• Picking and documenting the envelope shape
• deriveSearchOutputSchema(attributes) in components/mcp/tools/schemas/derive.ts
• Wiring it into the search_* addTool call in application.ts
• OpenAPI response schemas for search_* paths via the same shape
• static outputSchemas.search override path for programmatic Resources (parallel to the other verbs from [MCP] Quality descriptions and richer tool-payload metadata for LLM tool selection #1095)

Out:

• Pagination input semantics (cursor vs limit/offset) — separate concern, already partially handled by deriveSearchSchema
• List operations in the operations profile (list_users, list_roles) — consider separately or fold in once envelope shape is picked
• Streaming responses — MCP doesn't currently model these in outputSchema

Acceptance criteria (sketch — refine after design)

• Envelope shape chosen and documented
• deriveSearchOutputSchema(attributes) implemented in derive.ts
• application.ts search_* addTool call passes outputSchema
• OpenAPI responses[200] for search paths emits the same shape
• Integration test: a search_* call's return matches the declared outputSchema (validated via the official MCP SDK)
• Migration note in harper/documentation if the envelope differs from any pre-[MCP] Quality descriptions and richer tool-payload metadata for LLM tool selection #1095 wire format

Dependencies

• Blocks: any MCP-client UX work that wants to surface paginated results with typed fields
• Blocked by: nothing structural; can land in parallel with or after [MCP] Quality descriptions and richer tool-payload metadata for LLM tool selection #1095

Reference

• [MCP] Quality descriptions and richer tool-payload metadata for LLM tool selection #1095 — parent issue (MCP/OpenAPI shared descriptions); search_* outputSchema deliberately deferred there.
• MCP rev 2025-06-18 spec — outputSchema is optional JSON Schema on tool descriptors.