Skip to content

feat: add replay harness with golden fixture comparison - #357

Draft
meowgorithm wants to merge 4 commits into
mainfrom
charm-2124-f01-fixture-replay-harness-with-golden-comparison
Draft

meowgorithm wants to merge 4 commits into
mainfrom
charm-2124-f01-fixture-replay-harness-with-golden-comparison

Conversation

@meowgorithm

@meowgorithm meowgorithm commented Sep 10, 2026 •

Copy link
Copy Markdown
Member

This PR adds a test harness that plays recorded model responses through the real provider code and compares what comes out against saved golden files, so any change in how we handle streaming shows up as a reviewable diff instead of slipping through.

If someone touches the streaming code and the sequence of events we emit changes, CI now fails with a readable diff showing exactly which event changed and how.

What this looks like in practice:

  • Each test case is a tiny folder with a canned model response (an SSE stream, hand-written and scrubbed of anything real) and a golden file listing the stream events the provider should emit, e.g. "reasoning started, reasoning text, text started, ... finished because tool_calls".
  • A table-driven test replays every one of those responses through the actual provider and compares the event sequence against the golden file. Today that covers 28 cases across chat-completions-style endpoints and the Anthropic protocol, including the gnarly ones: reasoning text, tool calls that arrive in fragments or in parallel, streams that die mid-tool-call, odd finish reasons, and usage stats arriving in different chunks.
  • There is also an agent-level helper that runs one agent step, records which tools it dispatched, and captures the exact request the agent sent back upstream — this pins the bug fix where a model's reasoning had to round-trip byte-for-byte on the next request.
  • To keep goldens stable across runs, generated IDs now flow through an injectable generator (fantasy.NewID) that the harness swaps for a simple counter during tests.
  • Run go test ./providertests -run TestFixtureShapes -update to regenerate goldens; the README explains the format and the rule that existing goldens are never silently regenerated.

Closes CHARM-2124 ([F01] Fixture replay harness with golden comparison)

What changed

  • New importable package charm.land/fantasy/replaytest (module root, not under internal/):
    • Load(dir) reads request.json plus exactly one of response.sse/response.json; response.sse is split into events on blank-line boundaries, each event kept verbatim.
    • Serve(t, fixture, opts...) runs an httptest server that writes one SSE event at a time with a flush after each (never merging or splitting events), records every request body in Server.Requests(), and answers second and later requests with a canned minimal finish response shaped by the request itself (streaming or not, messages or chat-completions endpoint) unless WithSubsequentSSE/WithSubsequentJSON supplies one. A fixture event of exactly <connection closed> flushes and closes the connection.
    • PartRecord/UsageRecord plus Collect(stream) normalize every StreamPart into a stable, comparable record; only fields meaningful for the part type are emitted (omitempty), usage zeros are omitted, and error text has the replay server's listener port normalized.
    • AssertGolden(t, path, records) marshals with 2-space indent and stable key order, compares bytes with the golden file, writes it under -update (flag name update), and on mismatch fails with a unified diff.
    • RunAgentStep(t, model, fixture, tools...) runs Agent.Stream for one step with stub tools (dispatches recorded), retries disabled, and returns content as PartRecords, the finish reason, dispatched tool names, and the raw body of the next request the agent sent (next_request), so step goldens keep byte-for-byte next-request assertions.
  • Determinism: providers now generate IDs through the new injectable fantasy.NewID (default uuid.NewString); call sites routed in providers/openai/language_model.go, providers/openai/responses_language_model.go, providers/google/google.go, and providers/kronk/language_model.go. RunAgentStep swaps in a counter (id-1, id-2, ...) for the duration of a test and restores it.
  • First fixtures under providertests/testdata/shapes/<shape>/<case>/ (meta.json, request.json, response.sse, reviewed parts.golden.json), all shape-named, scrubbed, and documenting current behavior: plain_text/simple, reasoning_then_text/basic, reasoning_tail_batched_with_content/basic, toolcall/single, toolcall/parallel_two, finish_stop_with_toolcalls/basic, finish_unknown_with_toolcalls/basic, truncated_toolcall_length/basic, usage_in_trailing_chunk/basic, usage_in_finish_chunk/basic, usage_without_total_tokens/basic, usage_then_empty_chunk/basic, reasoning_field_object/basic, anthropic/toolcall_stream/basic, anthropic/thinking_with_signature/basic, plus the shapes migrated from providers/openaicompat/replay_test.go (reasoning_then_toolcall/interleaved_with_nulls, reasoning_only/finish_length, reasoning_tail_batched_with_toolcall/basic, reasoning_then_text/null_finish, reasoning_empty_then_toolcall/basic, toolcall/connection_closed, multi_choice/reasoning_two_choices, multi_choice/reordered_choices, toolcall/finish_insufficient_resource, toolcall/finish_content_filter, toolcall/finish_missing_truncated, toolcall/finish_missing_valid, toolcall/finish_missing_no_args).
  • providertests/shapes_test.go: one table-driven test that loads every fixture under shapes/, serves it, streams through the provider named in meta.json, and asserts parts.golden.json.
  • providers/openaicompat/replay_test.go: all existing cases migrated onto the harness; the duplicated local serveSSE server code is removed. The manual two-step round trip keeps its byte-for-byte reasoning_content assertion on the second request body; the agent-level round trips run through RunAgentStep and pin step.golden.json.
  • replaytest/README.md documents the fixture layout, the golden format, the -update flag, the rule that existing goldens are never regenerated by an agent, and that fixtures are shape-named and scrubbed.

Why

Recorded-fixture replay with golden comparison makes regressions in stream handling visible diffs across providers.

Test evidence

  • Failing before (test written first):

    providertests/shapes_test.go:13:2: no required module provides package charm.land/fantasy/replaytest
    FAIL    charm.land/fantasy/providertests [setup failed]
    
  • Passing after:

    • go build ./... — clean
    • go test ./... -count=1 — all 14 packages ok (includes 28 shapes fixtures asserting goldens, migrated openaicompat replay cases, and replaytest unit tests); run twice to confirm stability
    • golangci-lint run — 0 issues (gofumpt/gofmt clean)
    • go test ./providertests -run TestFixtureShapes -update regenerates the goldens; every golden was reviewed (see note below)

Golden diffs

  • No existing golden was regenerated. All goldens in this PR are new, created once with -update for fixtures added by this card, then reviewed:
    • openai-compat chat completions: text/reasoning part ordering on batched boundary chunks (text delta before reasoning_end), tool-call suppression with warnings on length/content_filter/insufficient_system_resource, unknown finish reason (eos → unknown) still dispatching complete calls, usage semantics (trailing chunk and inline finish chunk both surface; usage without total_tokens is discarded; a following empty chunk resets usage to zero), reasoning field as a JSON object surfacing a stream error part.
    • anthropic: tool-input deltas carry partial JSON in input with an empty delta (documented as-is; the later tool-input card changes this), thinking block with signature metadata on the signature delta and on reasoning_end, finish carries message id and computed total tokens.

Follow-ups noticed (not done)

  • Fixture.Request is stored but never asserted; a future card can add request-side golden assertions against it.
  • Canned subsequent responses are detected by endpoint shape (messages vs chat-completions); a provider needing a different canned shape should supply WithSubsequentSSE/WithSubsequentJSON.
  • The openai text stream keys text parts by a constant id ("0"), so multi-choice text deltas all share one id while reasoning parts follow choice index (visible in the multi_choice goldens).
  • usage_without_total_tokens and usage_then_empty_chunk both end with zero usage on the finish part; the usage-semantics card (F03) changes this and will update those goldens.
  • reasoning_field_object surfaces a stream error part instead of tolerating the object shape (F08 target).

💘 Generated with Crush

@meowgorithm
meowgorithm marked this pull request as draft September 10, 2026 12:17
Adds charm.land/fantasy/replaytest: Load replays request.json plus
response.sse/response.json through an httptest server, Collect turns
StreamParts into stable PartRecords, AssertGolden compares them against
goldens under -update with a unified diff, and RunAgentStep drives one
agent step and captures the next request body. Providers now generate
IDs through the injectable fantasy.NewID so harness runs are
deterministic. Shapes fixtures under providertests/testdata/shapes
document current stream behavior for openaicompat and anthropic, and
the openaicompat replay cases run on the harness.

💘 Generated with Crush

Assisted-by: Crush:glm-5.3
@meowgorithm
meowgorithm force-pushed the charm-2124-f01-fixture-replay-harness-with-golden-comparison branch from 084fb08 to 01ab8e5 Compare September 10, 2026 12:19
Git checks text files out with CRLF on Windows runners, which broke
golden byte comparison, merged SSE events when splitting response.sse,
and produced backslash subtest names. Line endings are now normalized
to LF before golden comparison and before splitting SSE events, the
unified diff emits one line per diff line, and shape subtest names are
slashed. Coverage added for the CRLF paths.

💘 Generated with Crush

Assisted-by: Crush:glm-5.3-flash
@meowgorithm
meowgorithm force-pushed the charm-2124-f01-fixture-replay-harness-with-golden-comparison branch from 0e60b8e to e844391 Compare September 10, 2026 12:26
💘 Generated with Crush

Assisted-by: Crush:gpt-6-astra
💘 Generated with Crush

Assisted-by: Crush:gpt-6-astra
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant