feat: add replay harness with golden fixture comparison - #357
Draft
meowgorithm wants to merge 4 commits into
Draft
meowgorithm wants to merge 4 commits into
meowgorithm wants to merge 4 commits into
Conversation
meowgorithm
marked this pull request as draft
September 10, 2026 12:17
Adds charm.land/fantasy/replaytest: Load replays request.json plus response.sse/response.json through an httptest server, Collect turns StreamParts into stable PartRecords, AssertGolden compares them against goldens under -update with a unified diff, and RunAgentStep drives one agent step and captures the next request body. Providers now generate IDs through the injectable fantasy.NewID so harness runs are deterministic. Shapes fixtures under providertests/testdata/shapes document current stream behavior for openaicompat and anthropic, and the openaicompat replay cases run on the harness. 💘 Generated with Crush Assisted-by: Crush:glm-5.3
meowgorithm
force-pushed
the
charm-2124-f01-fixture-replay-harness-with-golden-comparison
branch
from
September 10, 2026 12:19
084fb08 to
01ab8e5
Compare
Git checks text files out with CRLF on Windows runners, which broke golden byte comparison, merged SSE events when splitting response.sse, and produced backslash subtest names. Line endings are now normalized to LF before golden comparison and before splitting SSE events, the unified diff emits one line per diff line, and shape subtest names are slashed. Coverage added for the CRLF paths. 💘 Generated with Crush Assisted-by: Crush:glm-5.3-flash
meowgorithm
force-pushed
the
charm-2124-f01-fixture-replay-harness-with-golden-comparison
branch
from
September 10, 2026 12:26
0e60b8e to
e844391
Compare
💘 Generated with Crush Assisted-by: Crush:gpt-6-astra
💘 Generated with Crush Assisted-by: Crush:gpt-6-astra
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a test harness that plays recorded model responses through the real provider code and compares what comes out against saved golden files, so any change in how we handle streaming shows up as a reviewable diff instead of slipping through.
If someone touches the streaming code and the sequence of events we emit changes, CI now fails with a readable diff showing exactly which event changed and how.
What this looks like in practice:
fantasy.NewID) that the harness swaps for a simple counter during tests.go test ./providertests -run TestFixtureShapes -updateto regenerate goldens; the README explains the format and the rule that existing goldens are never silently regenerated.Closes CHARM-2124 ([F01] Fixture replay harness with golden comparison)
What changed
charm.land/fantasy/replaytest(module root, not under internal/):Load(dir)readsrequest.jsonplus exactly one ofresponse.sse/response.json;response.sseis split into events on blank-line boundaries, each event kept verbatim.Serve(t, fixture, opts...)runs an httptest server that writes one SSE event at a time with a flush after each (never merging or splitting events), records every request body inServer.Requests(), and answers second and later requests with a canned minimal finish response shaped by the request itself (streaming or not, messages or chat-completions endpoint) unlessWithSubsequentSSE/WithSubsequentJSONsupplies one. A fixture event of exactly<connection closed>flushes and closes the connection.PartRecord/UsageRecordplusCollect(stream)normalize everyStreamPartinto a stable, comparable record; only fields meaningful for the part type are emitted (omitempty), usage zeros are omitted, and error text has the replay server's listener port normalized.AssertGolden(t, path, records)marshals with 2-space indent and stable key order, compares bytes with the golden file, writes it under-update(flag nameupdate), and on mismatch fails with a unified diff.RunAgentStep(t, model, fixture, tools...)runsAgent.Streamfor one step with stub tools (dispatches recorded), retries disabled, and returns content asPartRecords, the finish reason, dispatched tool names, and the raw body of the next request the agent sent (next_request), so step goldens keep byte-for-byte next-request assertions.fantasy.NewID(defaultuuid.NewString); call sites routed inproviders/openai/language_model.go,providers/openai/responses_language_model.go,providers/google/google.go, andproviders/kronk/language_model.go.RunAgentStepswaps in a counter (id-1,id-2, ...) for the duration of a test and restores it.providertests/testdata/shapes/<shape>/<case>/(meta.json,request.json,response.sse, reviewedparts.golden.json), all shape-named, scrubbed, and documenting current behavior:plain_text/simple,reasoning_then_text/basic,reasoning_tail_batched_with_content/basic,toolcall/single,toolcall/parallel_two,finish_stop_with_toolcalls/basic,finish_unknown_with_toolcalls/basic,truncated_toolcall_length/basic,usage_in_trailing_chunk/basic,usage_in_finish_chunk/basic,usage_without_total_tokens/basic,usage_then_empty_chunk/basic,reasoning_field_object/basic,anthropic/toolcall_stream/basic,anthropic/thinking_with_signature/basic, plus the shapes migrated fromproviders/openaicompat/replay_test.go(reasoning_then_toolcall/interleaved_with_nulls,reasoning_only/finish_length,reasoning_tail_batched_with_toolcall/basic,reasoning_then_text/null_finish,reasoning_empty_then_toolcall/basic,toolcall/connection_closed,multi_choice/reasoning_two_choices,multi_choice/reordered_choices,toolcall/finish_insufficient_resource,toolcall/finish_content_filter,toolcall/finish_missing_truncated,toolcall/finish_missing_valid,toolcall/finish_missing_no_args).providertests/shapes_test.go: one table-driven test that loads every fixture undershapes/, serves it, streams through the provider named inmeta.json, and assertsparts.golden.json.providers/openaicompat/replay_test.go: all existing cases migrated onto the harness; the duplicated localserveSSEserver code is removed. The manual two-step round trip keeps its byte-for-bytereasoning_contentassertion on the second request body; the agent-level round trips run throughRunAgentStepand pinstep.golden.json.replaytest/README.mddocuments the fixture layout, the golden format, the-updateflag, the rule that existing goldens are never regenerated by an agent, and that fixtures are shape-named and scrubbed.Why
Recorded-fixture replay with golden comparison makes regressions in stream handling visible diffs across providers.
Test evidence
Failing before (test written first):
Passing after:
go build ./...— cleango test ./... -count=1— all 14 packages ok (includes 28 shapes fixtures asserting goldens, migrated openaicompat replay cases, and replaytest unit tests); run twice to confirm stabilitygolangci-lint run— 0 issues (gofumpt/gofmt clean)go test ./providertests -run TestFixtureShapes -updateregenerates the goldens; every golden was reviewed (see note below)Golden diffs
-updatefor fixtures added by this card, then reviewed:reasoning_end), tool-call suppression with warnings onlength/content_filter/insufficient_system_resource, unknown finish reason (eos→unknown) still dispatching complete calls, usage semantics (trailing chunk and inline finish chunk both surface; usage withouttotal_tokensis discarded; a following empty chunk resets usage to zero),reasoningfield as a JSON object surfacing a stream error part.inputwith an emptydelta(documented as-is; the later tool-input card changes this), thinking block with signature metadata on the signature delta and onreasoning_end, finish carries message id and computed total tokens.Follow-ups noticed (not done)
Fixture.Requestis stored but never asserted; a future card can add request-side golden assertions against it.Cannedsubsequent responses are detected by endpoint shape (messages vs chat-completions); a provider needing a different canned shape should supplyWithSubsequentSSE/WithSubsequentJSON."0"), so multi-choice text deltas all share one id while reasoning parts follow choice index (visible in the multi_choice goldens).usage_without_total_tokensandusage_then_empty_chunkboth end with zero usage on the finish part; the usage-semantics card (F03) changes this and will update those goldens.reasoning_field_objectsurfaces a stream error part instead of tolerating the object shape (F08 target).💘 Generated with Crush