Skip to content

The solution card: what drove a run, recorded and named (without the submit gate) - #37

Open
Ruqii wants to merge 1 commit into
mainfrom
feat/solution-card-model
Open

Ruqii wants to merge 1 commit into
mainfrom
feat/solution-card-model

Conversation

@Ruqii

@Ruqii Ruqii commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Split out of #34 — the additive half, with the submit-time capability gate left behind.

What lands here

The solution card: the labels that decide what a run measured — shape, agent build, model, provider, the options that actually applied, skill, command template, setup line, timeout — plus a sha256 digest over the identity-bearing fields, from canonical JSON so the site can recompute it and refuse a mismatch.

  • report.json records it under provenance.solution.adapter and provenance.solution.adapter_digest. Both additive and optional; no existing key changed (CLAUDE.md constraint 4).
  • A live-synced run is named after it. identity.name, model.config and skills.installed come from the card, so a site-graded row reads claude-agent-acp@0.76.0 · sonnet · a/b@0123456 instead of "model not recorded".
  • tp submit shows the command line before publishing it. The card's cmd and setup are echoed verbatim — never paraphrased, truncated or re-quoted — and unconditionally, including under --yes: that flag means the user pre-consented to skipping the prompt, not that they were never told what a carded submit makes public.

docs/reference/solution-card.md + tests/data/solution_card_vectors.json are the contract for the TypeScript side (W1/W2/W6/W7). Both worked examples and all four shared vectors recompute byte-for-byte from live code.

What is deliberately left out, and why

#34 also gated tp submit on a GET /api/v2/capabilities probe for features.solution_adapter.supported, withholding the solution repository when the server would not confirm it stores cards. Verified against the deployed site today:

GET https://trapstreet.run/api/v2/capabilities
features: local_live_progress, platform_grading, checkpoint_recovery, run_context

There is no solution_adapter key, so that gate resolves one way for every caller. Merging it now would mean every carded, anchored tp submit loses its repo and goes up unranked — which is exactly the built-in-shape runs, the path that exists so someone can measure an agent without writing a solution. Three things compound it:

  1. No override. Nothing on the command line can opt back in.
  2. A probe failure is indistinguishable from a server "no." capabilities() catches httpx.HTTPError and returns {}, which takes the same branch, and the user-facing message reads the same either way — so a flaky network silently demotes a run.
  3. The secret scan that would make the verbatim display a net is not built (cmd/setup are shown, not checked), so the gate adds surface without adding protection.

None of that is an argument against the gate — it is an argument for landing it with the site work that makes it resolve both ways. The card model, card_digest and card_label here are unchanged from #34, so that half rebases onto this cleanly.

Gates

Gate Result
ruff format --check 123 files already formatted
ruff check All checks passed
scripts/lint_docs.py docs lint OK
pyright src/ 0 errors
pytest --cov 1336 passed, 15 skipped, 100% branch coverage

Built on top of #35, whose RUN_SETTLED changes are preserved intact.

🤖 Generated with Claude Code

The solution card -- shape, agent build, model, provider, the options that
actually applied, skill, command template, setup line, timeout -- plus a
sha256 digest over the identity-bearing fields, computed from canonical JSON
so a server in another language can recompute it and refuse a mismatch.

- `report.json` records it under `provenance.solution.adapter` and
  `provenance.solution.adapter_digest`. Both additive and optional; no
  existing key changed.
- A live-synced run is named after it: `identity.name`, `model.config` and
  `skills.installed` come from the card, so a site-graded row reads
  `claude-agent-acp@0.76.0 · sonnet · a/b@0123456` instead of "model not
  recorded".
- `tp submit` shows the card's `cmd` and `setup` verbatim before publishing
  them -- unconditionally, including under `--yes`, since that flag means the
  user pre-consented to skipping the prompt, not that they were never told.

Split out of #34. The half left behind is the submit-time capability gate:
`GET /api/v2/capabilities` on the deployed site advertises no
`solution_adapter` key today, so that gate would withhold the solution repo
from every carded, anchored submit -- exactly the built-in-shape runs -- and
send them up unranked, with no flag to override and a probe failure reading
identically to a server that answered no. It belongs with the site-side work
(W1/W2) that makes it resolve both ways, not ahead of it.

`docs/reference/solution-card.md` plus `tests/data/solution_card_vectors.json`
are the contract for the TypeScript implementation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant