Add a sandbox-test skill for agents testing parser changes

[potiuk]

What

Adds .claude/skills/sandbox-test/SKILL.md, a procedure for coding agents (and a
readable checklist for humans) around the sandbox workflow from #127, plus pointers to
it from AGENTS.md and docs/testing-a-change.md.

#127 gave us the machinery to run a parser change against a real repository. It does not
say what to run, and the difference turns out to matter: a single happy-path
"does it turn on?" dispatch passes on code that does nothing at all.

Why now

I put seven open PRs through the workflow this week. The runs found three bugs, and every
one of them would have been missed by a happy-path run:

• Add GitHub immutable releases #84 — immutable_releases: true never enables anything. The state check reads the
  HTTP status (200 assumed to mean "enabled"), but that endpoint returns 200 in both
  states with enabled in the body. The run logs "are currently enabled" against a repo
  reporting {"enabled": false}, exits 0, and changes nothing. The same fault makes the
  absent-key path issue a DELETE against every repository on every apply.
• Add github.code_scanning for CodeQL default setup #122 and Add GitHub pull request creation cap support #111 — code_scanning: false / creation_cap: {enabled: false} cannot
  turn off a setting that is on, because of a guard that returns before the API call when
  the previous-config cache is cold. Only the unset row of a test matrix finds this;
  the enable path works perfectly.
• Fix: prevent_self_review breaks all environment creation via .asf.yaml #101 — changes a security default (prevent_self_review) and silently strips the
  control on a fallback path. Visible only by running the same config against patched and
  unpatched parsers and diffing the result.

Three findings, all in the edges: unset, update, and "leave alone what you do not manage".

What the skill says

• Choosing a sandbox — your own fork, or apache/... if you are a committer — and the
  token permission each feature area needs. A 403 here is usually the token's repository
  access list, not its permission level, which cost me a round trip to work out.
• Preflight — including the check that a branch can run at all. Every PR currently open
  predates the --branch flag that Add a workflow for testing parser changes against a sandbox repo #127 added to asfyaml-run, so every one of them dies
  with unrecognized arguments: --branch until it is rebased. Worth knowing before you
  dispatch.
• A set / update / idempotent / unset / removal / unmanaged / negative matrix, with the
  rule that a test whose pass state equals its start state proves nothing.
• Two traps that make an edge case silently untestable. github: directives only run
  when the processed branch is the repository's default branch, so the test/pr-<NNN>
  sandbox branch this repo's own docs suggest turns all of them into no-ops while the run
  stays green. And previous_yaml comes from a cache that exists only in production, so
  removal paths cannot be exercised here at all — better to say so than to imply coverage.
• Telling a parser bug from an environment limitation by replaying the request by hand.
  That is how I established that feat(github): add merge_queue convenience syntax for rulesets #119's merge queue rejection is GitHub refusing the rule
  type on a user-owned repo, not a bad payload — the same hand-written request fails
  identically.
• Reset, then propose. Draft a comment and wait for approval; recommend a disposition
  and let the human decide; merge only on explicit approval, with permission, after
  confirming the tested SHA is still the PR head.

Testing

Documentation only — no parser code changes. poetry run pytest (145 passed) and
poetry run pre-commit run --all-files are green. The CLAUDE.md / GEMINI.md symlinks
to AGENTS.md are untouched.

The procedure itself is the tested artefact: it is the written-down form of the runs
linked from #81, #84, #101, #111, #116, #119, #120 and #122.

Worth a maintainer's view

The .claude/skills/ location is the convention Claude Code looks for. If you would
rather this lived somewhere tool-neutral — docs/, with a pointer from .claude/ — say
so and I will move it.

🤖 Generated with Claude Code