Repository navigation
fix(sandcastle): Open one pull request per issue instead of merging locally - #236
Conversation
…ocally Port the host-side design from mpaulosky/Articles#308 to fix three Claude Review findings on #228: - An issue counts as done only when the implementer signals completion, and its branch is published whenever it holds commits origin/main lacks, not only when this run made commits. - The merger is gone. Each approved branch is pushed and gets its own draft PR with "Fixes #n", so nothing lands on main and no issue is closed outside the normal PR process. - The host reads issues with its own gh auth and keeps only issues and comments by owners, members and collaborators. The sandbox gets no GitHub token, and main.mts refuses to start if .sandcastle/.env sets one. It avoids the problems Articles' own review found in that design: the host pushes with hooks off (the pre-push hook is agent-editable), merges origin/main before the check, ignores fork PRs, drops repeated planner picks, trims the PR title, and re-checks and publishes the exact commit that passed when the reviewer moves HEAD. A test pins that only host-made values go inside prompt shell blocks. test:sandcastle now type-checks with tsc first, and engines requires Node 22.18 or later for type stripping. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in #231's check.sh. The host now runs it, so check.sh, CODING_STANDARDS.md and CLAUDE.md no longer call it advisory, and the Docker-backed suites are left to CI. sandcastle-tests.sh installs the locked packages first, since the tests import @ai-hero/sandcastle and zod and the script type-checks with typescript. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Test Results Summary485 tests 485 ✅ 27s ⏱️ Results for commit 6803f7d. ♻️ This comment has been updated with latest results. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #236 +/- ##
=======================================
Coverage 85.63% 85.63%
=======================================
Files 75 75
Lines 1566 1566
Branches 186 186
=======================================
Hits 1341 1341
Misses 190 190
Partials 35 35 🚀 New features to boost your workflow:
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Claude Review
This PR replaces Sandcastle's old plan → implement → agent-merge loop with a new pipeline built around one PR per issue. The host now reads only trusted issues and comments, using its own gh auth, and passes them to the agents in their prompts. The sandbox gets no GitHub token, and main.mts refuses to start if .env sets one. Each issue is implemented, merged with origin/main, checked with check.sh, reviewed for a JSON verdict, and checked again if the reviewer moved HEAD. The host then pushes the exact checked commit, with hooks off, and opens a draft PR that says "Fixes #n". The merger agent and its gh issue close are removed, and CI now installs with pnpm and type-checks .sandcastle with tsc.
The design is a clear improvement and is well tested at the unit level. Verdict: not ready to merge as is. Two gate gaps undercut the PR's own claims:
- The check is the agents' own script. The host runs the branch's
.sandcastle/check.shandneeds-docker.sh, which the agents can edit. - Hooks are only one way host git runs commands. The host now runs git in a repository the sandbox can probably write to, and git config can run commands too, not only hooks.
There are also two loop-level problems:
- Dependent issues get built too early. Later rounds build issues whose blockers are only in review, not on main.
- Failed issues are retried every round. An issue that fails is rebuilt, with a new failure comment each time, whenever any other issue publishes.
Caveat: the @ai-hero/sandcastle source isn't installed here, so I couldn't check two things against it. One is whether the main .git is mounted writable into the sandbox. The other is how createSandbox bases a branch that exists only on origin. Those two findings depend on that behaviour.
…leaks Address the review threads on #236: - Refuse host git and gh once .git/config has changed during the run. Sandcastle mounts the clone's .git writable into each sandbox, and git runs commands named there (credential helpers, sshCommand, fsmonitor, drivers) or follows rewritten remotes. gh now names the repository read before any sandbox started instead of reading the remotes. - Remove the sandbox's secrets (.sandcastle/.env values and token shapes) from every issue comment and PR body the host writes. - Rewrite "fixes #n"-style references in the reviewer's summary so only the host's own Fixes line closes an issue. - Flag in the PR body a branch that changes the files deciding what check.sh runs or skips. - Give the planner the in-review issues as open blockers, and don't rebuild an issue that already stopped in this run. - Create the local branch from origin's when only origin has it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Automated release blog posts for #236, opened by the release workflow. It holds every Release whose post isn't on main yet, rebuilt from main on each run. The [skip-release] title marker keeps its merge from starting another release. Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Claude Review
This PR changes the Sandcastle loop from "an agent merges every branch into main and closes the issues" to one draft PR per issue. Only the host talks to GitHub now. It reads issues with its own gh auth and drops anything written by people other than the owner, members and collaborators. The sandbox gets no GitHub token. Each issue is implemented, merged with origin/main and checked with check.sh in the sandbox, then reviewed and checked again if the reviewer moved HEAD. The host pushes exactly the commit that passed, with hooks off and a guard that refuses git once .git/config changes, and opens a draft PR that says "Fixes #n". The split into lib/ modules is well tested and the merge gating itself looks right: unfinished, rejected, conflicting or failing work is never pushed, and drafts are skipped by pr-automerge.
Verdict: needs changes before merging. The host-protection work is the point of this PR, but it misses three cases:
- Sandcastle's own host git calls (worktree add and remove) still run hooks and config that agents can edit.
- A detected config tamper doesn't stop the run.
- Pushed commits aren't checked for the sandbox's secrets.
The rest:
- The PR warning about changes to the check's own files misses most of the ways a test project can be skipped.
- A closing keyword in the issue title can close an unrelated issue.
- Issues that stopped earlier in the run are hidden from the planner as blockers.
What I couldn't check here: the @ai-hero/sandcastle source isn't installed in this checkout, so the first finding assumes createSandbox/close use plain git worktree add/remove on the host, inheriting the process environment. I didn't run tsc or the tests.
…#242) ## Why #236 replaced Sandcastle's local merger with one draft PR per issue. PR Auto-Merge merged it after its review cap, and Claude Review's pass over its final head then opened six threads. All six hold up: - **Agent-edited hooks could still run on the host.** #236 turned hooks off only for the host's own git. Sandcastle runs `git worktree add` on the host too. That runs `post-checkout` from the new worktree, which is the branch's copy of `.github/hooks`, and a branch can be one the agents wrote in an earlier run. - **Detected config tampering didn't stop the run.** A `GitConfigChangedError` was logged as one failed issue. `sandbox.close()` still ran Sandcastle's git under the tampered config, and the run could end with "All done." - **Pushed commits were never checked for the sandbox's secrets.** The Claude token or API key is in the sandbox, and an agent could commit it in a log, fixture or snapshot. - **The PR warning missed most ways to skip a test project.** Only a `RequiresDocker` change was flagged. A Docker-backed package, `IsTestProject=false`, a `ProjectReference`, or a deleted test project slipped through. The comparison also used `origin/main`, a ref the agents can move. - **Issue titles could close other issues.** A title like "Fix #12 regression" went unchanged into the PR title (the squash subject) and the body. - **Stopped issues vanished from the planner.** An issue that stopped earlier in a run was in neither of the planner's lists, so the issues it blocks looked unblocked. ## What changed - **`lib/shell.mts`, `main.mts`:** `protectHostGit()` runs first. It adds `core.hooksPath=/dev/null` and `core.fsmonitor=false` through `GIT_CONFIG_COUNT`/`GIT_CONFIG_KEY_n`/`GIT_CONFIG_VALUE_n`, after any entries already there. Every git process on the host inherits these, Sandcastle's included. They don't reach the sandbox: Sandcastle passes it only what `.sandcastle/.env` names. - **`lib/build.mts`, `main.mts`:** if a pipeline hits `GitConfigChangedError`, it leaves its sandbox open and rethrows. `main.mts` rethrows the first such error after the round, so the run exits non-zero with the warning. - **`lib/scan.mts`, `lib/build.mts`, `lib/sandbox-env.mts`:** before the push, the host scans what it would publish, from main's commit to the checked commit. That is every commit message and every added line, from `git log -p --cc -U0`, so merge commits are read too. Context and removed lines are left out, so main's fake test tokens don't stop unrelated branches. The scan looks for the values `.sandcastle/.env` gives the sandbox and for Claude and GitHub token shapes. A match stops the issue as `secret-in-commits`, with a comment that doesn't quote it. - **`lib/check.mts`:** the warning now covers MSBuild lines matching `RequiresDocker|IsTestProject|Testcontainers|Aspire.Hosting.Testing|Playwright|ProjectReference`, and files deleted or renamed away under `tests/`. - **`lib/branches.mts`, `lib/build.mts`:** `fetchMain()` returns main's commit as `git ls-remote origin` reports it. The merge, `commitsAhead`, the warning and the secret scan all compare with that commit id, not with `refs/remotes/origin/main`. - **`lib/publish.mts`:** `withoutClosingKeywords` now also rewrites the issue title, in the PR title and body, and full `github.com/.../issues/n` and `/pull/n` URLs. - **`lib/prompts.mts`, `plan-prompt.md`:** the planner's second list is now the held-back issues (`{{HELD_BACK_JSON}}`), both in review and stopped earlier in the run, and they count as blockers. ## Verification - `pnpm run test:sandcastle`: the type check is clean and 99 of 99 tests pass. The new tests run against real temp repositories: a `git worktree add` with `protectHostGit` doesn't run a `post-checkout` hook, and the same command without it does. `checkFileChanges` flags a Testcontainers reference, `IsTestProject=false`, a deleted test project, a renamed test file and `check.sh`, but not an ordinary `src/` package change. `publishedText` sees a secret added in a merge resolution, but not a fixture token in context. Other new tests cover the secret stop, the untouched sandbox on config tampering, and closing references in titles and URLs. - The pre-push gate passed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: T <t@e.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Why
This addresses the Claude Review findings on the merged #228, which committed the Sandcastle setup. Three of them are design problems with the old plan, implement, merge loop:
commits.length > 0for this run. A re-planned issue whose branch already held finished work made no new commits, so it was never merged. An implementer that ran out of iterations without<promise>COMPLETE</promise>still had its commits merged and its issue closed.main, where the hooks refuse commits and pushes) and then rangh issue close, with no PR, checks or review.GH_TOKEN. Every comment and issue body on aSandcastleissue went into the planner's prompt, and the implementer fetched the issue withgh. On a public repo anyone can comment, so injected instructions could steer agents that had an Issues-write token.What changed
This ports the host-side design from mpaulosky/Articles#308, without the problems that design's own review found.
merge-prompt.mdis gone.lib/build.mtsruns each issue in its own sandbox: implement, mergeorigin/main, run.sandcastle/check.sh, review, then push and open a draft PR whose body endsFixes #n. The issue closes when that PR merges through the normal checks. Anything that stops (unfinished, merge conflict, failed check, rejection, failed push) gets a comment on the issue and isn't pushed.git rev-list --count origin/main..HEADis above 0, so earlier finished work still gets its PR.<verdict>{"approved", "summary"}</verdict>(lib/verdict.mts). A missing or malformed verdict counts as a rejection.lib/github.mtsreads issues with the host'sghauth and keeps only issues opened by, and comments from, anOWNER,MEMBERorCOLLABORATOR. The agents get the issue through prompt arguments..env.exampleno longer hasGH_TOKEN, andmain.mtsrefuses to start if.sandcastle/.envsetsGH_TOKENorGITHUB_TOKEN, since Sandcastle passes every key in that file into the sandbox.-c core.hooksPath=/dev/null.core.hooksPathis.github/hooks, a path inside the worktree the agents edit, so running its pre-push hook on the host would run their code outside the sandbox. CI runs the full suite, the Docker-backed tests included.origin/mainin the sandbox before the check. On a conflict the merge is aborted and the issue gets a comment.isCrossRepository) neither hold an issue back nor pass for its PR.lib/plan.mts).Add search .becomesfeat: Add search, with no trailing space..gitinto each sandbox, writable, so an agent could write commands into.git/configfor the host's git to run.main.mtsrecordsgit config --local --listbefore any sandbox starts, and the host refuses to run git orghonce it has changed (lib/shell.mts).ghtakes--repofrom the name read at the start, not from the remotes..sandcastle/.envgives it, and anything shaped like a Claude or GitHub token.Fixes #nline closes an issue.check.shruns or skips (check.sh,needs-docker.sh,discover_tests.py,package.json, or aRequiresDockerflag) is named in a WARNING at the top of its PR..sandcastle/check.sh(from chore(sandcastle): Run a Docker-free check.sh in the sandbox #231) is now enforced. The host runs it in the sandbox throughlib/check.mts, and a passing check over uncommitted changes still fails.check.sh,CODING_STANDARDS.mdandCLAUDE.mdno longer call it advisory, and they say CI runs the Docker-backed suites. The implement and review prompts keep chore(sandcastle): Run a Docker-free check.sh in the sandbox #231's Docker guidance.test:sandcastlerunstsc --noEmit -p .sandcastlebefore the tests (new.sandcastle/tsconfig.json, withtypescriptand@types/nodeas dev dependencies).enginesrequires Node 22.18 or later, which strips the.mtstypes. The tests now import@ai-hero/sandcastleandzod, so.github/ci/sandcastle-tests.shrunspnpm install --frozen-lockfilefirst. It uses pnpm from the PATH, or corepack's when there isn't one.Findings checked but not changed here
branches.mts:45, bug issues onhotfix/: already fixed by fix(sandcastle): Name bug branches fix/ instead of hotfix/ #232.test:sandcastlenot run in CI: chore(sandcastle): Run a Docker-free check.sh in the sandbox #231 runs it from the gate and from CI's Build Solution job, through.github/ci/sandcastle-tests.sh.@ai-hero/sandcastle0.12.0,substitutePromptArgsmarks the template's own!`…`blocks with\x01before it fills in{{KEY}}arguments and strips that marker from argument values.preprocessPromptruns only marked blocks, so a block written in an issue title or body isn't run. A new test inprompts.test.mtschecks that only host-made values (BRANCH,BASE_BRANCH,TASK_ID) appear inside the prompts' shell blocks.Verification
bash .github/ci/sandcastle-tests.sh "": the type check is clean and 89 of 89 tests pass, including chore(sandcastle): Run a Docker-free check.sh in the sandbox #231's. New tests cover publishing earlier finished work, refusing unfinished work, merge conflicts, a reviewer that amends or resets, fork PRs, repeated picks, the trailing space in titles, the git config guard (against a real temp repo), redaction, closing keywords and the check-file warning.🤖 Generated with Claude Code