Skip to content

fix(sandcastle): Close the gaps Claude Review found after #236 merged - #242

Merged
mpaulosky merged 2 commits into
mainfrom
chore/sandcastle-review-followups
Oct 9, 2026
Merged

mpaulosky merged 2 commits into
mainfrom
chore/sandcastle-review-followups

Conversation

@mpaulosky

@mpaulosky mpaulosky commented Oct 9, 2026 •

Copy link
Copy Markdown
Owner

Why

#236 replaced Sandcastle's local merger with one draft PR per issue. PR Auto-Merge merged it after its review cap, and Claude Review's pass over its final head then opened six threads. All six hold up:

  • Agent-edited hooks could still run on the host. fix(sandcastle): Open one pull request per issue instead of merging locally #236 turned hooks off only for the host's own git. Sandcastle runs git worktree add on the host too. That runs post-checkout from the new worktree, which is the branch's copy of .github/hooks, and a branch can be one the agents wrote in an earlier run.
  • Detected config tampering didn't stop the run. A GitConfigChangedError was logged as one failed issue. sandbox.close() still ran Sandcastle's git under the tampered config, and the run could end with "All done."
  • Pushed commits were never checked for the sandbox's secrets. The Claude token or API key is in the sandbox, and an agent could commit it in a log, fixture or snapshot.
  • The PR warning missed most ways to skip a test project. Only a RequiresDocker change was flagged. A Docker-backed package, IsTestProject=false, a ProjectReference, or a deleted test project slipped through. The comparison also used origin/main, a ref the agents can move.
  • Issue titles could close other issues. A title like "Fix Bump Radzen.Blazor from 5.3.5 to 5.5.5 #12 regression" went unchanged into the PR title (the squash subject) and the body.
  • Stopped issues vanished from the planner. An issue that stopped earlier in a run was in neither of the planner's lists, so the issues it blocks looked unblocked.

What changed

  • lib/shell.mts, main.mts: protectHostGit() runs first. It adds core.hooksPath=/dev/null and core.fsmonitor=false through GIT_CONFIG_COUNT/GIT_CONFIG_KEY_n/GIT_CONFIG_VALUE_n, after any entries already there. Every git process on the host inherits these, Sandcastle's included. They don't reach the sandbox: Sandcastle passes it only what .sandcastle/.env names.
  • lib/build.mts, main.mts: if a pipeline hits GitConfigChangedError, it leaves its sandbox open and rethrows. main.mts rethrows the first such error after the round, so the run exits non-zero with the warning.
  • lib/scan.mts, lib/build.mts, lib/sandbox-env.mts: before the push, the host scans what it would publish, from main's commit to the checked commit. That is every commit message and every added line, from git log -p --cc -U0, so merge commits are read too. Context and removed lines are left out, so main's fake test tokens don't stop unrelated branches. The scan looks for the values .sandcastle/.env gives the sandbox and for Claude and GitHub token shapes. A match stops the issue as secret-in-commits, with a comment that doesn't quote it.
  • lib/check.mts: the warning now covers MSBuild lines matching RequiresDocker|IsTestProject|Testcontainers|Aspire.Hosting.Testing|Playwright|ProjectReference, and files deleted or renamed away under tests/.
  • lib/branches.mts, lib/build.mts: fetchMain() returns main's commit as git ls-remote origin reports it. The merge, commitsAhead, the warning and the secret scan all compare with that commit id, not with refs/remotes/origin/main.
  • lib/publish.mts: withoutClosingKeywords now also rewrites the issue title, in the PR title and body, and full github.com/.../issues/n and /pull/n URLs.
  • lib/prompts.mts, plan-prompt.md: the planner's second list is now the held-back issues ({{HELD_BACK_JSON}}), both in review and stopped earlier in the run, and they count as blockers.

Verification

  • pnpm run test:sandcastle: the type check is clean and 99 of 99 tests pass. The new tests run against real temp repositories: a git worktree add with protectHostGit doesn't run a post-checkout hook, and the same command without it does. checkFileChanges flags a Testcontainers reference, IsTestProject=false, a deleted test project, a renamed test file and check.sh, but not an ordinary src/ package change. publishedText sees a secret added in a merge resolution, but not a fixture token in context. Other new tests cover the secret stop, the untouched sandbox on config tampering, and closing references in titles and URLs.
  • The pre-push gate passed.

🤖 Generated with Claude Code

- Turn hooks and fsmonitor off for every git on the host through
  GIT_CONFIG_*, so Sandcastle's own `git worktree add` can't run a
  branch's agent-written post-checkout hook. A test adds a worktree
  with and without it.
- Stop the whole run when an agent changed .git/config: the pipeline
  leaves its sandbox open instead of letting Sandcastle remove the
  worktree under that config, and main.mts rethrows the error.
- Refuse to push commits whose messages or diffs hold a sandbox secret
  or a token shape, without quoting it on the issue.
- Widen the PR warning to IsTestProject, Docker-backed packages,
  ProjectReference and deleted tests/ files, and compare with main's
  commit as origin reports it, not the agent-writable origin/main ref.
- Rewrite closing references in the issue title (PR title and body)
  and in full issue URLs.
- Show the planner the issues that stopped earlier in the run as
  blockers, alongside the ones in review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mpaulosky mpaulosky added the review:claude Claude reviews each head, a backup to Copilot's review label Oct 9, 2026
@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Test Results Summary

485 tests  ±0   485 ✅ ±0   26s ⏱️ +2s
  9 suites ±0     0 💤 ±0 
  9 files   ±0     0 ❌ ±0 

Results for commit aafd44d. ± Comparison against base commit 2e96db0.

♻️ This comment has been updated with latest results.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Review

This PR tightens Sandcastle's publish path. Main is now pinned to the commit git ls-remote reports, not the agent-writable origin/main ref. Hooks and fsmonitor are turned off through GIT_CONFIG_* for every git the host runs, Sandcastle's own included. The branch's commits are scanned for sandbox secrets before the push. If an agent changes .git/config, the run stops and the sandbox is left open. checkFileChanges also flags more MSBuild lines and deleted test files. Closing references are now rewritten in PR titles and in URL form. Issues that stopped earlier in the run are held back as blockers for the planner.

I traced the git-config path. It holds even though the inner publish catch swallows a GitConfigChangedError: stop() → commentOnIssue → gh re-asserts the config and throws again, so the error reaches the outer catch and nothing is posted. Pinning base by commit id, threading base through commitsAhead/checkFileChanges, and the planner and prompt changes all look right. Squash-only merging means agents' commit messages can't close issues, so rewriting only the title and body is enough.

Verdict: mergeable once the secret scan covers merge commits. That is a real hole in a gate meant to stop a public leak, and the fix is one flag. The other two are smaller gaps worth fixing or knowingly accepting. I couldn't run git here, so the git behaviour behind these findings is from git's documented defaults:

  • git log -p shows no diff for merge commits.
  • git diff detects renames by default.
  • Pathspecs are applied before rename detection.

I also couldn't check the Sandcastle source (node_modules isn't installed). The new comment in lib/shell.mts says Sandcastle passes the sandbox only what .sandcastle/.env names, so GIT_CONFIG_* never reaches it; I've taken that as given.

Comment thread .sandcastle/lib/build.mts Outdated
Comment thread .sandcastle/lib/check.mts Outdated
Comment thread .sandcastle/lib/build.mts Outdated
@codecov

codecov Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.63%. Comparing base (2e96db0) to head (aafd44d).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main     #242   +/-   ##
=======================================
  Coverage   85.63%   85.63%           
=======================================
  Files          75       75           
  Lines        1566     1566           
  Branches      186      186           
=======================================
  Hits         1341     1341           
  Misses        190      190           
  Partials       35       35           
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…ests

- The secret scan reads merge commits (--cc), so content written while
  resolving or amending a merge is checked, and scans only commit
  messages and added lines (-U0), so main's fake tokens in context or
  removed lines don't stop unrelated branches.
- A file renamed away under tests/ is flagged like a deletion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mpaulosky
mpaulosky merged commit 39812be into main Oct 9, 2026
37 checks passed
@mpaulosky
mpaulosky deleted the chore/sandcastle-review-followups branch October 9, 2026 01:00

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Review

This PR tightens the Sandcastle host in six ways. The host now merges and compares against main's commit ID from ls-remote rather than origin/main, which agents can move. Every pushed commit is scanned for sandbox secrets first. The check-file warning now also covers deleted tests/ files and more MSBuild lines. Hooks and fsmonitor are switched off for every git on the host through GIT_CONFIG_*. Closing keywords in issue titles and full GitHub URLs are rewritten. Issues that stopped earlier in the run are held back as blockers, and the run aborts when .git/config changes.

Verdict: the direction is right, but fix the first finding before merging. The handling for a changed git config misses its main case. The publish step's catch-all handler turns GitConfigChangedError into publish-failed. And a pipeline only skips close() if it saw the error itself. So Sandcastle's own git worktree remove can still run on the host under a config an agent wrote, which is exactly what this change is meant to prevent.

The secret scan has smaller gaps:

  • It drops added lines whose content starts with ++ .
  • It finds nothing if the host's git config forces colour.
  • Replace refs (refs/replace/*) in the agent-writable .git can hide content from the host's git log/git diff.

fetchMain's ls-remote pattern matches by suffix, so another branch can be taken for main. The check-file warning can still be avoided with multi-line edits to MSBuild files.

What I relied on without being able to check here: Sandcastle isn't installed in this checkout, so I couldn't read its close() or confirm that it doesn't pass the host environment into the container. Three points come from git's documented behaviour: the -G regex is extended, ls-remote patterns match on a ref's suffix, and pack-objects ignores replace refs.

Comment thread .sandcastle/lib/build.mts
Comment thread .sandcastle/lib/scan.mts
Comment thread .sandcastle/lib/scan.mts
Comment thread .sandcastle/lib/shell.mts
Comment thread .sandcastle/lib/branches.mts
Comment thread .sandcastle/lib/check.mts
mpaulosky added a commit that referenced this pull request Oct 9, 2026
Automated release blog posts for #242, opened by the release workflow.
It holds every Release whose post isn't on main yet, rebuilt from main
on each run. The [skip-release] title marker keeps its merge from
starting another release.

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@mpaulosky mpaulosky removed the review:claude Claude reviews each head, a backup to Copilot's review label Oct 9, 2026
mpaulosky added a commit that referenced this pull request Oct 9, 2026
…ecret scan (#244)

## Why

PR Auto-Merge merged #242 before Claude Review finished with its final
head. That review opened six threads on the hardening #242 added, and
all six are real:

- **A config change found while publishing was swallowed.** It turned
into a `publish-failed` comment, and `sandbox.close()` still ran
Sandcastle's git under the changed config. That also happened when
another pipeline's agent changed the config after this pipeline's last
host git call.
- **The secret scan could miss added lines.** It skipped every line
starting with `+++ `, which also drops an added line whose content
starts with `++ `. It didn't scan file paths. With a user's
`color.diff=always`, it saw no added lines at all.
- **Replace refs could hide content.** `refs/replace` sits in the same
writable `.git`, and `git log`/`diff`/`rev-list` honour it, but a push
sends the real objects.
- **`ls-remote origin refs/heads/main` could pick another branch.** It
also matches a branch like `feature/refs/heads/main`, which sorts first.
- **The check-file warning could still be dodged.** `discover_tests.py`
and `needs-docker.sh` read whole XML documents. A `<!--`/`-->` pair or a
`Condition` on its own line can drop a test project without changing any
line the pattern matches.

## What changed

- **`lib/build.mts`, `lib/shell.mts`:** the publish step rethrows
`GitConfigChangedError`. `finally` checks the config again before it
closes the sandbox, and throws `GitConfigChangedError` if it changed,
whatever the pipeline returned (published, rejected, nothing to
publish). The throw replaces the return value, so `main.mts` stops the
run before Sandcastle's own git runs under the changed config.
- **`lib/scan.mts`:** `addedLines` reads the patch with state. File
headers run until a hunk header, and the number of `@`s gives the number
of parent columns. A line is added when one of its columns is `+` and
none is `-`, so an added `++ token` line is scanned. Header lines, and
so the paths, are scanned too. `git log` gets `--no-color`. The commits
are read with `--format=raw` rather than `%B`, so the author and
committer names and emails, `encoding`, `gpgsig` and any extra header,
which the push publishes too, are scanned.
- **`lib/shell.mts`:** `HOST_GIT_CONFIG` adds
`core.useReplaceRefs=false` for every git on the host, Sandcastle's
included. The `git()` helper also passes `-c core.useReplaceRefs=false`
itself, so the scan and the check-file warning hold up without
`protectHostGit`.
- **`lib/branches.mts`:** `mainCommit` keeps only the exact
`refs/heads/main` line, and throws unless there is exactly one.
- **`lib/check.mts`, `CODING_STANDARDS.md`:** every change is named for
the MSBuild files under `tests/` and every `Directory.Build.*` file at
any depth. The pathspecs carry explicit `:(glob)` magic
(`:(glob)**/Directory.Build.*` and so on), so they match nested files
under a host's `GIT_GLOB_PATHSPECS=1` too. The keyword pattern still
covers MSBuild files elsewhere.

## Verification

- `pnpm run test:sandcastle`: the type check is clean and 109 of 109
tests pass. New tests cover a config change found during publish and one
found at the end of a pipeline that published, was rejected or had
nothing to publish (each rejects with `GitConfigChangedError` and leaves
the sandbox open), tokens in an author name and a committer email, a
replace ref hiding a token and a check-file change (with and without
`protectHostGit`, plus a control), nested `tests/Directory.Build.props`
and `src/X/Directory.Build.targets` changes with and without
`GIT_GLOB_PATHSPECS=1`, an added line that looks like a header, a scan
with `color.ui=always`, an ls-remote result with a decoy branch, and a
test project commented out on lines of their own plus a
`Directory.Build.props` change.
- The pre-push gate passed.

Its final Claude Review round opened four threads; all four are fixed
here, and `review:claude` is off.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: T <t@e.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant