Repository navigation
fix(sandbox): bound every docker invocation and stop an orphaned exec - #480
Merged
Merged
Conversation
The single docker chokepoint called `spawnSync("docker", args, ...)` with NO
timeout, and `execInSandbox` widened only `maxBuffer` before waiting
synchronously. So an in-container `sleep infinity`, a wedged daemon, or a
stalled build had no deadline, no cancellation path, and no cleanup escalation:
the host process blocked until something external killed it, and untrusted
repository code could deterministically wedge `sandbox exec`.
ca-sandbox is the driver that clones UNTRUSTED repositories into containers. A
hang it cannot escape is a control failure, not an inconvenience.
BOUNDED. Every invocation now carries a finite per-operation deadline, chosen
from the docker subcommand and layered into the ONE runner so `extra` can still
widen maxBuffer but cannot remove the deadline. The map is deliberately generous
where the work is genuinely long -- a cold `build` or `pull` over a slow network,
and an `exec` running the user's own build inside the box, are all legitimately
slow. The bound exists so that "slow" cannot become "never", not to second-guess
how long real work takes.
TYPED. A deadline kill returns `timedOut: true` with exit 124 and a message
naming the deadline. spawnSync signals it through `error`/`signal`, and a plain
`{code: 1}` is what an ordinary docker failure looks like -- a caller that
cannot tell them apart cannot escalate, which is the entire point.
ESCALATED. The deadline kills the docker CLIENT; the process it started INSIDE
the container keeps running, holding the box open. A timed-out exec now reaches
back in and stops the container it targeted:
* ONLY on a timeout. Escalation stops a container, and doing that on an
ordinary non-zero exit would destroy a working box every time a command
returned 1.
* With `--time 0`, not a grace window. The graceful window is what the
deadline already spent -- and measured, a `stop --time 5` under a 4s runner
deadline never completes, so the "cleanup" times out and the container
survives it. That was a real finding from the docker-backed test, not a
theoretical one.
* A failed escalation is reported but never masks the original timeout. A
wedged daemon is exactly the case where the cleanup cannot succeed, and the
timeout is the more important fact.
Tests: 9 in deadline.test.ts, in two genuinely different halves.
Contract (no daemon): every operation has a finite declared deadline; the
deadline is actually HANDED to spawnSync rather than merely exported and
asserted; a timeout is typed; an ordinary failure is NOT typed as one; a
timed-out exec escalates; a merely-failed exec does not; and a failed
escalation still returns the timeout.
Real daemon (docker-gated under #406's discipline, new `deadline` layer, which
check_sandbox_docker_layers.py now discovers and requires): a genuinely
hanging `sleep 600` exec is killed at its deadline and leaves NOTHING running
in the container. That is the assertion the argv tests cannot make.
Verified red first: 4 contract failures plus the real-docker case hanging the
worker. Green after: 9/9. Full ca-sandbox suite 248 passed across 20 files,
typecheck clean, sandbox.js rebuilt, and the docker-layer sentinel reports all
13 declared layers executed.
AC-1 every production docker invocation has a finite documented timeout and
returns a typed timeout result.
AC-2 a hanging `sandbox exec` terminates within its deadline and leaves no
targeted process running -- asserted against a real container.
AC-4 the hung-command case runs on every OS where Docker is available,
through the existing docker-gate.
AC-3 IS NOT DELIVERED HERE. "Cancellation can interrupt an active Docker call"
cannot be satisfied by spawnSync at all -- a synchronous spawn is by
construction uninterruptible, and honouring an AbortSignal mid-call requires
converting the whole docker boundary to async, rippling through build.ts,
claude-inside.ts, cli.ts, cp.ts, create.ts, destroy.ts, exec.ts, registry.ts
and run.ts. That is a separate change with its own risk, and pretending
otherwise would ship a deadline that looks cancellable and is not. Filed
separately rather than left implied.
ca-sandbox stays at 0.1.5: that version is unreleased (no `ca-sandbox-v0.1.5`
tag), so this change rides in it rather than inventing a version for an
unpublished line.
Refs #394.
This was referenced Jul 25, 2026
SUaDtL
added a commit
that referenced
this pull request
Jul 26, 2026
#480 gave every invocation through docker.ts a finite deadline, and its docblock claims "every docker invocation is bounded". That claim was FALSE, and this corrects it. build.ts never imports the shared runner - it takes only DOCKER_ENV - and had its own `run()` helper with no timeout at all. That helper is what executes `docker image inspect` and BOTH `docker build` calls, so `sandbox create` against a wedged builder or a stalled registry pull hung forever, while DOCKER_OPERATION_TIMEOUTS_MS.build and .pull sat as dead constants nothing in production could reach. Build and pull are the two longest operations in the driver. That made them the two most likely to hang, and the only ones with no escape. The deadline comes from docker.ts's per-operation map, not a second policy invented beside it: two maps drift the first time either moves. A non-docker probe (nixpacks, wsl.exe) takes the shared default, which is right - those are fast commands and a wedged one is just as unrecoverable. On expiry the child is SIGKILLed, because the deadline has already spent whatever grace was available, and the result carries the same typed `timedOut` flag docker.ts returns so a caller can tell a hang from an ordinary failure. A MUTATION CAUGHT A HOLE IN MY OWN TEST. Removing the `c.kill()` left the suite GREEN: the timer still settled the promise with timedOut:true, so every result-shaped assertion passed while the child kept running - which for a `docker build` means a leaked process still consuming the builder. The test now has the child print its own pid and asserts the process is actually dead afterwards, via the signal-0 liveness probe. With that, the kill mutation fails. Mutation results after: removing the kill -> 1 failed; reporting timedOut:false -> 1 failed. Full ca-sandbox suite 268 passed across 20 files; typecheck clean; sandbox.js rebuilt. ca-sandbox stays at 0.1.5, which is unreleased. Refs #394. Co-authored-by: SUaDtL <SUaDtL@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The defect
The single Docker chokepoint called
spawnSync("docker", args, ...)with no timeout, andexecInSandboxwidened onlymaxBufferbefore waiting synchronously. So an in-containersleep infinity, a wedged daemon, or a stalled build had no deadline, no cancellation path, and no cleanup escalation: the host process blocked until something external killed it, and untrusted repository code could deterministically wedgesandbox exec.ca-sandbox is the driver that clones untrusted repositories into containers. A hang it cannot escape is a control failure, not an inconvenience.
The fix, in three parts
Bounded. Every invocation carries a finite per-operation deadline, chosen from the docker subcommand and layered into the one runner — so
extracan still widenmaxBufferbut cannot remove the deadline. The map is deliberately generous where work is genuinely long (build,pull,exec) and short for a plain client call (inspect,ps). The bound exists so that "slow" cannot become "never", not to second-guess how long real work takes.Typed. A deadline kill returns
timedOut: truewith exit 124 and a message naming the deadline.spawnSyncsignals it througherror/signal, and a plain{code: 1}is what an ordinary docker failure looks like — a caller that cannot tell them apart cannot escalate, which is the entire point.Escalated. The deadline kills the docker client; the process it started inside the container keeps running, holding the box open. A timed-out exec now reaches back in and stops the container it targeted:
--time 0, not a grace window. The graceful window is what the deadline already spent — and measured, astop --time 5under a 4s runner deadline never completes, so the "cleanup" itself times out and the container survives it. That was a real finding from the Docker-backed test, not a theoretical one.Tests
9 tests in
deadline.test.ts, in two genuinely different halves.Contract (no daemon): every operation has a finite declared deadline; the deadline is actually handed to
spawnSyncrather than merely exported and asserted; a timeout is typed; an ordinary failure is not typed as one; a timed-out exec escalates; a merely-failed exec does not; and a failed escalation still returns the timeout.Real daemon (docker-gated under #406's discipline, via a new
deadlinelayer thatcheck_sandbox_docker_layers.pydiscovers and requires): a genuinely hangingsleep 600exec is killed at its deadline and leaves nothing running in the container. That is the assertion the argv tests cannot make.Verified red first: 4 contract failures, plus the real-Docker case hanging the worker outright. Green after: 9/9. Full ca-sandbox suite 248 passed across 20 files, typecheck clean,
sandbox.jsrebuilt, and the docker-layer sentinel reports all 13 declared layers executed.Acceptance criteria
sandbox execterminates within its deadline and leaves no targeted process runningAC-3 is not in this PR, deliberately.
spawnSyncis by construction uninterruptible: a deadline can be set on it, but nothing can interrupt the call in flight, because the calling thread is blocked inside the syscall. Honouring anAbortSignalrequires converting the whole docker boundary to async, rippling throughbuild.ts,claude-inside.ts,cli.ts,cp.ts,create.ts,destroy.ts,exec.ts,registry.tsandrun.tswhile preserving the CLI's synchronous exit-code contract. That is a separate change with its own risk, and shipping a deadline that looks cancellable and is not would be worse than the honest gap.Filed as #479 rather than left implied.
Version
ca-sandboxstays at 0.1.5 — that version is unreleased (noca-sandbox-v0.1.5tag), so this change rides in it rather than inventing a version for an unpublished line.Refs #394.