Skip to content

fix(sandbox): bound every docker invocation and stop an orphaned exec - #480

Merged
SUaDtL merged 1 commit into
mainfrom
fix/394-bound-docker-invocations
Jul 25, 2026
Merged

SUaDtL merged 1 commit into
mainfrom
fix/394-bound-docker-invocations

Conversation

@SUaDtL

@SUaDtL SUaDtL commented Jul 25, 2026

Copy link
Copy Markdown
Collaborator

The defect

The single Docker chokepoint called spawnSync("docker", args, ...) with no timeout, and execInSandbox widened only maxBuffer before waiting synchronously. So an in-container sleep infinity, a wedged daemon, or a stalled build had no deadline, no cancellation path, and no cleanup escalation: the host process blocked until something external killed it, and untrusted repository code could deterministically wedge sandbox exec.

ca-sandbox is the driver that clones untrusted repositories into containers. A hang it cannot escape is a control failure, not an inconvenience.

The fix, in three parts

Bounded. Every invocation carries a finite per-operation deadline, chosen from the docker subcommand and layered into the one runner — so extra can still widen maxBuffer but cannot remove the deadline. The map is deliberately generous where work is genuinely long (build, pull, exec) and short for a plain client call (inspect, ps). The bound exists so that "slow" cannot become "never", not to second-guess how long real work takes.

Typed. A deadline kill returns timedOut: true with exit 124 and a message naming the deadline. spawnSync signals it through error/signal, and a plain {code: 1} is what an ordinary docker failure looks like — a caller that cannot tell them apart cannot escalate, which is the entire point.

Escalated. The deadline kills the docker client; the process it started inside the container keeps running, holding the box open. A timed-out exec now reaches back in and stops the container it targeted:

  • Only on a timeout. Escalation stops a container; doing that on an ordinary non-zero exit would destroy a working box every time a command returned 1.
  • With --time 0, not a grace window. The graceful window is what the deadline already spent — and measured, a stop --time 5 under a 4s runner deadline never completes, so the "cleanup" itself times out and the container survives it. That was a real finding from the Docker-backed test, not a theoretical one.
  • A failed escalation never masks the original timeout. A wedged daemon is exactly when the cleanup cannot succeed, and the timeout is the more important fact.

Tests

9 tests in deadline.test.ts, in two genuinely different halves.

Contract (no daemon): every operation has a finite declared deadline; the deadline is actually handed to spawnSync rather than merely exported and asserted; a timeout is typed; an ordinary failure is not typed as one; a timed-out exec escalates; a merely-failed exec does not; and a failed escalation still returns the timeout.

Real daemon (docker-gated under #406's discipline, via a new deadline layer that check_sandbox_docker_layers.py discovers and requires): a genuinely hanging sleep 600 exec is killed at its deadline and leaves nothing running in the container. That is the assertion the argv tests cannot make.

Verified red first: 4 contract failures, plus the real-Docker case hanging the worker outright. Green after: 9/9. Full ca-sandbox suite 248 passed across 20 files, typecheck clean, sandbox.js rebuilt, and the docker-layer sentinel reports all 13 declared layers executed.

Acceptance criteria

AC status
AC-1 every production Docker invocation has a finite documented timeout and returns a typed timeout result ✅
AC-2 a hanging sandbox exec terminates within its deadline and leaves no targeted process running ✅ asserted against a real container
AC-4 covered on every OS where Docker is available ✅ through the existing docker-gate
AC-3 cancellation can interrupt an active Docker call ❌ not delivered — see below

AC-3 is not in this PR, deliberately. spawnSync is by construction uninterruptible: a deadline can be set on it, but nothing can interrupt the call in flight, because the calling thread is blocked inside the syscall. Honouring an AbortSignal requires converting the whole docker boundary to async, rippling through build.ts, claude-inside.ts, cli.ts, cp.ts, create.ts, destroy.ts, exec.ts, registry.ts and run.ts while preserving the CLI's synchronous exit-code contract. That is a separate change with its own risk, and shipping a deadline that looks cancellable and is not would be worse than the honest gap.

Filed as #479 rather than left implied.

Version

ca-sandbox stays at 0.1.5 — that version is unreleased (no ca-sandbox-v0.1.5 tag), so this change rides in it rather than inventing a version for an unpublished line.

Refs #394.

The single docker chokepoint called `spawnSync("docker", args, ...)` with NO
timeout, and `execInSandbox` widened only `maxBuffer` before waiting
synchronously. So an in-container `sleep infinity`, a wedged daemon, or a
stalled build had no deadline, no cancellation path, and no cleanup escalation:
the host process blocked until something external killed it, and untrusted
repository code could deterministically wedge `sandbox exec`.

ca-sandbox is the driver that clones UNTRUSTED repositories into containers. A
hang it cannot escape is a control failure, not an inconvenience.

BOUNDED. Every invocation now carries a finite per-operation deadline, chosen
from the docker subcommand and layered into the ONE runner so `extra` can still
widen maxBuffer but cannot remove the deadline. The map is deliberately generous
where the work is genuinely long -- a cold `build` or `pull` over a slow network,
and an `exec` running the user's own build inside the box, are all legitimately
slow. The bound exists so that "slow" cannot become "never", not to second-guess
how long real work takes.

TYPED. A deadline kill returns `timedOut: true` with exit 124 and a message
naming the deadline. spawnSync signals it through `error`/`signal`, and a plain
`{code: 1}` is what an ordinary docker failure looks like -- a caller that
cannot tell them apart cannot escalate, which is the entire point.

ESCALATED. The deadline kills the docker CLIENT; the process it started INSIDE
the container keeps running, holding the box open. A timed-out exec now reaches
back in and stops the container it targeted:

  * ONLY on a timeout. Escalation stops a container, and doing that on an
    ordinary non-zero exit would destroy a working box every time a command
    returned 1.
  * With `--time 0`, not a grace window. The graceful window is what the
    deadline already spent -- and measured, a `stop --time 5` under a 4s runner
    deadline never completes, so the "cleanup" times out and the container
    survives it. That was a real finding from the docker-backed test, not a
    theoretical one.
  * A failed escalation is reported but never masks the original timeout. A
    wedged daemon is exactly the case where the cleanup cannot succeed, and the
    timeout is the more important fact.

Tests: 9 in deadline.test.ts, in two genuinely different halves.

  Contract (no daemon): every operation has a finite declared deadline; the
  deadline is actually HANDED to spawnSync rather than merely exported and
  asserted; a timeout is typed; an ordinary failure is NOT typed as one; a
  timed-out exec escalates; a merely-failed exec does not; and a failed
  escalation still returns the timeout.

  Real daemon (docker-gated under #406's discipline, new `deadline` layer, which
  check_sandbox_docker_layers.py now discovers and requires): a genuinely
  hanging `sleep 600` exec is killed at its deadline and leaves NOTHING running
  in the container. That is the assertion the argv tests cannot make.

Verified red first: 4 contract failures plus the real-docker case hanging the
worker. Green after: 9/9. Full ca-sandbox suite 248 passed across 20 files,
typecheck clean, sandbox.js rebuilt, and the docker-layer sentinel reports all
13 declared layers executed.

  AC-1  every production docker invocation has a finite documented timeout and
        returns a typed timeout result.
  AC-2  a hanging `sandbox exec` terminates within its deadline and leaves no
        targeted process running -- asserted against a real container.
  AC-4  the hung-command case runs on every OS where Docker is available,
        through the existing docker-gate.

  AC-3 IS NOT DELIVERED HERE. "Cancellation can interrupt an active Docker call"
  cannot be satisfied by spawnSync at all -- a synchronous spawn is by
  construction uninterruptible, and honouring an AbortSignal mid-call requires
  converting the whole docker boundary to async, rippling through build.ts,
  claude-inside.ts, cli.ts, cp.ts, create.ts, destroy.ts, exec.ts, registry.ts
  and run.ts. That is a separate change with its own risk, and pretending
  otherwise would ship a deadline that looks cancellable and is not. Filed
  separately rather than left implied.

ca-sandbox stays at 0.1.5: that version is unreleased (no `ca-sandbox-v0.1.5`
tag), so this change rides in it rather than inventing a version for an
unpublished line.

Refs #394.
@SUaDtL
SUaDtL merged commit 32b8370 into main Jul 25, 2026
32 checks passed
@SUaDtL
SUaDtL deleted the fix/394-bound-docker-invocations branch July 25, 2026 23:40
SUaDtL added a commit that referenced this pull request Jul 26, 2026
#480 gave every invocation through docker.ts a finite deadline, and its docblock
claims "every docker invocation is bounded". That claim was FALSE, and this
corrects it.

build.ts never imports the shared runner - it takes only DOCKER_ENV - and had
its own `run()` helper with no timeout at all. That helper is what executes
`docker image inspect` and BOTH `docker build` calls, so `sandbox create`
against a wedged builder or a stalled registry pull hung forever, while
DOCKER_OPERATION_TIMEOUTS_MS.build and .pull sat as dead constants nothing in
production could reach.

Build and pull are the two longest operations in the driver. That made them the
two most likely to hang, and the only ones with no escape.

The deadline comes from docker.ts's per-operation map, not a second policy
invented beside it: two maps drift the first time either moves. A non-docker
probe (nixpacks, wsl.exe) takes the shared default, which is right - those are
fast commands and a wedged one is just as unrecoverable. On expiry the child is
SIGKILLed, because the deadline has already spent whatever grace was available,
and the result carries the same typed `timedOut` flag docker.ts returns so a
caller can tell a hang from an ordinary failure.

A MUTATION CAUGHT A HOLE IN MY OWN TEST. Removing the `c.kill()` left the suite
GREEN: the timer still settled the promise with timedOut:true, so every
result-shaped assertion passed while the child kept running - which for a
`docker build` means a leaked process still consuming the builder. The test now
has the child print its own pid and asserts the process is actually dead
afterwards, via the signal-0 liveness probe. With that, the kill mutation fails.

Mutation results after: removing the kill -> 1 failed; reporting timedOut:false
-> 1 failed.

Full ca-sandbox suite 268 passed across 20 files; typecheck clean; sandbox.js
rebuilt. ca-sandbox stays at 0.1.5, which is unreleased.

Refs #394.

Co-authored-by: SUaDtL <SUaDtL@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant