feat(agents): optionally run CLI agents in a container with a scoped cluster identity - #72
feat(agents): optionally run CLI agents in a container with a scoped cluster identity#72geojaz wants to merge 5 commits into
Conversation
…r identity
The CLI agent harnesses invoke the agent binary as a plain subprocess inheriting
the parent environment, so a shell tool under `--approval-mode yolo` has the
operator's entire filesystem. In observed runs an agent read another task's seed
and verification scripts off disk, searched the home directory for its own
task's fixtures by namespace name, and copied an unrelated archive out of
~/Downloads before running rm -rf. The agent's own workspace sandbox does not
help: it guards the native file tools, not the shell.
That is two problems. Task definitions, seed scripts and scoring rubrics living
on the same filesystem the agent can read makes results unverifiable; the rest
is an operator-safety hazard.
Adds an opt-in containerised execution path (BENCH_AGENT_SANDBOX=docker) with
two independent boundaries:
* The CONTAINER removes the host filesystem and the operator's environment.
The mount set is deliberately short, and what is absent matters most: no
repository, no $HOME (HOME is repointed at the workspace so a bare ~ cannot
resolve to the operator's profile), no Docker socket, and no Application
Default Credentials. Model credentials cross by value as explicit -e flags
from the harness's already-resolved config, never by mounting a config dir
or inheriting whatever the operator exported.
* A SCOPED TOKEN decides what the agent may do to the cluster. If a task seeds
a `bench-agent` ServiceAccount, a short-lived token is minted for it, making
the agent's identity part of the topology the task declares. Absent one, it
falls back to the operator's credential and logs a warning, since that is a
much larger grant than most tasks intend.
Neither boundary closes a third channel: an agent that can read a cluster can
read anything a task put IN that cluster, and a task needing the agent to
inspect a workload cannot use RBAC to hide that workload's definition. Answer
material must not be seeded into the cluster; the module documents this rather
than implying the sandbox covers it.
Networking: kind creates a Docker network of the same name, so the container
joins it and reaches the API server at https://<cluster>-control-plane:6443. The
kubeconfig kind writes points at 127.0.0.1, which is meaningless in a container,
so it is rewritten per run. kind's API server certificate covers the
control-plane node name, so TLS still verifies without --insecure-skip-tls-verify.
If a sandbox kubeconfig cannot be built the run is refused rather than silently
falling back to the host: a containment control that quietly degrades is worse
than none.
Includes hack/agent-sandbox.Dockerfile and hack/agent-credential.sh, which mints
a credential restricted to one API rather than reusing ADC.
Only the gemini harness is wired so far. openclaw needs the same treatment plus
a KUBECONFIG-ordering fix (it threads the path into its MCP config, which must
be the in-container path). antigravity ships a Mach-O binary and cannot run in a
Linux container without a Linux build.
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: geojaz The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Warning Review limit reached
Next review available in: 59 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (8)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
kubernetes-prow
Bot
added
the
cncf-cla: yes
|
Hi @geojaz. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
kubernetes-prow
Bot
added
needs-ok-to-test
…their record The gemini CLI agent discarded core.subprocess.run's captured stdout whenever the run timed out, so a turn that worked the whole way through (kubectl calls, generated parsers, a web search) left an empty trajectory once the process hit the deadline. subprocess already buffers whatever the child wrote before the kill; the agent now runs it through parse_stream_json instead of throwing it away, and records the timeout / recovered-trajectory outcome on AgentResult.metadata so the harness can tell a forensic partial run from a clean one. Signed-off-by: Eric Hole <ehole@onixnet.com>
_build_success_record hardcoded status: "success" regardless of whether the agent process actually completed, so an AgentResult.errored() run (timeout, non-zero exit, missing binary, SDK fault) still scored a normal-looking OutcomeScore made entirely of the task's do_nothing baseline plus a perfect safety axis. The record now reads agent_timeout / agent_error whenever the agent reported an error, mirroring how a "failed" record already opts out of scoring: _score excludes all three statuses from the metrics batch, so a degraded run can no longer produce a composite score that reads as if the agent had a fair turn. Signed-off-by: Eric Hole <ehole@onixnet.com>
A sandboxed gemini run's `docker run --rm` container survived a timeout: the harness only SIGKILLs the local docker client process, which never signals the daemon to stop the container itself, so a wedged agent kept burning model API calls for as long as it stayed up after the harness moved on. Each run now gets a container named deterministically from its own workspace dir, wrapped in a container_guard that kills it by name in a finally on every exit from the run (normal, exception, or timeout), and the harness sweeps stray containers from a prior crashed run at the start of each batch. Signed-off-by: Eric Hole <ehole@onixnet.com>
|
Pushed three follow-ups that belong with this change rather than in a separate PR, since they are all sandbox and agent-turn lifecycle:
|
kubernetes-prow
Bot
added
size/XXL
|
Noting a related follow-up that is deliberately not in this PR, plus one thing worth knowing about this branch. A fix that expands More generally, this branch has drifted from the line the rest of the verification work is on. It has no hold mode, and its |
janetkuo
added
ok-to-test
|
Note for reviewers: there is a staged follow-up commit for this PR that bind-mounts the harness scratch root into the sandbox container, read-write, at the same path inside and outside. That lets a task seed a host-side artifact (for example a bare git repo) that both the sandboxed agent and the host-side verifiers reach at one path. It depends on scratch_root() from #60, so I am holding it until #60 merges and this branch picks that up. Keep it in mind when reviewing the mount set here: the workspace-plus-kubeconfig-only posture in this diff will grow by one deliberate mount, and the owned-root guards in #60 are what make a read-write mount safe. |
2 participants
Agent transcripts showed CLI agents ranging well outside the task: reading files from
the operator's home directory, copying archives out of ~/Downloads, running
rm -rf,and in one case running
find ~ | grep -i <task-name>to hunt for the benchmark's ownfixtures. On a workstation checkout that is the harness author's whole filesystem, and
any answer material sitting on disk is reachable.
This adds an opt-in container sandbox for CLI agents. When
BENCH_AGENT_SANDBOXis setto
docker,1, ortrue, the agent argv is wrapped indocker runwith:filesystem and the operator's environment are not visible
HOMEpointed at the workspace, so agent config and history stay inside the runkindnetwork with the kubeconfig server rewritten to thecontrol-plane node, so TLS still verifies against the existing cert
BENCH_AGENT_IMAGEnames the image.hack/agent-sandbox.Dockerfilebuilds a suitableone (node + kubectl + the gemini CLI).
hack/agent-credential.shmints a credential forthe agent: by default an API key restricted to generativelanguage.googleapis.com, or a
short-lived impersonated token with
--vertex-sa, plus--revoketo clean up.Off by default so it can be A/B'd against current behaviour while tasks are still being
debugged.
Worth being explicit about scope: this is filesystem and environment isolation only. It
does not scope what the agent can do to the cluster, which is a separate concern best
handled with per-task RBAC, and it does not help if answer material is seeded inside the
cluster itself.