Skip to content

fix(sandbox): stop blocking bash commands (e.g. servers) from hanging the turn - #3864

Merged
WillemJiang merged 5 commits into
mainfrom
fix/bash-blocking-command-timeout
Jun 30, 2026
Merged

WillemJiang merged 5 commits into
mainfrom
fix/bash-blocking-command-timeout

Conversation

@hetaoBackend

@hetaoBackend hetaoBackend commented Jun 29, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Starting a server through the host bash tool — e.g. python -m http.server — could hang the entire turn for the full 600s timeout.

LocalSandbox.execute_command ran commands via subprocess.run(capture_output=True, timeout=600). The captured stdout/stderr pipes are inherited by every process the command spawns, so:

  • A backgrounded long-lived process (server &) keeps the read end open → communicate() blocks until the timeout fires, even though the foreground command already returned and the user correctly backgrounded it.
  • A command that reads stdin (read x) blocked until the timeout because stdin was inherited rather than /dev/null.
  • On timeout, only the direct child was killed, leaving orphaned process groups (the server kept running).

Fix

1. Code (execute_command, POSIX path). Run commands with stdin from /dev/null, a dedicated session/process group, and bounded pipe-drain capture:

Scenario Before After
server > log 2>&1 & (backgrounded) hangs 600s returns immediately, server keeps running
server & without redirect hangs 600s returns immediately; output is drained with bounded capture so it does not grow anonymous temp files
server (foreground, no &) hangs 600s, then a generic error bounded by timeout, whole group killed, clear notice to background long-lived processes
read x (reads stdin) blocks to timeout returns immediately (EOF)

The wall-clock timeout is configurable via sandbox.bash_command_timeout (default 600, unchanged default behavior). Timeout notices now format fractional/singular values correctly.

2. Windows timeout path. Windows still uses the existing subprocess.run(...) call shape, but TimeoutExpired is now caught so lowered bash_command_timeout values produce the same explanatory timeout notice instead of bubbling into the generic unexpected-error path.

3. Prompt/docs. The bash tool description still instructs the model to background long-lived processes (e.g. servers) with output redirected, so logs stay inspectable and the tool returns immediately. README.md, backend/AGENTS.md, and config.example.yaml describe the timeout/backgrounding behavior.

Tests

  • backend/tests/test_local_sandbox_command_timeout.py covers backgrounded prompt return, unredirected background stdout not inheriting deleted temp capture on Linux, foreground timeout notice, whole-process-group kill, stdin EOF, ordinary stdout/stderr/exit-code behavior, config default, bash-tool guidance, Windows TimeoutExpired, and fractional/singular timeout wording.
  • Updated backend/tests/test_local_sandbox_provider_mounts.py for the Popen-based path.
  • Local verification:
    • cd backend && make format
    • cd backend && make lint
    • cd backend && PYTHONPATH=. uv run pytest tests/test_local_sandbox_command_timeout.py tests/test_local_sandbox_provider_mounts.py tests/test_local_sandbox_encoding.py tests/test_local_sandbox_virtual_path_contract.py -q
    • cd backend && make test → 5199 passed, 20 skipped

Docs

Updated README.md, backend/AGENTS.md, and config.example.yaml.

🤖 Generated with Claude Code

Starting a server through the host bash tool (e.g. `python -m http.server`)
could hang the whole turn for the full 600s timeout. `LocalSandbox.execute_command`
used `subprocess.run(capture_output=True)`, whose captured pipes are inherited
by any process the command spawns — so a backgrounded long-lived process
(`server &`) keeps the read end open and blocks `communicate()` until the
timeout fires, even though the foreground command already returned. Commands
that read stdin blocked the same way, and on timeout only the direct child was
killed, leaving orphaned process groups.

Rework the POSIX path to capture stdout/stderr via temp files instead of pipes,
take stdin from /dev/null, and run the command in its own session/process group:

- Backgrounded long-lived processes (servers) now return immediately while the
  process keeps running.
- A command reading stdin gets immediate EOF instead of blocking.
- A genuinely blocking foreground command is bounded by a configurable
  wall-clock timeout; on timeout the whole process group is killed and the agent
  gets an explanatory notice telling it to background long-lived processes.

The timeout is configurable via `sandbox.bash_command_timeout` (default 600).
The Windows path is unchanged. Adds focused regression tests and updates docs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions github-actions Bot added area:backend Gateway / runtime / core backend under backend/ area:docs Documentation and Markdown only area:sandbox Sandboxed execution and docker/ risk:high High risk: backend API, agents, sandbox, auth, deps, CI size/M PR changes 100-300 lines labels Jun 29, 2026
Comment thread backend/packages/harness/deerflow/sandbox/local/local_sandbox.py Fixed
hetaoBackend and others added 2 commits June 29, 2026 16:55
The code fix bounds a foreground server with a timeout, but the turn still
waits the full timeout before the run continues. Add the prompt-side half:
the bash tool description now tells the model to ALWAYS start long-lived
processes (e.g. web servers) in the background with output redirected, so the
tool returns immediately. The timeout notice points at the same readable
workspace log path. Pins the guidance with a test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Address automated review: the inner `except OSError: pass` in
_terminate_process_group silently swallowed the case where the direct-child
fallback kill found the process already gone. Make the intent explicit with a
comment and a debug log instead of a bare pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@willem-bd willem-bd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid fix for the pipe-inheritance hang — temp-file capture + start_new_session + group kill is the right shape for POSIX. Three inline notes below, the first (silent /tmp leak for un-redirected backgrounded processes) is the one I'd most want discussed before merge since it sits on the PR's headline use case.

args,
shell=False,
stdin=subprocess.DEVNULL,
stdout=out_file,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heads-up on a latent leak: these temp files are dup2'd into the child's fd 1/2, so a backgrounded process (python -m http.server &) inherits them. tempfile.TemporaryFile() on POSIX is anonymous (mkstemp + immediate unlink), so when the surrounding with block closes, the parent's fd is dropped but the background process still holds its dup — every byte it writes thereafter lands in an unnamed /tmp inode that can't be read, truncated, or discovered until the process exits.

That's a silent, unbounded disk leak on exactly the long-running-background case this PR enables, whenever the model omits the > log 2>&1 redirect (the tool docstring nudges it, but nothing enforces it). The old pipe behavior hung the turn; this trades it for a leak. Worth either calling the redirect out as load-bearing in the AGENTS.md note, or making the capture bounded — at minimum flagging it so it's a conscious tradeoff rather than a surprise.

capture_output=True,
text=True,
timeout=600,
timeout=timeout,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Windows this still calls subprocess.run(..., timeout=timeout), which raises TimeoutExpired uncaught on timeout — timed_out stays False, so the explanatory notice is never produced and the process group is never killed.

The PR now flows bash_command_timeout into this path (via command_timeout in bash_tool), so the contract here did change: a Windows user who lowers it (e.g. bash_command_timeout: 30) gets a raw TimeoutExpired → bash_tool's generic except Exception → "Error: Unexpected error executing command: …" instead of the bounded-with-notice behavior advertised for POSIX. The body says "Windows path is intentionally unchanged," but the knob now reaches it. Either don't pass command_timeout on Windows (keep the hardcoded 600), or catch TimeoutExpired and set timed_out=True to mirror the POSIX path.

output += f"\nStd Error:\n{stderr}" if output else stderr
if timed_out:
notice = (
f"Command timed out after {int(timeout)} seconds and was terminated. "

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: execute_command accepts float | None, but the notice does int(timeout), so a non-integral timeout renders truncated and mispluralized — e.g. timeout=1.5 → "Command timed out after 1 seconds and was terminated.". If float timeouts are meant to be supported (the tests pass 1), round it and handle the singular; otherwise document timeout as integral.

@willem-bd willem-bd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid fix for the pipe-inheritance hang — temp-file capture + start_new_session + group kill is the right shape for POSIX. Three inline notes below, the first (silent /tmp leak for un-redirected backgrounded processes) is the one I'd most want discussed before merge since it sits on the PR's headline use case.

@github-actions github-actions Bot added the reviewing A maintainer is reviewing this PR label Jun 29, 2026
@bytedance bytedance deleted a comment from willem-bd Jun 29, 2026
@bytedance bytedance deleted a comment from willem-bd Jun 29, 2026
@bytedance bytedance deleted a comment from willem-bd Jun 29, 2026
@github-actions github-actions Bot added size/L PR changes 300-700 lines and removed size/M PR changes 100-300 lines labels Jun 30, 2026
Comment thread backend/packages/harness/deerflow/sandbox/local/local_sandbox.py Fixed
Comment thread backend/packages/harness/deerflow/sandbox/local/local_sandbox.py Fixed
Comment thread backend/packages/harness/deerflow/sandbox/local/local_sandbox.py Fixed
@WillemJiang
WillemJiang merged commit fe82552 into main Jun 30, 2026
19 checks passed
@WillemJiang
WillemJiang deleted the fix/bash-blocking-command-timeout branch June 30, 2026 06:55
jtaynl added a commit to jtaynl/deer-flow that referenced this pull request Jun 30, 2026
… for bytedance#3864)

Upstream bytedance#3864 (fe82552) made the shared bash tool call
sandbox.execute_command(command, timeout=...) for EVERY provider and updated
LocalSandbox to accept timeout, but did NOT update the community AioSandbox
provider we run in production. Without this, every agent bash call would raise
TypeError: unexpected keyword argument 'timeout'. Accept the kwarg for interface
compatibility; the AIO sandbox already bounds commands via its own
no_change_timeout, so the param is accepted but not separately applied.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
yogyoho added a commit to yogyoho/eai-flow-main that referenced this pull request Jul 3, 2026
fe82552 LocalSandbox.execute_command 改临时文件捕获+进程组终止+可配置 bash_command_timeout
AGENTS.md 冲突取 fork 版(eai AGENTS.md 仅重定向 CLAUDE.md,不用上游完整版,#3770已跳过)
jtaynl added a commit to jtaynl/deer-flow that referenced this pull request Jul 3, 2026
Conflict (1): community/aio_sandbox/aio_sandbox.py — resolved TAKE-UPSTREAM.
Upstream bytedance#3871 reworked execute_command to (self, command, env=None, timeout=None),
which SUPERSEDES our local timeout= shim (added 2026-06-30 for bytedance#3864) AND adds the
new env= for request-scoped skill secrets. Dropping our shim — our fork divergence
shrinks by one patch. Normal bash path (env=None) is byte-identical (legacy
shell.exec_command); the timeout= call from tools.py no longer TypeErrors.

The other 12 auto-merged clean. Notable: bytedance#3904 unified DB config (no-op for us —
our checkpointer.type=postgres section takes precedence), bytedance#3903 recursion clamp
(ceiling 1000 = our value → no-op), bytedance#3872 csrf cookie lifetime fix (positive),
#4fcb subagent step history (persists to existing run_events table, no migration),
bytedance#3871 request-scoped secrets. No new alembic migration → head stays 0002.
i18n auto-merged, WRI branding preserved. Login/setup pages untouched.
marvin9551 pushed a commit to marvin9551/deer-flow that referenced this pull request Aug 21, 2026
… the turn (bytedance#3864)

* fix(sandbox): stop blocking bash commands from hanging the turn

Starting a server through the host bash tool (e.g. `python -m http.server`)
could hang the whole turn for the full 600s timeout. `LocalSandbox.execute_command`
used `subprocess.run(capture_output=True)`, whose captured pipes are inherited
by any process the command spawns — so a backgrounded long-lived process
(`server &`) keeps the read end open and blocks `communicate()` until the
timeout fires, even though the foreground command already returned. Commands
that read stdin blocked the same way, and on timeout only the direct child was
killed, leaving orphaned process groups.

Rework the POSIX path to capture stdout/stderr via temp files instead of pipes,
take stdin from /dev/null, and run the command in its own session/process group:

- Backgrounded long-lived processes (servers) now return immediately while the
  process keeps running.
- A command reading stdin gets immediate EOF instead of blocking.
- A genuinely blocking foreground command is bounded by a configurable
  wall-clock timeout; on timeout the whole process group is killed and the agent
  gets an explanatory notice telling it to background long-lived processes.

The timeout is configurable via `sandbox.bash_command_timeout` (default 600).
The Windows path is unchanged. Adds focused regression tests and updates docs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(sandbox): instruct the agent to background long-lived processes

The code fix bounds a foreground server with a timeout, but the turn still
waits the full timeout before the run continues. Add the prompt-side half:
the bash tool description now tells the model to ALWAYS start long-lived
processes (e.g. web servers) in the background with output redirected, so the
tool returns immediately. The timeout notice points at the same readable
workspace log path. Pins the guidance with a test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(sandbox): make fallback-kill exception explicit and observable

Address automated review: the inner `except OSError: pass` in
_terminate_process_group silently swallowed the case where the direct-child
fallback kill found the process already gone. Make the intent explicit with a
comment and a debug log instead of a bare pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(sandbox): address bash timeout review feedback

* fix(sandbox): document fd cleanup races

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
jihtsan pushed a commit to jihtsan/dnx-deer-flow that referenced this pull request Aug 29, 2026
… the turn (bytedance#3864)

* fix(sandbox): stop blocking bash commands from hanging the turn

Starting a server through the host bash tool (e.g. `python -m http.server`)
could hang the whole turn for the full 600s timeout. `LocalSandbox.execute_command`
used `subprocess.run(capture_output=True)`, whose captured pipes are inherited
by any process the command spawns — so a backgrounded long-lived process
(`server &`) keeps the read end open and blocks `communicate()` until the
timeout fires, even though the foreground command already returned. Commands
that read stdin blocked the same way, and on timeout only the direct child was
killed, leaving orphaned process groups.

Rework the POSIX path to capture stdout/stderr via temp files instead of pipes,
take stdin from /dev/null, and run the command in its own session/process group:

- Backgrounded long-lived processes (servers) now return immediately while the
  process keeps running.
- A command reading stdin gets immediate EOF instead of blocking.
- A genuinely blocking foreground command is bounded by a configurable
  wall-clock timeout; on timeout the whole process group is killed and the agent
  gets an explanatory notice telling it to background long-lived processes.

The timeout is configurable via `sandbox.bash_command_timeout` (default 600).
The Windows path is unchanged. Adds focused regression tests and updates docs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(sandbox): instruct the agent to background long-lived processes

The code fix bounds a foreground server with a timeout, but the turn still
waits the full timeout before the run continues. Add the prompt-side half:
the bash tool description now tells the model to ALWAYS start long-lived
processes (e.g. web servers) in the background with output redirected, so the
tool returns immediately. The timeout notice points at the same readable
workspace log path. Pins the guidance with a test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(sandbox): make fallback-kill exception explicit and observable

Address automated review: the inner `except OSError: pass` in
_terminate_process_group silently swallowed the case where the direct-child
fallback kill found the process already gone. Make the intent explicit with a
comment and a debug log instead of a bare pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(sandbox): address bash timeout review feedback

* fix(sandbox): document fd cleanup races

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:backend Gateway / runtime / core backend under backend/ area:docs Documentation and Markdown only area:sandbox Sandboxed execution and docker/ reviewing A maintainer is reviewing this PR risk:high High risk: backend API, agents, sandbox, auth, deps, CI size/L PR changes 300-700 lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants