Skip to content

perf(runtime): import local sources via single tar put_archive#695

Open
aYoung-CS wants to merge 2 commits into
usestrix:mainfrom
aYoung-CS:feat/tar-fast-local-source-import
Open

perf(runtime): import local sources via single tar put_archive#695
aYoung-CS wants to merge 2 commits into
usestrix:mainfrom
aYoung-CS:feat/tar-fast-local-source-import

Conversation

@aYoung-CS

@aYoung-CS aYoung-CS commented Jul 5, 2026

Copy link
Copy Markdown

Summary

Closes #694.

Copying a local source tree into the sandbox currently goes through the SDK's
per-file LocalDir materialization (session.start()LocalDir.apply()),
which issues several docker exec calls per file. On large repositories those
thousands of exec round-trips serialize against a non-thread-safe docker-py
client, producing multi-minute "stuck on loading" hangs and occasional
ExecTransportError.

This PR replaces that path with one container.put_archive() per source:
the tree is packed host-side into an in-memory tar and unpacked in the
container in a single shot (sub-second for thousands of files). It complements
the existing --mount option (#577): --mount is a zero-copy read-only bind
for repos you only want to read, while this speeds up the default copy path
for repos the agent needs to write to (or when you need .git inside the box).

Why it's faster (and by how much)

The cost is dominated by docker exec round-trips, not by data volume. The
SDK copies each file with a mkdir(parent) plus a write(), and write()
itself stages via mkdir + stream + move — roughly 4 exec round-trips per
file
, run against a client that serializes them. The tar path issues a
fixed 3 exec calls total (mkdir + put_archive + chown) regardless of
file count, so cost stays flat as the tree grows.

Measured against a real strix-sandbox container (synthetic source tree of
small files, timing the copy/import step):

files SDK per-file LocalDir tar put_archive speedup
100 69.84 s 0.23 s ~300x

At 100 files the per-file path already takes ~70 s (~0.7 s/file); it scales
linearly with file count while the tar path stays sub-second — which is why
real repos produced the multi-minute "stuck on loading" hangs this PR removes.
So yes: even for well-under-1 GB trees the win is large, because it's the file
count, not the byte size, that hurts.

What changed

  • strix/runtime/session_manager.py
    • build_session_entriessplit_local_sources: non-mount sources are now
      returned as a plain list to copy after start, instead of LocalDir
      manifest entries. The manifest carries entries={}.
    • New _build_source_tar: packs a source tree into an in-memory tar.
    • New _import_local_sources: put_archive the tar, then chown to the
      container's runtime user (pentester), since put_archive unpacks as root
      with the tar's host uids.
  • strix/runtime/backends.py, strix/runtime/docker_client.py: docstring/comment
    updates only — the --mount bind-mount path is unchanged.
  • tests/test_session_entries.py: rewritten to cover split_local_sources
    (copy vs bind split, slash-stripping, incomplete entries) and
    _build_source_tar (arc prefix, dotfile/.git preservation, symlink skip,
    no-descend into symlinked dirs, empty dir).

Behavior notes for reviewers

Two intentional differences from the SDK's LocalDir:

  1. Dotfiles / .git are preserved (copied as-is). This keeps source-aware
    and git-diff scoping working, and stays consistent with the existing
    find_oversized_local_targets size guard, which already measures .git.
  2. Symlinks are skipped and counted, not followed. The SDK's LocalDir
    hard-errors (LocalDirReadError) on any symlink; skipping is strictly more
    permissive and avoids both host path escapes and dangling links inside the
    container. The skip count is logged.

Why put_archive is safe here (the SDK deliberately avoids it, see
agents/sandbox/sandboxes/docker.py:709): the SDK's caveat is specific to
Docker volume-driver-backed mounts, whose plugins can re-run mount setup
during archive ops. The copy path attaches no such mounts. --mount uses a
plain read-only bind at a different subdir, so there's no volume driver to
trip.

Test plan

  • pytest tests/test_session_entries.py — 10 passed
  • Full suite pytest — 68 passed (2 files skipped only because the local
    env's Python lacks _sqlite3; unrelated to this change)
  • ruff check . — the changed code adds zero findings. The branch
    reports the exact same pre-existing errors as main (confirmed by running
    the same command on both via git worktree); none are in changed lines.
  • ruff format --check — clean on all changed files
  • pyright strix/runtime/session_manager.py — 0 errors. pyright strix/runtime/docker_client.py reports the same pre-existing errors as
    main (unchanged there; only a comment was touched).
  • bandit -c pyproject.toml strix/runtime/session_manager.py — no issues
  • Manual: scan a large local repo (strix --target ./big-repo) and confirm
    it lands quickly with no "stuck on loading" hang

Note: make check-all also runs mypy strix/, which currently crashes with
an INTERNAL ERROR in my local env (mypy 2.1.0) — but it crashes identically
on an unmodified main, so it's an environment/version issue, not something
this PR introduces. Deferring to CI for the authoritative mypy run.

Notes / tradeoffs

  • The tar is built in memory before put_archive. It's bounded by the existing
    STRIX_MAX_LOCAL_COPY_MB guard (default 1024 MB), which rejects oversized
    non-mount targets in main.py before they reach this code and points users
    at --mount. If a smaller footprint is ever wanted, the builder can spill to
    a temp file — kept in-memory here for simplicity within that bound.
  • _container_of reaches session._inner._container, pinned to
    openai-agents==0.14.6 (the same private-attr reliance already present in
    docker_client.py).

Copy local source trees into the sandbox with one docker put_archive per
source instead of the SDK's per-file LocalDir materialization, which issues
several docker exec calls per file. On large trees those thousands of exec
round-trips serialize against a non-thread-safe docker-py client, causing
multi-minute "stuck on loading" hangs and occasional ExecTransportError.
put_archive lands the whole tree in one shot (sub-second for thousands of
files).

The manifest now carries no source entries; sources are packed host-side
into an in-memory tar and unpacked after start(). Dotfiles (including .git)
are preserved so source-aware and git-diff analysis keep working; symlinks
are skipped-and-counted rather than followed (avoids host path escapes and
dangling links) — strictly more permissive than the SDK, which hard-errors
on any symlink.

The read-only --mount bind-mount path is unchanged. put_archive is safe on
the copy path because it attaches no volume-driver mounts (the reason the
SDK avoids put_archive); --mount uses plain bind mounts at a separate subdir.
@greptile-apps

greptile-apps Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR changes local source imports to use a tar archive copied into the container. The main changes are:

  • Replaces SDK LocalDir entries with post-start put_archive imports.
  • Splits local sources into copied sources and read-only bind mounts.
  • Adds tar-building tests for dotfiles, symlinks, and source splitting.
  • Updates runtime comments to describe the new copy path.

Confidence Score: 4/5

The new source import path needs fixes before merging.

  • Unsafe workspace subdir values can escape the intended copy destination.
  • Failed ownership changes can leave copied sources unwritable while session creation still succeeds.
  • Empty directories are not preserved by the new tar builder.

strix/runtime/session_manager.py

Security Review

The new tar import path uses workspace_subdir in archive names without rejecting parent-directory components, so unsafe source metadata can write outside the intended workspace destination.

Important Files Changed

Filename Overview
strix/runtime/session_manager.py Adds the tar-based source import path and source splitting; path validation, ownership checks, and empty-directory preservation need follow-up.
tests/test_session_entries.py Rewrites tests around source splitting and tar construction, with coverage for dotfiles and symlink skipping.
strix/runtime/backends.py Updates backend documentation for the new post-start local source import flow.
strix/runtime/docker_client.py Updates a bind-mount comment to match the new local source copy behavior.
Prompt To Fix All With AI
Fix the following 3 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 3
strix/runtime/session_manager.py:49
**Workspace Prefix Escapes Root**

When `workspace_subdir` contains parent-directory segments, the tar member names and later `chown` target are built from that value unchanged. A copied source such as `../tmp/repo` can extract outside the intended `/workspace/<subdir>` tree, so this new import path needs to reject `..` and other unsafe path components before building archive names.

### Issue 2 of 3
strix/runtime/session_manager.py:160-168
**Ownership Failure Is Ignored**

`exec_run()` reports command failures in its result, but this `chown` result is ignored. If the target path was not created as expected or ownership cannot be changed, `create_or_reuse()` still returns a session with root-owned sources, and later agent tools fail when they try to write in the workspace.

### Issue 3 of 3
strix/runtime/session_manager.py:89-96
**Empty Directories Are Dropped**

The tar builder only adds regular files, so directories with no files are never created in the sandbox. Repositories that rely on committed empty directories for generated output, cache paths, or package scaffolding can run with those paths missing after the new copy path completes.

Reviews (1): Last reviewed commit: "perf(runtime): import local sources via ..." | Re-trigger Greptile

Comment thread strix/runtime/session_manager.py
Comment thread strix/runtime/session_manager.py Outdated
Comment thread strix/runtime/session_manager.py
- reject workspace_subdir values with `..`/empty path segments before they
  become tar member names or a chown target (path-escape guard), applied to
  both copied and bind-mount sources
- check exec_run exit codes for mkdir/chown via _run_checked and raise with
  the command's diagnostics, so a failed chown no longer yields a session
  with root-owned, unwritable sources
- pack directory entries (including the arc-prefix root and committed empty
  dirs) into the tar so empty scaffolding directories survive the import

Tests cover the escape guard, empty-dir preservation, and the root-prefix dir.
@aYoung-CS

Copy link
Copy Markdown
Author

Thanks for the thorough review! I've pushed a follow-up commit (05d9ffb) that
addresses all three issues:

  1. Workspace prefix escapes rootsplit_local_sources now rejects any
    workspace_subdir containing .. or empty path segments (via
    _is_safe_workspace_subdir) before the value is used as a tar member
    prefix or chown target. Applied to both copied and bind-mount sources.

  2. Ownership failure is ignoredmkdir/chown now go through a
    _run_checked helper that inspects the exec_run exit code and raises with
    the command's stderr on failure, so a failed chown no longer yields a
    session with root-owned, unwritable sources.

  3. Empty directories are dropped_build_source_tar now packs directory
    entries (including the arc-prefix root and committed empty dirs), so empty
    scaffolding directories survive the import.

New tests cover the escape guard, empty-dir preservation, and the root-prefix
dir. I also verified end-to-end against a real strix-sandbox container that
files, nested paths, .git, and empty dirs all land correctly, ownership is
handed to pentester, and _run_checked raises on a failing command.

pytest tests/test_session_entries.py (13 passed), ruff check,
ruff format --check, pyright strix/runtime/session_manager.py (0 errors),
and bandit are all clean on the changed files.

@andreieuganox

Copy link
Copy Markdown

The approach is ignoring gitignores... This is the big issue for files that should never be on Production, such as local .env...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Local source copy hangs on large repositories (per-file LocalDir exec loop)

2 participants