perf(runtime): import local sources via single tar put_archive#695
perf(runtime): import local sources via single tar put_archive#695aYoung-CS wants to merge 2 commits into
Conversation
Copy local source trees into the sandbox with one docker put_archive per source instead of the SDK's per-file LocalDir materialization, which issues several docker exec calls per file. On large trees those thousands of exec round-trips serialize against a non-thread-safe docker-py client, causing multi-minute "stuck on loading" hangs and occasional ExecTransportError. put_archive lands the whole tree in one shot (sub-second for thousands of files). The manifest now carries no source entries; sources are packed host-side into an in-memory tar and unpacked after start(). Dotfiles (including .git) are preserved so source-aware and git-diff analysis keep working; symlinks are skipped-and-counted rather than followed (avoids host path escapes and dangling links) — strictly more permissive than the SDK, which hard-errors on any symlink. The read-only --mount bind-mount path is unchanged. put_archive is safe on the copy path because it attaches no volume-driver mounts (the reason the SDK avoids put_archive); --mount uses plain bind mounts at a separate subdir.
Greptile SummaryThis PR changes local source imports to use a tar archive copied into the container. The main changes are:
Confidence Score: 4/5The new source import path needs fixes before merging.
strix/runtime/session_manager.py
|
| Filename | Overview |
|---|---|
| strix/runtime/session_manager.py | Adds the tar-based source import path and source splitting; path validation, ownership checks, and empty-directory preservation need follow-up. |
| tests/test_session_entries.py | Rewrites tests around source splitting and tar construction, with coverage for dotfiles and symlink skipping. |
| strix/runtime/backends.py | Updates backend documentation for the new post-start local source import flow. |
| strix/runtime/docker_client.py | Updates a bind-mount comment to match the new local source copy behavior. |
Prompt To Fix All With AI
Fix the following 3 code review issues. Work through them one at a time, proposing concise fixes.
---
### Issue 1 of 3
strix/runtime/session_manager.py:49
**Workspace Prefix Escapes Root**
When `workspace_subdir` contains parent-directory segments, the tar member names and later `chown` target are built from that value unchanged. A copied source such as `../tmp/repo` can extract outside the intended `/workspace/<subdir>` tree, so this new import path needs to reject `..` and other unsafe path components before building archive names.
### Issue 2 of 3
strix/runtime/session_manager.py:160-168
**Ownership Failure Is Ignored**
`exec_run()` reports command failures in its result, but this `chown` result is ignored. If the target path was not created as expected or ownership cannot be changed, `create_or_reuse()` still returns a session with root-owned sources, and later agent tools fail when they try to write in the workspace.
### Issue 3 of 3
strix/runtime/session_manager.py:89-96
**Empty Directories Are Dropped**
The tar builder only adds regular files, so directories with no files are never created in the sandbox. Repositories that rely on committed empty directories for generated output, cache paths, or package scaffolding can run with those paths missing after the new copy path completes.
Reviews (1): Last reviewed commit: "perf(runtime): import local sources via ..." | Re-trigger Greptile
- reject workspace_subdir values with `..`/empty path segments before they become tar member names or a chown target (path-escape guard), applied to both copied and bind-mount sources - check exec_run exit codes for mkdir/chown via _run_checked and raise with the command's diagnostics, so a failed chown no longer yields a session with root-owned, unwritable sources - pack directory entries (including the arc-prefix root and committed empty dirs) into the tar so empty scaffolding directories survive the import Tests cover the escape guard, empty-dir preservation, and the root-prefix dir.
|
Thanks for the thorough review! I've pushed a follow-up commit (05d9ffb) that
New tests cover the escape guard, empty-dir preservation, and the root-prefix
|
|
The approach is ignoring gitignores... This is the big issue for files that should never be on Production, such as local .env... |
Summary
Closes #694.
Copying a local source tree into the sandbox currently goes through the SDK's
per-file
LocalDirmaterialization (session.start()→LocalDir.apply()),which issues several
docker execcalls per file. On large repositories thosethousands of exec round-trips serialize against a non-thread-safe
docker-pyclient, producing multi-minute "stuck on loading" hangs and occasional
ExecTransportError.This PR replaces that path with one
container.put_archive()per source:the tree is packed host-side into an in-memory tar and unpacked in the
container in a single shot (sub-second for thousands of files). It complements
the existing
--mountoption (#577):--mountis a zero-copy read-only bindfor repos you only want to read, while this speeds up the default copy path
for repos the agent needs to write to (or when you need
.gitinside the box).Why it's faster (and by how much)
The cost is dominated by
docker execround-trips, not by data volume. TheSDK copies each file with a
mkdir(parent)plus awrite(), andwrite()itself stages via mkdir + stream + move — roughly 4 exec round-trips per
file, run against a client that serializes them. The tar path issues a
fixed 3 exec calls total (
mkdir+put_archive+chown) regardless offile count, so cost stays flat as the tree grows.
Measured against a real
strix-sandboxcontainer (synthetic source tree ofsmall files, timing the copy/import step):
LocalDirput_archiveAt 100 files the per-file path already takes ~70 s (~0.7 s/file); it scales
linearly with file count while the tar path stays sub-second — which is why
real repos produced the multi-minute "stuck on loading" hangs this PR removes.
So yes: even for well-under-1 GB trees the win is large, because it's the file
count, not the byte size, that hurts.
What changed
strix/runtime/session_manager.pybuild_session_entries→split_local_sources: non-mount sources are nowreturned as a plain list to copy after start, instead of
LocalDirmanifest entries. The manifest carries
entries={}._build_source_tar: packs a source tree into an in-memory tar._import_local_sources:put_archivethe tar, thenchownto thecontainer's runtime user (
pentester), sinceput_archiveunpacks as rootwith the tar's host uids.
strix/runtime/backends.py,strix/runtime/docker_client.py: docstring/commentupdates only — the
--mountbind-mount path is unchanged.tests/test_session_entries.py: rewritten to coversplit_local_sources(copy vs bind split, slash-stripping, incomplete entries) and
_build_source_tar(arc prefix, dotfile/.gitpreservation, symlink skip,no-descend into symlinked dirs, empty dir).
Behavior notes for reviewers
Two intentional differences from the SDK's
LocalDir:.gitare preserved (copied as-is). This keeps source-awareand git-diff scoping working, and stays consistent with the existing
find_oversized_local_targetssize guard, which already measures.git.LocalDirhard-errors (
LocalDirReadError) on any symlink; skipping is strictly morepermissive and avoids both host path escapes and dangling links inside the
container. The skip count is logged.
Why
put_archiveis safe here (the SDK deliberately avoids it, seeagents/sandbox/sandboxes/docker.py:709): the SDK's caveat is specific toDocker volume-driver-backed mounts, whose plugins can re-run mount setup
during archive ops. The copy path attaches no such mounts.
--mountuses aplain read-only bind at a different subdir, so there's no volume driver to
trip.
Test plan
pytest tests/test_session_entries.py— 10 passedpytest— 68 passed (2 files skipped only because the localenv's Python lacks
_sqlite3; unrelated to this change)ruff check .— the changed code adds zero findings. The branchreports the exact same pre-existing errors as
main(confirmed by runningthe same command on both via
git worktree); none are in changed lines.ruff format --check— clean on all changed filespyright strix/runtime/session_manager.py— 0 errors.pyright strix/runtime/docker_client.pyreports the same pre-existing errors asmain(unchanged there; only a comment was touched).bandit -c pyproject.toml strix/runtime/session_manager.py— no issuesstrix --target ./big-repo) and confirmit lands quickly with no "stuck on loading" hang
Notes / tradeoffs
put_archive. It's bounded by the existingSTRIX_MAX_LOCAL_COPY_MBguard (default 1024 MB), which rejects oversizednon-mount targets in
main.pybefore they reach this code and points usersat
--mount. If a smaller footprint is ever wanted, the builder can spill toa temp file — kept in-memory here for simplicity within that bound.
_container_ofreachessession._inner._container, pinned toopenai-agents==0.14.6(the same private-attr reliance already present indocker_client.py).