Use git worktrees for parallel task isolation

[chanakyav]

All autopilot tasks currently share a single git working directory. Branch locking prevents two tasks on the same branch, but true parallel execution on different branches would cause git state corruption (git checkout in one task blows away another's working tree).

Use Case

Run multiple autopilot tasks simultaneously on different branches — e.g., implement feature A while fixing CI on feature B.

Proposed Design

Use git worktrees to give each task its own isolated checkout:

~/repo/                              # main checkout (user's terminal)
~/.autopilot-loop/worktrees/abc123/  # worktree for task abc123
~/.autopilot-loop/worktrees/def456/  # worktree for task def456

Changes Needed

1. On task creation, run git worktree add <path> -b <branch> (or git worktree add <path> <existing-branch>)
2. Pass cwd=<worktree_path> to subprocess.Popen in run_agent()
3. On task completion/cleanup, run git worktree remove <path>
4. Update prompts — agent no longer needs to git checkout -b, it's already on the right branch
5. Store worktree path in the task DB record

Benefits

• True parallel task execution with zero git conflicts
• Each agent sees a clean, isolated working tree
• Prerequisite for task chaining (#TBD) and research mode (#TBD)

Risks

• Worktree paths need cleanup on crash/failure
• Some tools may not work correctly in worktrees (rare)
• Disk usage increases (each worktree is a full checkout)

[m13v]

been doing exactly this for months and it works great once you sort out a few things.

git worktrees share the .git directory but each gets its own working tree and index, so parallel checkout/stage/commit is fully safe. the only shared resource that needs coordination is worktree creation itself (git config lock) - we serialize that through a file lock.

one design choice worth making early: do worktrees get their own branch or work on existing branches? we create a fresh branch per worktree (worktree/task-123) and merge back to the target branch when done. this avoids the situation where two tasks try to push to the same branch and one gets a non-fast-forward rejection.

also consider the disk space implications - each worktree is a full copy of the working tree (minus .git). on large repos this adds up fast. we prune completed worktrees aggressively.

[m13v]

the tmux-based parallel task runner that handles worktree lifecycle: https://github.com/m13v/tmux-background-agents/blob/main/SKILL.md

[chanakyav]

@m13v Thanks for feedback!

also consider the disk space implications - each worktree is a full copy of the working tree (minus .git). on large repos this adds up fast. we prune completed worktrees aggressively.

Thats a good point! I typically spin up 1 codespace per feature but I should still consider aggressively pruning as well. Especially when autopilot-loop is running locally 👍🏽

[m13v]

yeah good call on the disk space point. we've found that for most repos the worktree overhead is manageable since it shares .git, but for monorepos with large build artifacts it can add up. cleaning up worktrees after the task completes helps, and you can automate that in the post-task hook.

[chanakyav]

Architecture Note: File Placement

All worktree logic should live in a new worktree.py module (not in orchestrator.py or cli.py). Functions to include:

• create_worktree(task_id, branch, base_branch) -> path
• remove_worktree(task_id)
• get_worktree_path(task_id)
• cleanup_orphaned_worktrees()

Tests go in a new test_worktree.py.

This keeps orchestrator.py (1,357 lines) from growing and gives every downstream issue (#53, #54, #57, #59) a clean import target.