Design untracked-file capture for trial outcome diffs #92

This was generated by AI during triage.

Summary

Agent Eval Lab currently captures trial diffs with git diff --binary, which records tracked file modifications but does not include untracked files unless an agent explicitly stages them. That means a trial can create a new file that affects tests while the persisted diff.patch, file-change metrics, and review evidence may not fully represent the workspace outcome.

This issue is a follow-up from #90. Base-only workspaces should preserve the current diff behavior for the first history-isolation slice; untracked-file capture should be designed separately because it changes outcome evidence and metrics.

Current Local Context

agentlab.execution.workspace.capture_diff uses:

• git diff --binary for patch capture
• git diff --name-only for changed-file lists

Those commands ignore untracked files by default.

Why This Matters

For coding-agent evals, the outcome should represent the final workspace state. If an agent creates a new source file, fixture, config file, or test file and does not stage it, the grader may see it but the stored outcome evidence may not. This can affect:

• diff.patch
• files_changed
• line additions/deletions
• max_files_changed enforcement
• human review evidence
• capability digests and report claims

Design Questions

• Should untracked files be included in the persisted diff.patch?
• Should ignored files remain excluded, or should task-specific allowlists control this?
• How should large generated files, virtualenvs, package caches, build outputs, and screenshots be excluded?
• Should setup-created untracked files be snapshotted before the agent phase so only agent-created untracked files count?
• Should untracked capture apply to reference verification as well as agent trials?
• How should binary untracked files be represented?

Acceptance Criteria

• A design decision is recorded before implementation changes diff semantics.
• The design distinguishes setup-created files from agent-created files.
• The design explains how ignored/generated/cache files are excluded.
• Tests cover an agent-created untracked file that affects a grader.
• Tests cover setup-created untracked files that should not count as model changes.
• Reports/results continue to make clear, reproducible claims about the final outcome.

Out of Scope For #90

Do not change untracked-file capture while implementing #90 unless the maintainer explicitly expands #90. #90 should focus on making agent-facing git history base-only.