Fix terminal-bench integration test in CI #5
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Fix pyproject.toml: Add [tool.hatch.build.targets.wheel] packages specification - Enable terminal-bench test in CI workflow with proper conditions - Add PATH fixes for uv and terminal-bench binaries - Add timeouts to prevent hanging (20min job, 15min step) - Add .actrc for local GitHub Actions testing with act - Add .secrets to .gitignore The terminal-bench adapter package was failing to build because hatchling didn't know which files to include. Now it correctly includes the adapters/ directory and the test can run in CI.
The adapter was missing the memory module when copying Fireteam code into the terminal-bench container, causing ModuleNotFoundError when orchestrator.py tried to import memory.manager. This wasn't caught in local act testing because we interrupted the test before it reached actual execution inside the container.
The --livestream flag was causing terminal-bench to hide console output and stream to tmux instead, making it impossible to see progress or debug issues in CI/act logs. Removing it allows output to appear normally in pytest/CI logs.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the terminal-bench integration test that was disabled in CI.
Root Cause
The
benchmark/pyproject.tomlwas missing package configuration, causing the Fireteam adapter installation to fail with:Changes
[tool.hatch.build.targets.wheel]withpackages = ["adapters"]if: falseto run on main and e/* branches$HOME/.local/binto PATH for uv and terminal-bench.actrcfor testing withactlocallyTesting
Local Iteration
Developers can now test this workflow locally using:
This PR will verify the test runs successfully in the actual CI environment.
Note: Branch name starts with
e/to trigger integration tests as configured in the workflow.