An autonomous AI-powered software repair system that detects failing tests, localizes failures, infers expected behavior, generates structured code repairs, applies them safely, and validates the result through targeted and regression testing.
The system is designed around a closed-loop repair workflow rather than one-shot code generation.
This project treats software repair as an engineering pipeline:
Failing Tests
|
v
Failure Parsing
|
v
Failure Localization
|
v
Expected Behavior Extraction
|
v
Repair Context Construction
|
v
Structured Repair Generation
|
v
Safe Statement-Level Application
|
+----------------+
| |
v v
Targeted Tests Regression Tests
| |
+--------+-------+
|
v
Repair Decision
|
+-----+-----+
| |
ACCEPT REFLECT
| |
v v
END RETRY
- Parse and structure pytest failures
- Localize failures to relevant source files and functions
- Extract expected behavior from failing tests
- Build focused repair contexts
- Generate structured repair proposals using a local LLM
- Resolve control-flow locations to executable statements
- Apply statement-level repairs instead of unrestricted source rewrites
- Validate repair proposals before modifying source code
- Run targeted tests for fast feedback
- Run the complete regression suite after a repair
- Detect newly introduced regressions
- Reflect on failed repair attempts and retry
- Create Git checkpoints before modifying source files
- Support safe rollback of repair changes
- Run with a locally hosted Ollama model
app/
├── agent/
│ ├── analyzer.py
│ ├── coder.py
│ ├── expected_behavior.py
│ ├── failure_localizer.py
│ ├── failure_parser.py
│ ├── graph.py
│ ├── planner.py
│ ├── reflector.py
│ ├── regression_test_runner.py
│ ├── repair_agent.py
│ ├── repair_applier.py
│ ├── repair_boundary.py
│ ├── repair_context.py
│ ├── repair_decision.py
│ ├── repair_generator.py
│ ├── repair_proposal.py
│ ├── scanner.py
│ ├── state.py
│ ├── statement_localizer.py
│ ├── statement_replacer.py
│ ├── targeted_test_runner.py
│ └── test_runner.py
│
├── evaluation/
│ └── evaluator.py
│
├── indexing/
│ ├── chunker.py
│ ├── embeddings.py
│ ├── parser.py
│ └── vector_store.py
│
├── tools/
│ ├── file_tools.py
│ ├── git_tools.py
│ ├── search_tools.py
│ └── test_tools.py
│
├── config.py
├── main.py
└── login_project/
The CLI executes the target repository's pytest suite and captures its output.
The failure parser extracts structured information from pytest output, including failing tests and relevant locations.
The system connects the failing test to the relevant source file and function.
The system analyzes the failing test to determine the expected behavior of the implementation.
Relevant source code, failure information, expected behavior, and localized statements are combined into a focused repair context.
A local LLM generates a structured repair proposal instead of directly rewriting an entire source file.
Example:
{
"repair_type": "replace_statement",
"function": "normalize_score",
"path": [1],
"code": "return 0 if score < 0 else score"
}The proposal is validated before being applied.
The repair system resolves an LLM-selected control-flow location to the actual executable statement.
This prevents an entire control-flow container from being replaced when only a nested statement needs modification.
The failing test is executed against the proposed repair before the broader regression suite.
The complete test suite is executed after targeted validation succeeds.
The system compares baseline and post-repair failures to identify newly introduced regressions.
If a repair fails validation, the system can reflect on the failure and generate another repair attempt within the configured retry limit.
A repair is accepted only when the required targeted and regression validation succeeds.
Before modifying a target source file, the agent can create a path-scoped Git repair checkpoint.
The checkpoint is scoped to the files being repaired rather than resetting or committing the entire parent repository.
Repository
|
v
Git Checkpoint
|
v
Generate Repair
|
v
Apply Repair
|
+-------------------+
| |
v v
Validation Pass Validation Fail
| |
v v
Accept Rollback / Retry
This provides a recovery mechanism when a repair attempt needs to be reverted.
The project is designed to run without paid API credits.
The current development setup uses Ollama with a locally hosted Qwen 2.5 7B model.
qwen2.5:7b
The LangChain Ollama integration is used for model interaction.
Example:
from langchain_ollama import ChatOllama
llm = ChatOllama(
model="qwen2.5:7b",
temperature=0
)Make sure Python, Pytest, Ollama, and the project dependencies are installed.
Verify Ollama:
ollama listDownload the model if necessary:
ollama pull qwen2.5:7bVerify the model:
ollama run qwen2.5:7b "MODEL WORKS"From the project root:
python3 -m app.main \
--repository ./sandbox/repair_bench/retry_test \
--tests tests/test_scorer.py \
--task "Fix the failing normalize_score behavior so negative scores are normalized to 0."The CLI reports:
- Initial test results
- Failure analysis
- Repair generation
- Repair application
- Targeted test results
- Regression test results
- Final repair decision
The repository contains multiple repair benchmarks:
sandbox/repair_bench/
├── arithmetic/
├── boolean_logic/
├── boundary/
├── dictionary_lookup/
├── list_filter/
├── multiple_branches/
├── nested_statement/
├── off_by_one/
├── retry_test/
└── string_transform/
These benchmarks exercise different repair scenarios including arithmetic behavior, boundary conditions, nested statements, retry behavior, and string transformations.
The repair agent was evaluated across 10 intentionally faulty software benchmarks.
| Metric | Result |
|---|---|
| Benchmarks evaluated | 10 |
| Reproducible failing benchmarks | 10 |
| Successful repairs | 9 |
| Regression-free repairs | 9 |
| Repair success rate | 90.0% |
| Regression-free rate | 100.0% |
| Total repair attempts | 10 |
| First-attempt successes | 8 |
| Retry-assisted successes | 1 |
| Maximum repair attempts | 2 |
| Benchmark | Baseline | Repair | Regression | Attempts |
|---|---|---|---|---|
| arithmetic | FAIL | PASS | PASS | 1 |
| boolean_logic | FAIL | ERROR | ERROR | 0 |
| boundary | FAIL | PASS | PASS | 1 |
| dictionary_lookup | FAIL | PASS | PASS | 1 |
| list_filter | FAIL | PASS | PASS | 1 |
| multiple_branches | FAIL | PASS | PASS | 2 |
| nested_statement | FAIL | PASS | PASS | 1 |
| off_by_one | FAIL | PASS | PASS | 1 |
| retry_test | FAIL | PASS | PASS | 1 |
| string_transform | FAIL | PASS | PASS | 1 |
Of the 9 successful repairs:
- 8 were accepted on the first generated repair.
- 1 required a second repair attempt.
multiple_branchessucceeded after 2 repair attempts.- The maximum number of attempts for a successful repair was 2.
One benchmark, boolean_logic, encountered an evaluation error before
repair execution and is reported separately rather than being counted
as a repair success or failure.
Each benchmark contains a seeded software defect.
The evaluator first verifies that the defect reproduces through the benchmark test suite.
The autonomous repair pipeline is then executed against the benchmark.
A repair is considered successful when the pipeline accepts the generated repair after validation.
A regression-free repair is one that is accepted without introducing new regression failures.
The retry_test benchmark contains a deliberately failing implementation.
Initial state:
1 failed, 3 passed
The agent generated and applied:
return 0 if score < 0 else scoreThe targeted test then passed:
1 passed
The complete regression suite subsequently passed:
4 passed
The final decision was:
REPAIR ACCEPTED
No new regression was introduced.
Initial Repository
|
v
Test Failure
|
v
Failure Parser
|
v
Failure Localizer
|
v
Expected Behavior
|
v
Repair Context
|
v
Structured LLM Proposal
|
v
Statement Validation
|
v
Apply Repair
|
v
Targeted Test
|
v
Regression Suite
|
+---+---+
| |
PASS FAIL
| |
v v
ACCEPT REFLECT
|
v
RETRY
- Python
- LangChain
- LangGraph
- Ollama
- Qwen 2.5 7B
- Pytest
- Git
- Python AST-based source analysis
The system does not stop after generating code. Repairs must be executed and validated.
Repairs are represented as structured proposals rather than unrestricted source-file rewrites.
The failing test is validated directly before running the broader regression suite.
The system distinguishes baseline failures from failures introduced by a repair.
Git checkpoints provide a recovery mechanism before source changes are applied.
The development setup can operate entirely with a locally hosted LLM through Ollama.
The primary implemented workflow focuses on autonomous test-driven code repair:
failure
->
localization
->
behavior extraction
->
repair generation
->
repair application
->
targeted validation
->
regression validation
->
decision
->
reflection/retry
The repository also contains indexing and evaluation components intended to support future repository-scale repair and retrieval capabilities.
Potential extensions include:
- Repository-scale semantic code retrieval
- Tree-sitter-based parsing
- Embedding-backed code search
- Vector database integration
- Larger repair benchmarks
- Dockerized execution sandboxes
- Expanded repair strategies
- Automated repair quality evaluation
- API-based service deployment
- Execution and repair monitoring
The goal of this project is to explore how agentic AI can be applied to software engineering workflows where correctness must be established through execution rather than generation alone.
The core principle is:
Generate a repair, prove it works, and only then accept it.