`exec_module` of proposer-written `harness.py` runs module-scope code in-process; the candidate gate is `compile()`-only

[EvolveAegis]

What

BaseAgent.__init__ loads the per-workspace harness.py via spec.loader.exec_module(module) (agent_evolve/protocol/base_agent.py:107, reached from _load_harness:92 <- reload_from_fs:48), running the file's module-top-level code in the a-evolve process. The candidate gate MetaHarnessEngine._validate_candidate (agent_evolve/algorithms/meta_harness/engine.py:579-615) only calls compile() at :594 — syntax-only, no AST/capability/provenance check, and it does not exec the code.

The proposer writes the file directly: _run_claude_code (engine.py:715) runs the Claude Code CLI with cwd=workspace.root and --dangerously-skip-permissions, so the proposer's filesystem edits land harness.py at the workspace root. The mutation is then captured by _git_diff (engine.py:160) and put through _validate_candidate (engine.py:161). (AgentWorkspace.write_harness at agent_evolve/contract/workspace.py:169 is the workspace API for the same write, but it currently has no callers in the repo — the proposer writes the file directly rather than through it.) Line numbers at commit c9d4789f2be499589d543aa08e74d05d10d93177 (current main HEAD).

How to reproduce

Pinned to c9d4789f2be499589d543aa08e74d05d10d93177:

git clone https://github.com/A-EVO-Lab/a-evolve.git
cd a-evolve
git checkout c9d4789f2be499589d543aa08e74d05d10d93177
python3 - <<'PY'
import sys, secrets, pathlib, logging, shutil
sys.path.insert(0, ".")
logging.disable(logging.CRITICAL)
from agent_evolve.contract.workspace import AgentWorkspace
from agent_evolve.protocol.base_agent import BaseAgent
from agent_evolve.algorithms.meta_harness.engine import MetaHarnessEngine
from agent_evolve.config import EvolveConfig

nonce = "canary-" + secrets.token_hex(8)
ws_root = pathlib.Path("/tmp/poc_ws")
shutil.rmtree(ws_root, ignore_errors=True); ws_root.mkdir()
(ws_root / "prompts").mkdir()
(ws_root / "prompts" / "system.md").write_text("be a good agent")  # non-empty, so the gate result reflects harness.py alone

# harness.py as the proposer would write it to the workspace root
(ws_root / "harness.py").write_text(
    f'import pathlib\n'
    f'pathlib.Path("/tmp/poc_harness_fired").write_text("{nonce}")\n'
    f'def build_system_prompt(p, s):\n    return p\n'
)
ws = AgentWorkspace(ws_root)

# gate is compile()-only: a syntax-valid harness passes, and the gate does not exec it
engine = MetaHarnessEngine(EvolveConfig(extra={}))
print("_validate_candidate ->", engine._validate_candidate(ws))

fired = pathlib.Path("/tmp/poc_harness_fired")
fired.unlink(missing_ok=True)  # clear any marker from a previous run
print("canary present before BaseAgent constructed:", fired.exists())

# BaseAgent.__init__ -> reload_from_fs -> _load_harness -> exec_module runs the top-level code here
class _Agent(BaseAgent):
    def solve(self, task): ...
a = _Agent(ws_root)
print("OBSERVED:", pathlib.Path("/tmp/poc_harness_fired").read_text(), "(nonce:", nonce + ")")
PY

Observed on a clean run:

_validate_candidate -> (True, '')
canary present before BaseAgent constructed: False
OBSERVED: canary-877f50b61e61babd (nonce: canary-877f50b61e61babd)

A syntax-broken harness fails _validate_candidate (compile() raises), but any syntax-valid harness passes the gate and runs at construction.

Impact / scope

The compile-only gate never exec's the harness; the harness runs in-process via exec_module whenever a BaseAgent is constructed from a workspace that contains harness.py. In the parallel evaluator path (_evaluate_parallel, engine.py:377), a candidate's harness runs in the temp-copy worker that scores it; in the serial path, it runs when an agent is next constructed from that workspace (a later reload).

This is conditional local hardening, not a network vector. I am not claiming the LLM-mediated hop — i.e. that the proposer can be induced to author a malicious harness.py; that depends on the proposer prompt and model. The point of this report is what happens once a proposer-authored or operator-adopted harness.py is in place: its module-top-level code runs as the a-evolve process user, with no capability gate between the write and execution.

Suggested change

Run proposer/agent-authored harness.py out of process — a subprocess or container that returns only structured results over IPC — instead of exec_module in the agent's own process. Failing that, narrow the harness contract to a fixed, non-Python hook surface (declared function names plus config) rather than arbitrary importable Python. Happy to open a PR.