reset_per_test isolation can silently fail: cli/http graders never reset; docker reset ignores errors

[lezama]

Summary

reset_per_test isolation (#25) is the foundation of reproducible scores, but two edge paths let a run claim isolation it didn't actually get. Both are pre-existing observations from the #25/#26 reviews — neither is a regression, and neither affects the documented wp_env_dir + docker config, which resets correctly.

Gap 1: reset() silently no-ops for cli / http graders

WordPressEnvironment.reset() only acts on two branches:

def reset(self) -> None:
    if self.config.wp_env_dir:            # resets ✓
        ...
    elif self.config.kind == "docker":    # resets ✓
        ...
    # kind == "cli"  → nothing
    # kind == "http" → nothing (currently rejected at config load, so moot until implemented)

But execute_code()/_exec() fully support kind: cli (runs wp against a real, stateful WordPress). So a grader.kind: cli run under the default execution_isolation: reset_per_test:

• executes serially,
• resets nothing,
• and still stamps metadata.runtime_isolation: "reset_per_test" in the results file.

That's a false guarantee in the auditable record — exactly the failure mode the isolation work exists to eliminate.

Suggested fix (fail loud, consistent with the existing concurrency validator): raise in reset() when the grader kind has no reset implementation, or reject reset_per_test + cli-without-wp_env_dir at config validation with a clear message.

Gap 2: docker-only reset() discards _exec() results

The bare-docker branch (kind: docker without wp_env_dir) calls:

self._exec(["wp", "db", "reset", "--yes"])
self._exec(install_cmd)

_exec() returns (stdout, stderr, returncode, timed_out) (#26), but both results are ignored. If db reset or core install times out or exits nonzero, reset silently "succeeds" and the next test runs against a dirty or uninstalled WordPress — producing confusing downstream assertion failures attributed to the model rather than the harness.

Contrast: the wp_env_dir branch uses _run_wp_env(), which raises EnvironmentSetupTimeout on timeout and RuntimeError on nonzero exit. The docker branch should get the same semantics (a reset failure is a harness/environment failure, not a model result).

Why it matters

Provider-grade reproducibility rests on runtime_isolation: reset_per_test meaning what it says. Both gaps break that contract only on non-default configs, but silently — an honest error beats quietly-wrong isolation (same principle as #28's config hardening).

Happy to pick this up — it's a small diff plus tests (reset-failure raises; cli+reset_per_test rejected or raises).

🤖 Generated with Claude Code