Skip to content

fix: isolate WordPress runtime state per execution test - #25

Merged
lezama merged 1 commit into
trunkfrom
wpbench/01-isolate-runtime-state
Jul 10, 2026
Merged

fix: isolate WordPress runtime state per execution test#25
lezama merged 1 commit into
trunkfrom
wpbench/01-isolate-runtime-state

Conversation

@JasonTheAdams

@JasonTheAdams JasonTheAdams commented Jul 8, 2026

Copy link
Copy Markdown
Member

Why this matters

WP-Bench scores are only meaningful if they're reproducible. Today, execution tests run concurrently against one shared WordPress instance with no reset between tests. WordPress is deeply stateful — options, posts, roles, hooks, transients, cron events all persist in the database — so code generated for one test can silently change the outcome of the next. In multi-model mode the problem compounds: one model's leftover state can help or hurt the next model's scores. That means rankings can shift based on test ordering and scheduling luck, and a provider who re-runs the suite may get different numbers. No leaderboard can be trusted on that foundation. This change makes every execution test start from a known-clean WordPress baseline, which is the prerequisite for every scoring and auditability improvement that follows.

Changes

  • RunConfig: new execution_isolation (reset_per_test default | none) and execution_concurrency (default 1). A model validator rejects execution_concurrency > 1 under reset_per_test with a clear error instead of silently sharing state.
  • core.py: single-model, multi-model, and reference-solution execution paths now route through one shared _run_isolated_execution_loop() that calls environment.reset() before every test. Knowledge tests keep their existing concurrency (they never touch the WordPress runtime).
  • WordPressEnvironment.reset(): wp db reset drops all tables and leaves WordPress uninstalled, so reset now follows with wp core install to return to a deterministic just-installed baseline (grader.base_url added for the install URL).
  • Result metadata: metadata.runtime_isolation records the mode used, in both single- and multi-model payloads, so any results file shows how it was produced.
  • Docs: README + wp-bench.example.yaml document the isolation semantics and the official-run requirement.

Verification

  • pytest python: 41 passed (8 new isolation tests: reset-before-every-test in all three modes, reset between models, concurrency rejection, metadata recording, legacy opt-out)
  • ruff check python: clean
  • mypy python: 3 pre-existing trunk errors only (none introduced)
  • Runtime smoke test vs Docker/wp-env: not run in this PR (no local runtime up); the reset→reinstall sequence follows the wp-env docs and a state-leak interleaving assertion is covered in the mocked tests.

Notes for reviewers

  • First PR in a series moving the harness toward provider-grade reproducibility; subsequent PRs stack on this branch.
  • A per-worker isolated-runtime pool (parallel execution with full isolation) is intentionally deferred to keep this diff reviewable — the execution_isolation config surface leaves room for a future container_per_worker mode.

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: JasonTheAdams <jason_the_adams@git.wordpress.org>
Co-authored-by: lezama <migueluy@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

Execution tests previously ran concurrently against one shared
WordPress runtime with no reset between tests, so generated code
could leak options, posts, roles, hooks, and other DB state into
later tests and later model runs, producing nondeterministic and
unauditable scores.

- Add run.execution_isolation ('reset_per_test' default, 'none' opt-out)
  and run.execution_concurrency to RunConfig, with validation that
  rejects concurrency > 1 under reset_per_test instead of silently
  sharing state.
- Route BenchmarkRunner, SingleModelRunner (multi-model mode), and
  reference-solution mode through one shared execution loop that
  resets the environment before every execution test.
- Make WordPressEnvironment.reset() restore a usable baseline:
  wp db reset drops all tables, so it now reinstalls core
  (grader.base_url added for the install URL).
- Record the isolation mode in result metadata (runtime_isolation)
  for single-model and multi-model payloads.
- Knowledge tests keep their existing concurrency; they do not touch
  the WordPress runtime.
- Document isolation semantics in README and wp-bench.example.yaml.

Verified: pytest python (41 passed), ruff check python (clean),
mypy python (3 pre-existing trunk errors only).
@lezama
lezama force-pushed the wpbench/01-isolate-runtime-state branch from 602dbc3 to b0cb9dc Compare July 10, 2026 12:27
@lezama

lezama commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

rebased on trunk + green, one fix for #24's required test_function field 👍

@lezama
lezama merged commit d82034a into trunk Jul 10, 2026
1 check passed
@lezama
lezama deleted the wpbench/01-isolate-runtime-state branch July 10, 2026 13:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants