fix: isolate WordPress runtime state per execution test - #25
Merged
Conversation
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
Execution tests previously ran concurrently against one shared
WordPress runtime with no reset between tests, so generated code
could leak options, posts, roles, hooks, and other DB state into
later tests and later model runs, producing nondeterministic and
unauditable scores.
- Add run.execution_isolation ('reset_per_test' default, 'none' opt-out)
and run.execution_concurrency to RunConfig, with validation that
rejects concurrency > 1 under reset_per_test instead of silently
sharing state.
- Route BenchmarkRunner, SingleModelRunner (multi-model mode), and
reference-solution mode through one shared execution loop that
resets the environment before every execution test.
- Make WordPressEnvironment.reset() restore a usable baseline:
wp db reset drops all tables, so it now reinstalls core
(grader.base_url added for the install URL).
- Record the isolation mode in result metadata (runtime_isolation)
for single-model and multi-model payloads.
- Knowledge tests keep their existing concurrency; they do not touch
the WordPress runtime.
- Document isolation semantics in README and wp-bench.example.yaml.
Verified: pytest python (41 passed), ruff check python (clean),
mypy python (3 pre-existing trunk errors only).
lezama
force-pushed
the
wpbench/01-isolate-runtime-state
branch
from
July 10, 2026 12:27
602dbc3 to
b0cb9dc
Compare
Contributor
|
rebased on trunk + green, one fix for #24's required test_function field 👍 |
This was referenced Jul 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why this matters
WP-Bench scores are only meaningful if they're reproducible. Today, execution tests run concurrently against one shared WordPress instance with no reset between tests. WordPress is deeply stateful — options, posts, roles, hooks, transients, cron events all persist in the database — so code generated for one test can silently change the outcome of the next. In multi-model mode the problem compounds: one model's leftover state can help or hurt the next model's scores. That means rankings can shift based on test ordering and scheduling luck, and a provider who re-runs the suite may get different numbers. No leaderboard can be trusted on that foundation. This change makes every execution test start from a known-clean WordPress baseline, which is the prerequisite for every scoring and auditability improvement that follows.
Changes
RunConfig: newexecution_isolation(reset_per_testdefault |none) andexecution_concurrency(default 1). A model validator rejectsexecution_concurrency > 1underreset_per_testwith a clear error instead of silently sharing state.core.py: single-model, multi-model, and reference-solution execution paths now route through one shared_run_isolated_execution_loop()that callsenvironment.reset()before every test. Knowledge tests keep their existing concurrency (they never touch the WordPress runtime).WordPressEnvironment.reset():wp db resetdrops all tables and leaves WordPress uninstalled, so reset now follows withwp core installto return to a deterministic just-installed baseline (grader.base_urladded for the install URL).metadata.runtime_isolationrecords the mode used, in both single- and multi-model payloads, so any results file shows how it was produced.wp-bench.example.yamldocument the isolation semantics and the official-run requirement.Verification
pytest python: 41 passed (8 new isolation tests: reset-before-every-test in all three modes, reset between models, concurrency rejection, metadata recording, legacy opt-out)ruff check python: cleanmypy python: 3 pre-existing trunk errors only (none introduced)Notes for reviewers
execution_isolationconfig surface leaves room for a futurecontainer_per_workermode.