feat: add wp_plugin_files artifact execution mode - #35
Merged
Conversation
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
lezama
force-pushed
the
wpbench/10-stdin-payload
branch
from
July 10, 2026 15:54
a07567e to
2255969
Compare
Execution tests could only evaluate single PHP snippets passed to eval, but real WordPress work is multi-file: plugin headers, includes, activation hooks, filesystem layout. The config already declared artifact kinds; this implements the first one beyond snippets. Python harness: - New wp_bench/artifacts.py: parse_artifact() turns completions into validated artifacts. php_snippet keeps existing fence-stripping; wp_plugin_files expects a JSON files map and validates paths (no absolute, no traversal, safe characters), sizes (256KB/file, 1MB total, 20 files), and presence of a top-level Plugin Name header. - ExecutionTest gains artifact_kind (default php_snippet) and reference_files; local parser, HF loader, and Parquet export carry both. - Execution prompts render kind-specific output instructions. - Parse/validation failures become scored per-test failures (execution_pass false, artifact_error recorded); the run continues. - Reference-solution mode runs reference_files for plugin tests. - WordPressEnvironment.execute_artifact() ships artifact_kind and files in the verifier payload; execute_code() remains for snippets. PHP runtime: - New Artifact_Installer writes candidate files into an isolated wp-content/plugins/wp-bench-candidate-* directory (re-validating paths server-side), loads the main plugin file, and removes the directory after verification. - Verifier routes on artifact_kind: plugin files are installed before assertions run; static analysis covers all files concatenated; install failures return a structured artifact_install_error result. - Per-test isolation resets the environment around each test, so candidate plugin state cannot leak. Verified: pytest python (122 passed, 17 new artifact tests: parsing, fenced JSON, traversal/absolute/backslash rejection, size and count limits, main-file requirement, unknown kinds, runner integration, parse-failure continuation, reference files, prompt rendering), php -l on all three runtime files, ruff clean, mypy (2 pre-existing trunk errors only).
lezama
force-pushed
the
wpbench/11-artifact-modes
branch
from
July 10, 2026 16:15
542ec74 to
18e5ecc
Compare
Contributor
|
rebased on trunk (merged a test_function docstring conflict), 124 unit tests green. since this writes model files to disk i reviewed both validation layers + booted the runtime and attacked it directly: valid plugin installs+loads fine, path-traversal and absolute-path payloads are rejected server-side with nothing written, and candidate dirs are cleaned up (the finally matters, #25's reset is DB-only so it wouldn't catch orphaned files). snippet path unregressed. security boundary holds! |
lezama
pushed a commit
that referenced
this pull request
Jul 10, 2026
## Why this matters
The config layer is the first thing every contributor and every run
touches, and right now it greets them with a deprecation warning on
literally every invocation — the project pins `pydantic>=2.8` but still
uses the v1 `@validator` API, which Pydantic removes entirely in v3.
Beyond the noise (which trains people to ignore warnings, exactly what a
benchmark repo can't afford), it's a time bomb: the first dependency
bump to Pydantic v3 would break config loading outright. This clears the
debt and uses the opportunity to tighten validation where invalid values
previously slipped through to fail confusingly mid-run.
## Changes
- `@validator('temperature')` → `@field_validator` + `@classmethod` (v2
idiom).
- New `top_p` validator (must be 0–1 or unset) — previously any float
was accepted and failed provider-side with a cryptic API error.
- Declarative `Field` constraints on numerics that must be positive:
`request_timeout`, `timeout_seconds`, `setup_timeout_seconds`,
`concurrency`, and `limit` — a `limit: 0` or zero timeout now fails at
config load with a clear message instead of producing a silently empty
or hanging run.
- **Regression guard**: a test constructs every config model under
`warnings.simplefilter('error', DeprecationWarning)`, so any future
deprecated-API usage in the config layer fails CI rather than
accumulating.
## Verification
- `pytest python`: **131 passed** (9 new validation tests) — and the
`PydanticDeprecatedSince20` warning that appeared in every previous test
run is gone
- `ruff check python`: clean
- `mypy python`: 2 pre-existing trunk errors only
- No v1 `validator` imports remain anywhere in the package
## Notes
- Stacked on #35.
- Constraint tightening is intentionally conservative — only bounds that
were already documented or obviously nonsensical to violate.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why this matters
WP-Bench wants to answer 'which model is best at WordPress work' — but real WordPress work isn't isolated named-function snippets fed to
eval. It's plugins with headers and file layout, activation hooks, includes, and eventually blocks and themes. A benchmark that only tests snippet-writing measures a narrow (and increasingly unrepresentative) slice of what people actually ask models to build, and models that are strong at structuring real projects get no credit for it. This adds the first realistic artifact mode: multi-file plugin generation, installed and verified in a live WordPress. It also establishes the parsing/validation/install pipeline that block, theme, and patch tracks can reuse.Since candidate artifacts are now written to the runtime filesystem, this is also a security boundary: everything is validated twice (harness and runtime), confined to a dedicated directory, size-capped, and cleaned up.
Changes
Harness
php_snippet(default, unchanged behavior) andwp_plugin_files, where the model must return a JSONfilesmap. Validation rejects absolute paths,../backslash traversal, hostile characters, oversized files (256KB), oversized artifacts (1MB / 20 files), and artifacts missing a top-levelPlugin Name:header.artifact_kind(plusreference_filesfor reference-solution runs); dataset loaders and the Parquet export carry both, and prompts render kind-specific output instructions.execution_pass: falsewith the artifact error in the grader payload, and the run continues — a model that can't produce a valid plugin has failed the task.Runtime
Artifact_Installerwrites candidate files into an isolatedwp-content/plugins/wp-bench-candidate-*directory (independently re-validating every path server-side — defense in depth against a compromised or buggy harness), loads the main plugin file, and deletes the directory after verification.Verifierroutes onartifact_kind: plugin files are installed before assertions run; static analysis covers all files concatenated; install failures come back as structuredartifact_install_errorresults.Verification
pytest python: 122 passed (17 new: parsing incl. fenced JSON, traversal/absolute/backslash rejection, size/count caps, main-file requirement, unknown-kind rejection, end-to-end runner integration, parse-failure continuation across tests, reference-files execution, kind-specific prompts)php -lon all touched runtime files: cleanruff check python datasets: cleanmypy python: 2 pre-existing trunk errors onlyNotes
wp_plugin_filesonly;block_plugin,wp_theme_files,js_module, andpatchreuse this pipeline in future PRs.