Skip to content

Site Editor Benchmark: add custom WordPress environment support#2614

Open
gcsecsey wants to merge 17 commits intotrunkfrom
stu-1305-studio-benchmark-local-sites-for-comparison
Open

Site Editor Benchmark: add custom WordPress environment support#2614
gcsecsey wants to merge 17 commits intotrunkfrom
stu-1305-studio-benchmark-local-sites-for-comparison

Conversation

@gcsecsey
Copy link
Contributor

@gcsecsey gcsecsey commented Feb 19, 2026

Related issues

Related to STU-1305
Follow up to #2600

Proposed Changes

Adds the ability to benchmark any running WordPress site, not just Studio and Playground environments. This PR:

  • Adds a repeatable --custom flag. A running WordPress site can be added to the benchmark run via --custom=<name>,<url>[,<user>,<password>]. The flag is repeatable, so multiple custom sites can be compared in a single run (eg. bare vs with plugins). Username defaults to admin, password to password.
  • Adds an --install-plugins flag. This installs the plugins in the plugins-blueprint.json file using the standard WordPress REST API (cookie auth + nonce). --install-plugins installs plugins for all custom environments, or --install-plugins=<name1>,<name2> targets specific ones.
  • Updates the navigation to handle setup wizard redirects. Custom environments log in via the WordPress login form, then navigate to /wp-admin/ explicitly.

Examples

# Benchmark a single custom site
npm run benchmark -- --custom=my-site,http://localhost:10003

# Compare bare vs. with plugins
npm run benchmark -- \
  --custom=custom-bare,http://localhost:10003 \
  --custom=custom-plugins,http://localhost:10004 \
  --install-plugins=custom-plugins

# Compare Studio vs. a custom site
npm run benchmark -- --only=studio,my-site \
  --custom=my-site,http://localhost:10003 --rounds=3

Testing Instructions

  • Set up a WordPress site (Local, Docker, MAMP, etc.) with an admin account
  • Run a bare custom benchmark:
    cd tools/benchmark-site-editor
    npm install
    npm run benchmark -- --skip-studio --skip-playground-cli --skip-playground-web \
      --custom=my-site,http://localhost:10003
  • Check that the benchmark completes and prints results
  • Run with --install-plugins and verify plugins are installed
  • Run again to verify already-installed plugins are skipped
  • Run with multiple --custom flags to verify multiple sites work
  • Verify existing environments still work: npm run benchmark -- --only=studio

Pre-merge Checklist

  • TypeScript compilation clean
  • Plugin installation works via REST API with proper activation hooks
  • Teardown guaranteed even on plugin install failure
  • Local environments excluded by default

Implements automated benchmarking across a 10-environment matrix to measure site editor performance impact of having 10 plugins installed. Includes orchestration script that:
- Sets up isolated Studio, Playground CLI, and Playground Web environments
- Supports bare, multi-worker, and plugin variants
- Installs 10 popular plugins via reusable blueprint
- Runs site-editor-benchmark test against each environment
- Outputs comparison table showing performance metrics across all variants

Files:
- metrics/tests/site-editor-benchmark.test.ts: Playwright benchmark test (from PR #2368)
- scripts/benchmark-site-editor/benchmark.ts: Main orchestration script with environment setup
- scripts/benchmark-site-editor/plugins-blueprint.json: Blueprint with 10 plugins
- scripts/benchmark-site-editor/{package.json,tsconfig.json,README.md}: Script configuration
Replace WP-CLI binary approach with HTTP-based REST API for cross-platform
simplicity and proper activation hook firing. Also improves login flow timeout
handling for Local sites and ensures teardown cleanup even if plugin install fails.

- install-plugins-local.ts: Use cookie-based auth + nonce extraction + POST /wp-json/wp/v2/plugins
- benchmark.ts: Wrap plugin install in try/catch, fix variable ordering, improve teardown guarantees
- measure-site-editor.ts: Use Promise.all for click+navigation to prevent timeout on redirects
@gcsecsey gcsecsey changed the title Add Local by Flywheel as benchmark environment (STU-1290) Add Local by Flywheel as benchmark environment Feb 19, 2026
@gcsecsey gcsecsey requested review from a team and fredrikekelund February 19, 2026 12:58
@gcsecsey gcsecsey marked this pull request as ready for review February 19, 2026 12:58
@gcsecsey gcsecsey marked this pull request as draft February 20, 2026 11:11
…port

Remove the Local GraphQL client and site lifecycle management (~500 lines).
Instead, accept any running WordPress site via repeatable --custom flags
with URL and credentials. Plugin installation uses the standard WP REST API
and gracefully skips already-installed plugins.
@gcsecsey gcsecsey changed the title Add Local by Flywheel as benchmark environment Add custom WordPress environment support to site editor benchmark Feb 20, 2026
@gcsecsey gcsecsey changed the title Add custom WordPress environment support to site editor benchmark Site Editor Benchmark: add custom WordPress environment support Feb 20, 2026
@gcsecsey gcsecsey marked this pull request as ready for review February 23, 2026 10:45
@wpmobilebot
Copy link
Collaborator

📊 Performance Test Results

Comparing e828ddc vs trunk

site-editor

Metric trunk e828ddc Diff Change
load 1436.00 ms 1417.00 ms -19.00 ms ⚪ 0.0%

site-startup

Metric trunk e828ddc Diff Change
siteCreation 7093.00 ms 7086.00 ms -7.00 ms ⚪ 0.0%
siteStartup 3932.00 ms 3937.00 ms +5.00 ms ⚪ 0.0%

Results are median values from multiple test runs.

Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants