evals: benchmark matrix (model × harness × tool_surface) + Braintrust routing + STAGEHAND_V4_SDK_PATH

[shriyatheunicorn]

Part of STG-2671 (cadence: model × harness × tool_surface; STAGEHAND_V4_SDK_PATH; report to Braintrust).

Encodes the benchmark cadence as data and wires attribution end to end.

What's inside

• benchmarks/ — manifest schema + expansion with validity rules (invalid combinations reported with reasons, never silently dropped); v4-vs-playwright manifest; GENERAL_HARNESS.md design doc; grading-decision README (v3-carried rubric verifier kept deliberately — grading only, never driving)
• framework/runner.ts — project routing: deterministic --sdk runs → stagehand-v4-deterministic, external-harness & benchmark-matrix runs → stagehand-v4; experiment names/metadata carry the (benchmark, harness, toolSurface) triple; sdk metadata inferred from the tool surface
• v4SdkLoader.ts — STAGEHAND_V4_SDK_PATH: dynamic import of the v4 SDK entry from a configured checkout (package link fallback, loud on misconfiguration)
• initV4.ts — anthropic browser-CORS header via model config (no SDK patch required); pure buildV4InitParams for browserless unit coverage; systemPrompt passthrough

Verification

• 15 unit tests (SDK-path resolution, importer behavior, key resolution incl. google aliases, CORS header, env mapping)
• Live: experiments verified in both Braintrust projects with full triple metadata; anthropic act verified against a pristine (unpatched) extension with the config-header path

Stacked on stg-2671-v4-harness.

---

Summary by cubic

Implements the benchmark matrix (model × harness × toolSurface) with end-to-end Braintrust routing/metadata, and makes the v4 SDK a configurable import via STAGEHAND_V4_SDK_PATH with committed types. Part of STG-2671.

• New Features

  • Benchmarks: schema + expansion with validity rules; invalid combos reported; includes v4-vs-playwright example.
  • Braintrust/routing: experiment names include the full triple; metadata stamps { benchmark, harness, toolSurface, sdk, v4Sha }; external-harness and matrix runs route to stagehand-v4, deterministic --sdk runs to stagehand-v4-deterministic; v4Sha resolved from the configured checkout.
  • V4 SDK: dynamic import from STAGEHAND_V4_SDK_PATH (no fallback); removed @browserbasehq/stagehand-v4-spike-sdk-ts link; committed .v4-sdk-types/ so typecheck works on fresh clones/CI.
  • V4 init: buildV4InitParams (unit-testable); Anthropic browser CORS header via model config; systemPrompt passthrough; SDK onLog bridged to the eval logger.
  • Tests: coverage for manifest expansion, Braintrust naming/metadata, v4 init params (incl. Google key aliases), and SDK path resolution/loader.

• Migration

  • Set STAGEHAND_V4_SDK_PATH to the v4 SDK entry (e.g. /packages/sdk-ts/src/index.ts). Regenerate and commit .v4-sdk-types/ if the SDK entry moves.

^{Written for commit 301f858. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 301f858

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR