fix: make browser verification model-independent (#250)

## Summary
- replace provider-backed browser observation with Stagehand native
accessibility snapshots
- keep XPath mappings inside the sandbox driver and expose only
page-bound, single-use refs
- return the post-action page tree so functional verification needs no
second inference call
- align web/mobile prompts, tool descriptions, capabilities, skills, and
architecture docs

## Architecture
The driver snapshots the active page locally, exposes bounded hyphenated
refs, and retains the ref-to-XPath map in memory. browser_act accepts
one ref plus a validated method/value, enforces URL and origin binding,
consumes the observation, executes deterministically with self-healing
disabled, and returns a fresh state tree.

## Decisions
- Native snapshot instead of changing observation models: removes
provider-specific structured-output failure and latency for every agent
model.
- Server-held selectors instead of returning XPath: prevents selector
invention and keeps the trust boundary inside the driver.
- Post-action tree in the act response instead of a second extract:
makes the acceptance flow bounded and deterministic.

## Verification
- pnpm lint
- pnpm typecheck
- pnpm turbo build --force
- pnpm deadcode
- pnpm architecture:check
- pnpm turbo skills:build
- node --check infra/containers/sandbox/browser-driver/server.js

Production browser QA follows after the sandbox image is built and
promoted.

Author: iamjr15

infra/containers/sandbox/README.md

@@ -156,11 +156,13 @@ suggested breaking Stagehand downgrade. Static checks fail on
 moderate-or-higher findings across every sandbox lock without hiding this
 low-severity report.
 
-Browser interaction uses Stagehand's observe-then-act boundary. The driver stores the exact
-actions from the latest observation for the active page and executes only an unchanged, single-use
-match. Direct natural-language actions and Stagehand self-healing are disabled so an interaction
-cannot trigger a second hidden model decision or keep retrying a stale selector. Navigation clears
-the observation, and origin interception remains active for the deterministic execution.
+Browser interaction uses Stagehand's native accessibility snapshot followed by deterministic act.
+The driver returns hyphenated page refs while retaining their XPath map server-side, then accepts
+one bounded method/value against a single-use ref from the latest active-page observation. The act
+result includes a fresh post-action tree, so verification needs no second observation inference.
+Stagehand observation inference, natural-language actions, and self-healing are disabled; browser
+behavior is independent of the selected model's structured-output quirks. Navigation clears the
+observation, and origin interception remains active for deterministic execution.
 
 Snapshot publication builds and scans the exact local AMD64 image before pushing it
 to Daytona. Trivy fails on every fixable medium-or-higher vulnerability and every

infra/containers/sandbox/browser-driver/server.js

@@ -19,26 +19,56 @@ const MAX_PROVIDER_RESPONSE_BYTES = 16 * 1024 * 1024;
 // An 8 MiB PNG expands to roughly 10.7 MiB as base64 before JSON framing.
 const MAX_RESPONSE_BYTES = 12 * 1024 * 1024;
 const MAX_SCREENSHOT_BYTES = 8 * 1024 * 1024;
+const MAX_SNAPSHOT_TREE_BYTES = 140 * 1024;
 const REQUEST_BODY_TIMEOUT_MS = 30 * 1000;
-const ObservedActionSchema = z.strictObject({
-  arguments: z.array(z.string().max(2_000)).max(10).optional(),
-  backendNodeId: z.number().int().positive().optional(),
-  description: z.string().min(1).max(2_000),
-  method: z.enum([
-    "click",
-    "doubleClick",
-    "dragAndDrop",
-    "fill",
-    "hover",
-    "nextChunk",
-    "press",
-    "prevChunk",
-    "scrollTo",
-    "selectOptionFromDropdown",
-    "type",
-  ]),
-  selector: z.string().min(1).max(4_096).startsWith("xpath="),
-});
+const BrowserElementRefSchema = z.string().regex(/^\d+-\d+$/u).max(64);
+const BrowserActionMethodSchema = z.enum([
+  "click",
+  "doubleClick",
+  "dragAndDrop",
+  "fill",
+  "hover",
+  "nextChunk",
+  "press",
+  "prevChunk",
+  "scrollTo",
+  "selectOptionFromDropdown",
+  "type",
+]);
+const BROWSER_VALUE_METHODS = new Set([
+  "fill",
+  "press",
+  "scrollTo",
+  "selectOptionFromDropdown",
+  "type",
+]);
+const BoundActionSchema = z.strictObject({
+  ref: BrowserElementRefSchema,
+  targetRef: BrowserElementRefSchema.optional(),
+  value: z.string().max(2_000).optional(),
+  method: BrowserActionMethodSchema,
+}).superRefine(validateBoundActionShape);
+
+function validateBoundActionShape(action, context) {
+  const needsValue = BROWSER_VALUE_METHODS.has(action.method);
+  if (needsValue !== (action.value !== undefined)) {
+    context.addIssue({
+      code: "custom",
+      message: needsValue ? `${action.method} requires value` : `${action.method} rejects value`,
+      path: ["value"],
+    });
+  }
+  const needsTarget = action.method === "dragAndDrop";
+  if (needsTarget !== (action.targetRef !== undefined)) {
+    context.addIssue({
+      code: "custom",
+      message: needsTarget
+        ? "dragAndDrop requires targetRef"
+        : `${action.method} rejects targetRef`,
+      path: ["targetRef"],
+    });
+  }
+}
 const bootstrap = await readBootstrapConfig();
 const PORT = bootstrap.port;
 const MODEL_NAME = bootstrap.modelName;
@@ -393,12 +423,9 @@ async function runAction(runtime, action) {
     return runGuardedAct(runtime, page, action);
   }
   if (action.type === "observe") {
-    const result = await stagehand.observe(action.instruction, { page });
-    latestObservation = {
-      actions: result.map(normalizeObservedAction),
-      url: page.url(),
-    };
-    return { result, type: action.type, url: page.url() };
+    const observation = await capturePageObservation(page);
+    latestObservation = observation.boundary;
+    return { result: observation.state, type: action.type, url: page.url() };
   }
   if (action.type === "extract") {
     const result = await stagehand.extract(action.instruction, { page });
@@ -422,7 +449,7 @@ async function runAction(runtime, action) {
 async function runGuardedAct(runtime, page, action) {
   const { stagehand } = runtime;
   assertExpectedBrowserTarget(page.url(), action.expectedUrl, action.allowedOrigin);
-  const observedAction = requireObservedAction(action.action, page.url());
+  const observedAction = requireBoundAction(action.action, page.url());
   latestObservation = undefined;
   let failure;
   let originInterceptor;
@@ -438,7 +465,15 @@ async function runGuardedAct(runtime, page, action) {
     await originInterceptor.assertHealthy();
     const activePage = await stagehand.context.awaitActivePage();
     assertAllowedBrowserOrigin(activePage.url(), action.allowedOrigin);
-    response = { result, type: action.type, url: activePage.url() };
+    response = {
+      result: {
+        action: { method: action.action.method, ref: action.action.ref },
+        stagehand: result,
+        state: await capturePageState(activePage),
+      },
+      type: action.type,
+      url: activePage.url(),
+    };
   } catch (error) {
     failure = error;
   }
@@ -456,32 +491,88 @@ async function runGuardedAct(runtime, page, action) {
   return response;
 }
 
-function normalizeObservedAction(action) {
-  const parsed = ObservedActionSchema.safeParse(action);
-  if (!parsed.success) {
-    throw new Error("Browser observation returned an invalid action");
-  }
-  return parsed.data;
+async function capturePageObservation(page) {
+  const snapshot = await page.snapshot({ includeIframes: true });
+  const state = boundedSnapshotState(snapshot.formattedTree);
+  return {
+    boundary: {
+      refs: snapshotRefMap(state.tree, snapshot.xpathMap),
+      url: page.url(),
+    },
+    state,
+  };
 }
 
-function requireObservedAction(action, pageUrl) {
-  const normalized = validateObservedAction(action);
+async function capturePageState(page) {
+  const snapshot = await page.snapshot({ includeIframes: true });
+  return boundedSnapshotState(snapshot.formattedTree);
+}
+
+function requireBoundAction(action, pageUrl) {
+  const normalized = validateBoundAction(action);
   const observation = latestObservation;
-  if (
-    !observation ||
-    observation.url !== pageUrl ||
-    !observation.actions.some((candidate) => actionsEqual(candidate, normalized))
-  ) {
+  if (!observation || observation.url !== pageUrl) {
     throw new RequestError(
       409,
-      "Browser action was not returned by the latest observation for this page",
+      "Browser action is not bound to the latest observation for this page",
     );
   }
-  return normalized;
+  const selector = observedSelector(observation.refs, normalized.ref);
+  const args = browserActionArguments(observation.refs, normalized);
+  return {
+    arguments: args,
+    description: `${normalized.method} observed element [${normalized.ref}]`,
+    method: normalized.method,
+    selector,
+  };
+}
+
+function observedSelector(refs, ref) {
+  const xpath = refs.get(ref);
+  if (!xpath) {
+    throw new RequestError(409, "Browser action ref is absent from the latest observation");
+  }
+  const selector = xpath.startsWith("xpath=") ? xpath : `xpath=${xpath}`;
+  if (selector.length > 4_096) {
+    throw new RequestError(409, "Browser action ref resolves to an invalid selector");
+  }
+  return selector;
+}
+
+function browserActionArguments(refs, action) {
+  if (action.method === "dragAndDrop") {
+    return [observedSelector(refs, action.targetRef)];
+  }
+  return action.value === undefined ? [] : [action.value];
 }
 
-function actionsEqual(left, right) {
-  return JSON.stringify(left) === JSON.stringify(right);
+function boundedSnapshotState(formattedTree) {
+  const tree = String(formattedTree ?? "");
+  if (Buffer.byteLength(tree) <= MAX_SNAPSHOT_TREE_BYTES) {
+    return { tree, truncated: false };
+  }
+  const suffix = "\n[Snapshot truncated: narrow the page state before observing again]";
+  const selected = [];
+  let byteLength = Buffer.byteLength(suffix);
+  for (const line of tree.split("\n")) {
+    const lineBytes = Buffer.byteLength(line) + (selected.length > 0 ? 1 : 0);
+    if (byteLength + lineBytes > MAX_SNAPSHOT_TREE_BYTES) break;
+    selected.push(line);
+    byteLength += lineBytes;
+  }
+  return { tree: `${selected.join("\n")}${suffix}`, truncated: true };
+}
+
+function snapshotRefMap(tree, xpathMap) {
+  const refs = new Map();
+  for (const match of tree.matchAll(/^\s*\[(\d+-\d+)\]/gmu)) {
+    const ref = match[1];
+    const xpath = xpathMap[ref];
+    if (typeof xpath === "string" && xpath.length > 0 && xpath.length <= 4_090) {
+      refs.set(ref, xpath);
+    }
+  }
+  return refs;
 }
 
 async function discardBrowserRuntime(runtime) {
@@ -587,7 +678,7 @@ function validateAction(action) {
     return action;
   }
   if (action.type === "act") {
-    action.action = validateObservedAction(action.action);
+    action.action = validateBoundAction(action.action);
     const expectedUrl = assertHttpUrl(action.expectedUrl);
     const allowedOrigin = assertHttpUrl(action.allowedOrigin);
     if (allowedOrigin.href !== `${allowedOrigin.origin}/` || expectedUrl.origin !== allowedOrigin.origin) {
@@ -601,7 +692,10 @@ function validateAction(action) {
     }
     return action;
   }
-  if (action.type === "observe" || action.type === "extract") {
+  if (action.type === "observe") {
+    return action;
+  }
+  if (action.type === "extract") {
     assertInstruction(action.instruction);
     return action;
   }
@@ -614,10 +708,10 @@ function validateAction(action) {
   throw new RequestError(400, "Browser action type is unsupported");
 }
 
-function validateObservedAction(action) {
-  const parsed = ObservedActionSchema.safeParse(action);
+function validateBoundAction(action) {
+  const parsed = BoundActionSchema.safeParse(action);
   if (!parsed.success) {
-    throw new RequestError(400, "Observed browser action is invalid");
+    throw new RequestError(400, "Ref-bound browser action is invalid");
   }
   return parsed.data;
 }

packages/agent-core/README.md

@@ -43,11 +43,12 @@ Tools execute autonomously inside the active request context. Sandbox operations
 remain project-root confined, browser actions remain origin-bound, connected-app
 actions remain scoped to the user's active account, and secret-bearing input is
 validated before execution. Deterministic prepare/execute boundaries keep dynamic
-ports and Git destinations stable between resolution and execution. The managed
-browser follows the same boundary: observation returns page-bound executable actions,
-and execution accepts only an unchanged action from the latest observation. Natural-language
-inference therefore runs once during observation; a click or fill cannot silently invoke another
-model decision, reuse a stale selector, or cross the active origin. The managed
+ports and Git destinations stable between resolution and execution. The managed browser follows
+the same boundary: observation reads Stagehand's native accessibility snapshot without model
+inference and returns page-bound element refs. Execution accepts only a single-use ref from the
+latest observation plus a bounded method/value, resolves its server-held XPath, and returns the
+post-action snapshot. A click or fill therefore cannot invoke a hidden model decision, expose a
+selector, reuse a stale ref, or cross the active origin. The managed
 preview tool owns Computer-visible dev servers, remaps a requested port to the
 project’s allocated port when necessary, injects the supported framework binding when the model
 omits it, and is distinct from generic background process tools so idle recovery always has a

packages/agent-core/src/mastra/system-prompt.ts

@@ -206,7 +206,7 @@ Beyond these you also have browser, document-generation, data-analysis, web-rese
 const WEB_MODULE = `## Building web apps
 
 Make the app real and complete: working features, real data flow, considered design. Default to a clean modern stack — React / Next.js. Ship something polished: sensible colour and type, responsive, mobile-first, no lorem ipsum, no dead buttons, no placeholder images. Write the files, add or update dependencies only when the app actually needs them, and start the dev server early with code_start_dev_server (request port 5173) so you're always working against the running app. The managed server restores an unchanged project's existing dependencies itself. Never substitute a shell process for the managed preview, even for a static server or when restarting an existing project.
-Verify it in the browser: open the app's INTERNAL address in the sandbox's headed Chromium — http://localhost:<port> (e.g. http://localhost:5173), NOT the external preview link (your sandbox browser can't reach that). Take one screenshot with the exact visual acceptance criterion; its result includes a visual PASS/FAIL assessment, so do not take another screenshot unless you changed the rendered output. To exercise one representative interaction, call browser_observe once for that exact interaction, pass one returned action unchanged to browser_act, then read the resulting state once. Never send prose directly to browser_act, invent selectors, write a separate Playwright/Python test, or install another browser. If either check fails, fix the concrete defect and repeat only that changed check once; never loop on equivalent screenshots or interactions. If the browser can't load it at all, note you couldn't visually verify and go straight to your closing summary. The running app is shown to the user automatically in the Computer panel's Browser tab — never paste the preview URL.`;
+Verify it in the browser: open the app's INTERNAL address in the sandbox's headed Chromium — http://localhost:<port> (e.g. http://localhost:5173), NOT the external preview link (your sandbox browser can't reach that). Take one screenshot with the exact visual acceptance criterion; its result includes a visual PASS/FAIL assessment, so do not take another screenshot unless you changed the rendered output. To exercise one representative interaction, call browser_observe once, choose one exact hyphenated element ref from its accessibility tree, and call browser_act with that ref plus the required method/value. browser_act returns the post-action page tree, so use that result to verify the interaction without another observation or extraction. Never invent a ref or selector, write a separate Playwright/Python test, or install another browser. If either check fails, fix the concrete defect and repeat only that changed check once; never loop on equivalent screenshots or interactions. If the browser can't load it at all, note you couldn't visually verify and go straight to your closing summary. The running app is shown to the user automatically in the Computer panel's Browser tab — never paste the preview URL.`;
 
 const MOBILE_MODULE = `## Building the mobile app
 
@@ -219,7 +219,7 @@ Build the Expo Router screens for a polished, native-feeling app: real screens,
 // keeps WEB_MODULE's "start the dev server yourself" guidance; this note only applies here.
 const APP_BUILDER_PREVIEW_NOTE = `## Your preview is already running — do not start your own
 
-This project is scaffolded at the workspace root and its dev server + live preview are ALREADY running and managed for you before your turn begins (for a mobile app that's Metro serving the app on web plus the Expo Go QR). Do NOT initialize, scaffold, or create another app or nested project. Do NOT start, restart, or reconfigure the server yourself — no code_start_dev_server, \`expo start\`, \`npm run dev\`/\`web\`, or \`npx expo …\`: a second server fights the managed one for the project's port and breaks the preview. Use pnpm, never npm/npx, only when dependency changes are necessary. Inspect and edit the existing root files; the preview hot-reloads on save. Verify by opening the running app in the sandbox's headed Chromium at its INTERNAL localhost address; it's shown to the user automatically in the Computer/App panel — never paste the preview URL. Metro may briefly show an empty document while rebuilding the first web bundle after edits: wait for page content once and reload at most once before treating it as a defect. Take one screenshot with the exact visual acceptance criterion and use its returned PASS/FAIL assessment; never judge screenshot byte size. Exercise one representative interaction by calling browser_observe once, passing one returned action unchanged to browser_act, and reading the resulting state once. Never write a separate Playwright/Python test or install another browser. If a check fails, fix the concrete defect and repeat only that changed check once. Once the requested content renders, that interaction passes, and no blocking browser error remains, finish.`;
+This project is scaffolded at the workspace root and its dev server + live preview are ALREADY running and managed for you before your turn begins (for a mobile app that's Metro serving the app on web plus the Expo Go QR). Do NOT initialize, scaffold, or create another app or nested project. Do NOT start, restart, or reconfigure the server yourself — no code_start_dev_server, \`expo start\`, \`npm run dev\`/\`web\`, or \`npx expo …\`: a second server fights the managed one for the project's port and breaks the preview. Use pnpm, never npm/npx, only when dependency changes are necessary. Inspect and edit the existing root files; the preview hot-reloads on save. Verify by opening the running app in the sandbox's headed Chromium at its INTERNAL localhost address; it's shown to the user automatically in the Computer/App panel — never paste the preview URL. Metro may briefly show an empty document while rebuilding the first web bundle after edits: wait for page content once and reload at most once before treating it as a defect. Take one screenshot with the exact visual acceptance criterion and use its returned PASS/FAIL assessment; never judge screenshot byte size. Exercise one representative interaction by calling browser_observe once, choosing one exact hyphenated element ref from its accessibility tree, and calling browser_act with that ref plus the required method/value. Use browser_act's post-action tree as the result check; do not observe or extract again. Never invent a ref or selector, write a separate Playwright/Python test, or install another browser. If a check fails, fix the concrete defect and repeat only that changed check once. Once the requested content renders, that interaction passes, and no blocking browser error remains, finish.`;
 
 const DOCS_MODULE = `## Building documents & slides
 

packages/agent-core/src/mastra/tool-defs/browser-tools.ts

@@ -28,7 +28,7 @@ export const mastraBrowserOpen = createTool({
 export const mastraBrowserAct = createTool({
   id: "browser_act",
   description:
-    "Execute one exact action returned by the immediately preceding browser_observe call. Pass the returned action object unchanged; invented or stale actions are rejected.",
+    "Execute one deterministic action against an exact element ref from the immediately preceding browser_observe tree. The ref is page-bound and single-use; the result includes the post-action page tree.",
   inputSchema: BrowserActInputSchema,
   outputSchema: BrowserActionsOutputSchema,
   execute: async (input, context) => {
@@ -49,7 +49,7 @@ export const mastraBrowserAct = createTool({
 export const mastraBrowserObserve = createTool({
   id: "browser_observe",
   description:
-    "Find executable actions for one explicit interaction in the current sandbox browser page. Select one returned action and pass it unchanged to browser_act.",
+    "Read the current sandbox page as a deterministic accessibility tree with page-bound element refs. Choose an exact hyphenated ref from the tree for browser_act; no secondary model is invoked.",
   inputSchema: BrowserObserveInputSchema,
   outputSchema: BrowserActionsOutputSchema,
   execute: async (input, context) =>