fix(browser): retain post-action refs for chained actions (#252)

## Summary
- Bind the fresh accessibility tree and XPath map after every successful
deterministic browser action.
- Allow required multi-step checks to chain directly from post-action
refs without another observation.
- Remove unused Stagehand action payloads and clear bound refs whenever
the browser runtime is discarded.

## Root Cause
The driver returned a post-action tree but cleared its server-held ref
map. A subsequent action selected a ref from that returned tree and
received a stale-state rejection, leading the agent to observe or reopen
the page repeatedly.

## Architecture
`browser_observe` and successful `browser_act` now produce the same
complete state boundary: bounded tree for the model plus the matching
XPath map retained only inside the sandbox driver. Consuming a ref
invalidates its prior boundary; success atomically replaces it with the
post-action boundary.

## Decisions Made
| Decision | Choice | Alternative | Reasoning |
|---|---|---|---|
| Multi-step state | Bind post-action tree and map atomically | Require
observe after every action | Makes the returned tree truthful and
removes redundant inference/round trips. |
| Action output | Return action identity and bounded state only | Echo
Stagehand result | The raw result is unused and adds token/serialization
cost. |
| Runtime discard | Clear bound observation | Preserve it | Refs cannot
outlive the Chromium runtime that created them. |

## Production Reproduction
- Ordinary prompt: build a grocery list and verify add, complete, and
delete.
- Fresh production run entered Working quickly, then repeated
observe/act/open for more than six minutes because chained refs were
rejected.
- Reproduction chat:
https://trycheatcode.com/chats/019ff5c6-a3c7-7e77-a513-176a478282b3

## Verification
- [x] `pnpm lint`
- [x] `pnpm typecheck`
- [x] `pnpm turbo build --force`
- [x] `pnpm deadcode`
- [x] `pnpm architecture:check`
- [x] `pnpm turbo skills:build`
- [x] `node --check infra/containers/sandbox/browser-driver/server.js`
- [ ] Build, scan, publish, and promote a new immutable Daytona snapshot
- [ ] Repeat ordinary-user web and mobile flows on production

## Related
- Deterministic browser tooling: #250
- Previous snapshot promotion: #251

Author: iamjr15

infra/containers/sandbox/README.md

@@ -158,8 +158,9 @@ low-severity report.
 
 Browser interaction uses Stagehand's native accessibility snapshot followed by deterministic act.
 The driver returns hyphenated page refs while retaining their XPath map server-side, then accepts
-one bounded method/value against a single-use ref from the latest active-page observation. The act
-result includes a fresh post-action tree, so verification needs no second observation inference.
+one bounded method/value against a single-use ref from the latest active-page state. Successful acts
+atomically replace the consumed ref map with the fresh post-action tree and its server-held map, so
+multi-step verification chains without another observation inference.
 Stagehand observation inference, natural-language actions, and self-healing are disabled; browser
 behavior is independent of the selected model's structured-output quirks. Navigation clears the
 observation, and origin interception remains active for deterministic execution.

infra/containers/sandbox/browser-driver/server.js

@@ -458,18 +458,19 @@ async function runGuardedAct(runtime, page, action) {
     originInterceptor = await installOriginInterceptor(stagehand, action.allowedOrigin);
     await originInterceptor.assertHealthy();
     assertExpectedBrowserTarget(page.url(), action.expectedUrl, action.allowedOrigin);
-    const result = await stagehand.act(observedAction, {
+    await stagehand.act(observedAction, {
       page,
       timeout: action.timeoutMs || 10000,
     });
     await originInterceptor.assertHealthy();
     const activePage = await stagehand.context.awaitActivePage();
     assertAllowedBrowserOrigin(activePage.url(), action.allowedOrigin);
+    const observation = await capturePageObservation(activePage);
+    latestObservation = observation.boundary;
     response = {
       result: {
         action: { method: action.action.method, ref: action.action.ref },
-        stagehand: result,
-        state: await capturePageState(activePage),
+        state: observation.state,
       },
       type: action.type,
       url: activePage.url(),
@@ -503,11 +504,6 @@ async function capturePageObservation(page) {
   };
 }
 
-async function capturePageState(page) {
-  const snapshot = await page.snapshot({ includeIframes: true });
-  return boundedSnapshotState(snapshot.formattedTree);
-}
-
 function requireBoundAction(action, pageUrl) {
   const normalized = validateBoundAction(action);
   const observation = latestObservation;
@@ -576,6 +572,7 @@ function snapshotRefMap(tree, xpathMap) {
 }
 
 async function discardBrowserRuntime(runtime) {
+  latestObservation = undefined;
   const pending = browserRuntimePromise;
   browserRuntimePromise = undefined;
   browserRuntimeStatus = "initializing";

packages/agent-core/README.md

@@ -46,9 +46,10 @@ validated before execution. Deterministic prepare/execute boundaries keep dynami
 ports and Git destinations stable between resolution and execution. The managed browser follows
 the same boundary: observation reads Stagehand's native accessibility snapshot without model
 inference and returns page-bound element refs. Execution accepts only a single-use ref from the
-latest observation plus a bounded method/value, resolves its server-held XPath, and returns the
-post-action snapshot. A click or fill therefore cannot invoke a hidden model decision, expose a
-selector, reuse a stale ref, or cross the active origin. The managed
+latest state tree plus a bounded method/value, resolves its server-held XPath, and atomically binds
+the returned post-action tree as the next actionable state. A click or fill therefore cannot
+invoke a hidden model decision, expose a selector, reuse a stale ref, or cross the active origin.
+The managed
 preview tool owns Computer-visible dev servers, remaps a requested port to the
 project’s allocated port when necessary, injects the supported framework binding when the model
 omits it, and is distinct from generic background process tools so idle recovery always has a

packages/agent-core/src/mastra/system-prompt.ts

@@ -206,7 +206,7 @@ Beyond these you also have browser, document-generation, data-analysis, web-rese
 const WEB_MODULE = `## Building web apps
 
 Make the app real and complete: working features, real data flow, considered design. Default to a clean modern stack — React / Next.js. Ship something polished: sensible colour and type, responsive, mobile-first, no lorem ipsum, no dead buttons, no placeholder images. Write the files, add or update dependencies only when the app actually needs them, and start the dev server early with code_start_dev_server (request port 5173) so you're always working against the running app. The managed server restores an unchanged project's existing dependencies itself. Never substitute a shell process for the managed preview, even for a static server or when restarting an existing project.
-Verify it in the browser: open the app's INTERNAL address in the sandbox's headed Chromium — http://localhost:<port> (e.g. http://localhost:5173), NOT the external preview link (your sandbox browser can't reach that). Take one screenshot with the exact visual acceptance criterion; its result includes a visual PASS/FAIL assessment, so do not take another screenshot unless you changed the rendered output. To exercise one representative interaction, call browser_observe once, choose one exact hyphenated element ref from its accessibility tree, and call browser_act with that ref plus the required method/value. browser_act returns the post-action page tree, so use that result to verify the interaction without another observation or extraction. Never invent a ref or selector, write a separate Playwright/Python test, or install another browser. If either check fails, fix the concrete defect and repeat only that changed check once; never loop on equivalent screenshots or interactions. If the browser can't load it at all, note you couldn't visually verify and go straight to your closing summary. The running app is shown to the user automatically in the Computer panel's Browser tab — never paste the preview URL.`;
+Verify it in the browser: open the app's INTERNAL address in the sandbox's headed Chromium — http://localhost:<port> (e.g. http://localhost:5173), NOT the external preview link (your sandbox browser can't reach that). Take one screenshot with the exact visual acceptance criterion; its result includes a visual PASS/FAIL assessment, so do not take another screenshot unless you changed the rendered output. To exercise one representative interaction, call browser_observe once, choose one exact hyphenated element ref from its accessibility tree, and call browser_act with that ref plus the required method/value. browser_act returns an actionable post-action page tree, so use that result to verify the interaction without another observation or extraction. If the request explicitly requires another interaction, chain it from a fresh ref in that returned tree; observe again only after navigation, an external page change, or when the returned tree lacks the required element. Never invent a ref or selector, write a separate Playwright/Python test, or install another browser. If either check fails, fix the concrete defect and repeat only that changed check once; never loop on equivalent screenshots or interactions. If the browser can't load it at all, note you couldn't visually verify and go straight to your closing summary. The running app is shown to the user automatically in the Computer panel's Browser tab — never paste the preview URL.`;
 
 const MOBILE_MODULE = `## Building the mobile app
 
@@ -219,7 +219,7 @@ Build the Expo Router screens for a polished, native-feeling app: real screens,
 // keeps WEB_MODULE's "start the dev server yourself" guidance; this note only applies here.
 const APP_BUILDER_PREVIEW_NOTE = `## Your preview is already running — do not start your own
 
-This project is scaffolded at the workspace root and its dev server + live preview are ALREADY running and managed for you before your turn begins (for a mobile app that's Metro serving the app on web plus the Expo Go QR). Do NOT initialize, scaffold, or create another app or nested project. Do NOT start, restart, or reconfigure the server yourself — no code_start_dev_server, \`expo start\`, \`npm run dev\`/\`web\`, or \`npx expo …\`: a second server fights the managed one for the project's port and breaks the preview. Use pnpm, never npm/npx, only when dependency changes are necessary. Inspect and edit the existing root files; the preview hot-reloads on save. Verify by opening the running app in the sandbox's headed Chromium at its INTERNAL localhost address; it's shown to the user automatically in the Computer/App panel — never paste the preview URL. Metro may briefly show an empty document while rebuilding the first web bundle after edits: wait for page content once and reload at most once before treating it as a defect. Take one screenshot with the exact visual acceptance criterion and use its returned PASS/FAIL assessment; never judge screenshot byte size. Exercise one representative interaction by calling browser_observe once, choosing one exact hyphenated element ref from its accessibility tree, and calling browser_act with that ref plus the required method/value. Use browser_act's post-action tree as the result check; do not observe or extract again. Never invent a ref or selector, write a separate Playwright/Python test, or install another browser. If a check fails, fix the concrete defect and repeat only that changed check once. Once the requested content renders, that interaction passes, and no blocking browser error remains, finish.`;
+This project is scaffolded at the workspace root and its dev server + live preview are ALREADY running and managed for you before your turn begins (for a mobile app that's Metro serving the app on web plus the Expo Go QR). Do NOT initialize, scaffold, or create another app or nested project. Do NOT start, restart, or reconfigure the server yourself — no code_start_dev_server, \`expo start\`, \`npm run dev\`/\`web\`, or \`npx expo …\`: a second server fights the managed one for the project's port and breaks the preview. Use pnpm, never npm/npx, only when dependency changes are necessary. Inspect and edit the existing root files; the preview hot-reloads on save. Verify by opening the running app in the sandbox's headed Chromium at its INTERNAL localhost address; it's shown to the user automatically in the Computer/App panel — never paste the preview URL. Metro may briefly show an empty document while rebuilding the first web bundle after edits: wait for page content once and reload at most once before treating it as a defect. Take one screenshot with the exact visual acceptance criterion and use its returned PASS/FAIL assessment; never judge screenshot byte size. Exercise one representative interaction by calling browser_observe once, choosing one exact hyphenated element ref from its accessibility tree, and calling browser_act with that ref plus the required method/value. Use browser_act's actionable post-action tree as the result check; do not observe or extract again. If the request explicitly requires another interaction, chain it from a fresh ref in the returned tree; observe again only after navigation, an external page change, or when that tree lacks the required element. Never invent a ref or selector, write a separate Playwright/Python test, or install another browser. If a check fails, fix the concrete defect and repeat only that changed check once. Once the requested content renders, that interaction passes, and no blocking browser error remains, finish.`;
 
 const DOCS_MODULE = `## Building documents & slides
 

packages/agent-core/src/mastra/tool-defs/browser-tools.ts

@@ -28,7 +28,7 @@ export const mastraBrowserOpen = createTool({
 export const mastraBrowserAct = createTool({
   id: "browser_act",
   description:
-    "Execute one deterministic action against an exact element ref from the immediately preceding browser_observe tree. The ref is page-bound and single-use; the result includes the post-action page tree.",
+    "Execute one deterministic action against an exact element ref from the latest browser_observe or browser_act tree. The consumed ref is page-bound and single-use; the result includes an actionable post-action page tree for the next step.",
   inputSchema: BrowserActInputSchema,
   outputSchema: BrowserActionsOutputSchema,
   execute: async (input, context) => {

packages/agent-core/src/tools/browser/actions.ts

@@ -52,7 +52,7 @@ const BrowserElementRefSchema = z
   .string()
   .regex(/^\d+-\d+$/u)
   .max(64)
-  .describe("Exact hyphenated element ref from the latest browser_observe tree.");
+  .describe("Exact hyphenated element ref from the latest browser_observe or browser_act tree.");
 
 const BrowserActionMethodSchema = z.enum([
   "click",
@@ -126,7 +126,7 @@ const BrowserBoundActionSchema = z
 
 export const BrowserActInputSchema = z.strictObject({
   action: BrowserBoundActionSchema.describe(
-    "A ref-bound action chosen from the immediately preceding browser_observe tree.",
+    "A ref-bound action chosen from the latest tree returned by browser_observe or browser_act.",
   ),
   timeoutMs: z
     .number()

skills/browser-use/SKILL.md

@@ -42,8 +42,10 @@ examples in this document into `shell_terminal` calls. If `cheatcode-browser` is
 fall back to the native browser tools without creating a project. Open the page, call
 `browser_observe` once, choose one exact hyphenated element ref from its accessibility tree, and
 call `browser_act` with that ref plus the required method/value. Native refs are page-bound and
-single-use, and `browser_act` returns the post-action page tree for verification. Never invent a ref
-or selector, and observe again before any later interaction after a DOM or navigation change.
+single-use, and `browser_act` returns an actionable post-action page tree for verification and any
+required next interaction. Never invent a ref or selector. Chain the next action from the fresh
+post-action refs; observe again only after navigation, a change outside `browser_act`, or when the
+returned tree does not contain the required element.
 
 ## Before You Act — THINK FIRST
 

skills/webapp-testing/SKILL.md

@@ -20,11 +20,14 @@ launch a second browser process, write a Playwright/Python test harness, or star
 3. For one representative functional interaction, call `browser_observe` once. Choose one exact
    hyphenated element ref from its accessibility tree and call `browser_act` with that ref plus the
    required method/value.
-4. Read the post-action tree returned by `browser_act` to decide whether the criterion passed.
+4. Read the post-action tree returned by `browser_act` to decide whether the criterion passed. If
+   the user explicitly requires another interaction, use its fresh ref directly with `browser_act`.
 5. If a criterion fails, fix the concrete app defect and repeat only that changed criterion once.
 
-Observed refs are page-bound and single-use. Observe again before any later interaction after a DOM
-or navigation change. Never invent a ref or selector.
+Each ref is page-bound and single-use. A successful `browser_act` atomically replaces the consumed
+observation with its post-action tree, so chain required interactions from those fresh refs without
+re-observing. Observe again only when the page changed outside `browser_act`, navigated, or the
+returned tree does not contain the required element. Never invent a ref or selector.
 
 Finish as soon as the requested content renders, the representative interaction passes, and no
 blocking browser error remains. Do not repeat equivalent screenshots, actions, or extractions.