fix(replay): port the engine to browser-use 0.13's CDP actor API (deterministic replay was 100% dead)

[Sangaibisi]

Problem

The replay engine is written against Playwright's API, but runs on browser-use 0.13's CDP actor surface, which has none of it. Verified with the pinned version (browser-use==0.13.4):

Called by the engine	On CDP actor Page/Element
page.locator / locator.wait_for	❌ doesn't exist
page.wait_for_selector / wait_for_load_state	❌ doesn't exist
page.check / page.uncheck	❌ doesn't exist
page.query_selector_all / el.is_visible() / el.text_content()	❌ doesn't exist
locator.click(force=True)	❌ no force= kwarg
locator.select_option(label=...)	❌ values only — option label text is invisible to it
locator.press	❌ Element has no press
page.screenshot(path=...)	❌ returns base64, no path=
evaluate treated as returning dict/bool	❌ always returns a string (JSON-stringified)

Because nearly every call site wraps these in broad except blocks, the failures are silent: get_best_element_handle exhausts all fallbacks and raises for every selector, so any recorded workflow with a cssSelector step fails on its first element interaction. Radio/checkbox steps only 'succeed' when the state already matches; form-validation detection always returns 'no errors' (rejected submits are reported SUCCESS); XPath fallback never fires; the multi-strategy finder reads node.text/node.aria_label etc. that don't exist on the slotted EnhancedDOMTreeNode, so no semantic strategy can ever match.

Fix (reviewable commit-by-commit)

1. test(contract) — an import-time API-contract test (workflows/tests/test_browser_use_contract.py, no browser launch):
   • asserts every Page/Element/Browser attribute the engine relies on exists with a compatible signature (fails at CI time on a breaking browser-use upgrade, instead of silently at replay time)
   • a static scan asserting no Playwright-only API call exists outside the compat layer — it failed with 35 real call sites before the port and is green after; it also prevents this bug class from regressing
2. fix(replay) deterministic path — new workflow_use/compat/cdp.py (single module that knows the Playwright↔CDP differences: polling wait_for_element, visibility via geometry+computed style, XPath resolution via a temporary marker attribute, JSON-decoding of evaluate string returns, readyState-based load waits, base64 screenshot-to-file, checkbox state setter, select-by-visible-text with proper input/change events) + ports controller/utils.py, controller/service.py, workflow/service.py. Also:
   • DISABLED_DEFAULT_ACTIONS listed pre-0.2 action names, so only 4 of 27 matched and 16 agent-only builtins stayed registered while custom click/input/scroll silently overwrote builtins — replaced with excluding all builtins by their real names + explicit registrations
   • go_back/go_forward steps built an empty ActionModel(), which Tools.act no-ops while reporting success — now real CDP navigations
   • dropped Playwright-only :has-text()/:visible selectors (invalid CSS that throws in querySelectorAll)
3. fix(replay) semantic executor / validation / element finder — same treatment for semantic_executor.py (11 page.check sites, 5 wait_for_selector sites, select-by-visible-text, AI extraction called ainvoke(str) and read .content while the API takes a message list and exposes .completion), validation_utils.py (rewritten as one page-evaluate pass), and element_finder.py (reads the real node sources: attributes dict, ax_node, get_all_children_text()).

Verification

cd workflows && uv run pytest tests/test_browser_use_contract.py -q   # 14 passed

Related: #165 (recording-side findings), #166 (recording pipeline fixes — independent; both PRs apply cleanly in either order aside from trivial context overlap).

🤖 Generated with Claude Code

---

Summary by cubic

Ports the replay engine from Playwright APIs to the browser-use 0.13 CDP actor surface, restoring deterministic replay. Adds a CDP compat layer and contract tests so API breaks fail fast instead of silently at runtime.

• Bug Fixes

  • Replaced Playwright-only calls with CDP equivalents: selector waits, visibility checks, XPath resolution, evaluate return decoding, load-state waits, screenshots, key press, checkbox/radio handling, and select-by-visible-text.
  • Controller now excludes all built-in agent actions and registers deterministic actions explicitly; implements real back/forward navigation (fixes prior no-op success).
  • Element finder reads real EnhancedDOMTreeNode sources (attributes/AX/text, including nested {'attributes': {...}}), uses CDP-safe text/XPath strategies with per-lookup marker tokens, and a page-side text finder that scans rendered innerText and polls until timeout; removed :has-text()/:visible.
  • Validation reworked to collect visible error texts in one pass; prevents false “success” after rejected submits.
  • Error reporting uses get_url/get_title and writes base64 screenshots; XPath click path now decodes evaluate JSON before inspecting; checkbox state handling is centralized via cdp.set_checkbox_state.

• New Features

  • New workflow_use/compat/cdp.py shims: wait_for_element, visibility via geometry+style, XPath resolution with tokenized markers, readyState-based load waits, base64 screenshot-to-file, checkbox state setter, select-by-visible-text, and focused key press.
  • Import-time contract tests assert Page/Element/Browser API surface and statically block Playwright-only calls to guard browser-use upgrades.
  • Dev setup: add pytest for the contract test; remove pytest-asyncio and pytest config. Run the gate with: uv run pytest tests/test_browser_use_contract.py.

^{Written for commit c25a7ef. Summary will update on new commits.}

[cubic-dev-ai]

5 issues found across 11 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="workflows/workflow_use/compat/cdp.py">

<violation number="1" location="workflows/workflow_use/compat/cdp.py:47">
P3: `evaluate` returns the wrong type for legitimate JavaScript strings equal to `True` or `False`; callers expecting text can receive booleans instead. Preserving the actor's result type (or exposing a typed boolean/result envelope instead of guessing from strings) would avoid this ambiguity.</violation>

<violation number="2" location="workflows/workflow_use/compat/cdp.py:150">
P2: XPath resolution can mutate page state and select a stale element because the fixed marker is not collision-safe and cleanup does not preserve a pre-existing attribute. A per-lookup marker/token with restoration of the original attribute would keep the temporary lookup side effect isolated.</violation>

<violation number="3" location="workflows/workflow_use/compat/cdp.py:204">
P3: The new CDP layer ships three helpers that nothing in the engine actually calls: `is_checked`, `set_checkbox_state`, and `element_inner_text`. Checkbox handling in the engine is done by `semantic_executor._set_checked_by_selector`/`_element_is_checked`, and text is read via `element_text_content` or inline `cdp.evaluate` JS, so these three are dead code. Consider removing them (the contract test's mention of `set_checkbox_state` in a comment doesn't exercise them), so the compat surface stays minimal and future readers aren't led to use untested paths.</violation>
</file>

<file name="workflows/workflow_use/controller/service.py">

<violation number="1" location="workflows/workflow_use/controller/service.py:57">
P2: A navigation that remains in a non-complete `readyState` for 10 seconds is still reported as successful because the compatibility wait's `False` result is discarded. Checking the return value and failing the action on timeout would prevent later deterministic steps from running against an incompletely loaded page; the same handling is needed for the history-navigation waits.</violation>
</file>

<file name="workflows/workflow_use/workflow/element_finder.py">

<violation number="1" location="workflows/workflow_use/workflow/element_finder.py:528">
P1: XPath strategies are reported as found, but the subsequent click fails because the CDP JSON result is still treated as a dict in `semantic_executor.py`. The click path should use the same decoded-evaluate compatibility helper (or decode the result) before calling `.get(...)`.</violation>
</file>

<sub>Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic</sub>

[cubic-dev-ai]

P1: XPath strategies are reported as found, but the subsequent click fails because the CDP JSON result is still treated as a dict in semantic_executor.py. The click path should use the same decoded-evaluate compatibility helper (or decode the result) before calling .get(...).

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At workflows/workflow_use/workflow/element_finder.py, line 528:

<comment>XPath strategies are reported as found, but the subsequent click fails because the CDP JSON result is still treated as a dict in `semantic_executor.py`. The click path should use the same decoded-evaluate compatibility helper (or decode the result) before calling `.get(...)`.</comment>

<file context>
@@ -501,7 +524,8 @@ async def _find_with_xpath(
 
-			result = await page.evaluate(js_code)
+			# Page.evaluate returns a JSON string; decode it before inspecting
+			result = await cdp.evaluate(page, js_code)
 
 			if not result:
</file context>

[cubic-dev-ai]

P2: XPath resolution can mutate page state and select a stale element because the fixed marker is not collision-safe and cleanup does not preserve a pre-existing attribute. A per-lookup marker/token with restoration of the original attribute would keep the temporary lookup side effect isolated.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At workflows/workflow_use/compat/cdp.py, line 150:

<comment>XPath resolution can mutate page state and select a stale element because the fixed marker is not collision-safe and cleanup does not preserve a pre-existing attribute. A per-lookup marker/token with restoration of the original attribute would keep the temporary lookup side effect isolated.</comment>

<file context>
@@ -0,0 +1,254 @@
+				if elements:
+					element = elements[0]
+					try:
+						await element.evaluate(f"() => this.removeAttribute('{_XPATH_MARKER_ATTR}')")
+					except Exception:
+						pass
</file context>

[cubic-dev-ai]

P2: A navigation that remains in a non-complete readyState for 10 seconds is still reported as successful because the compatibility wait's False result is discarded. Checking the return value and failing the action on timeout would prevent later deterministic steps from running against an incompletely loaded page; the same handling is needed for the history-navigation waits.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At workflows/workflow_use/controller/service.py, line 57:

<comment>A navigation that remains in a non-complete `readyState` for 10 seconds is still reported as successful because the compatibility wait's `False` result is discarded. Checking the return value and failing the action on timeout would prevent later deterministic steps from running against an incompletely loaded page; the same handling is needed for the history-navigation waits.</comment>

<file context>
@@ -65,15 +53,32 @@ async def navigation(params: NavigationAction, browser_session: Browser) -> Acti
-
-			await asyncio.sleep(2)
+			# CDP navigate doesn't wait automatically; wait for the document to load.
+			await cdp.wait_for_load_state(page, 'load', timeout_ms=10000)
 
 			msg = f'🔗  Navigated to URL: {params.url}'
</file context>

Suggested change

	await cdp.wait_for_load_state(page, 'load', timeout_ms=10000)
	if not await cdp.wait_for_load_state(page, 'load', timeout_ms=10000):
	raise TimeoutError(f' timed out waiting for page load after navigating to {params.url}')

[cubic-dev-ai]

P3: evaluate returns the wrong type for legitimate JavaScript strings equal to True or False; callers expecting text can receive booleans instead. Preserving the actor's result type (or exposing a typed boolean/result envelope instead of guessing from strings) would avoid this ambiguity.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At workflows/workflow_use/compat/cdp.py, line 47:

<comment>`evaluate` returns the wrong type for legitimate JavaScript strings equal to `True` or `False`; callers expecting text can receive booleans instead. Preserving the actor's result type (or exposing a typed boolean/result envelope instead of guessing from strings) would avoid this ambiguity.</comment>

<file context>
@@ -0,0 +1,254 @@
+	"""
+	if raw is None or raw == '':
+		return None
+	if raw == 'True':
+		return True
+	if raw == 'False':
</file context>

[cubic-dev-ai]

P3: The new CDP layer ships three helpers that nothing in the engine actually calls: is_checked, set_checkbox_state, and element_inner_text. Checkbox handling in the engine is done by semantic_executor._set_checked_by_selector/_element_is_checked, and text is read via element_text_content or inline cdp.evaluate JS, so these three are dead code. Consider removing them (the contract test's mention of set_checkbox_state in a comment doesn't exercise them), so the compat surface stays minimal and future readers aren't led to use untested paths.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At workflows/workflow_use/compat/cdp.py, line 204:

<comment>The new CDP layer ships three helpers that nothing in the engine actually calls: `is_checked`, `set_checkbox_state`, and `element_inner_text`. Checkbox handling in the engine is done by `semantic_executor._set_checked_by_selector`/`_element_is_checked`, and text is read via `element_text_content` or inline `cdp.evaluate` JS, so these three are dead code. Consider removing them (the contract test's mention of `set_checkbox_state` in a comment doesn't exercise them), so the compat surface stays minimal and future readers aren't led to use untested paths.</comment>

<file context>
@@ -0,0 +1,254 @@
+	return await evaluate(element, '() => !!this.checked') is True
+
+
+async def set_checkbox_state(element: 'Element', checked: bool) -> bool:
+	"""Playwright ``check()``/``uncheck()`` equivalent via user-like clicks.
+
</file context>

[Sangaibisi]

Thanks for the thorough review @cubic-dev-ai — addressed in a0fda9c:

• P1 text-fallback ignores timeout / single query → rewritten as one page-side pass in the compat layer, polled until timeout_ms.
• P2 40-element cap → gone; the page-side pass scans all candidates in a single evaluation.
• P2 hidden-text/attribute contract → matches rendered innerText plus aria-label/title/placeholder/value.
• P1 XPath click path treats JSON as dict → decoded via the compat helper; this was the click-then-report-failure trap.
• P2 marker collision/cleanup → per-lookup random tokens; fetch/untag extracted to a shared helper, so overlapping lookups can't select each other's element.
• P2 dict-node nested attributes → _node_attr reads both flat and nested {'attributes': {...}} shapes.
• P2 pytest-asyncio strict mode → added asyncio_mode = "auto" to pyproject.
• P3 unused helpers → element_inner_text removed; checkbox handling now reuses cdp.set_checkbox_state (no duplicate implementation left).

Two items deliberately unchanged, happy to revisit:

• P2 load-state timeout treated as success: pre-PR behavior was an unconditional sleep(2) with no check at all, so warn-and-continue is already stricter than before. Failing the action on a 10s readyState timeout would regress pages that never reach complete (long-polling/streaming resources) — those were replayable before. I'd rather tackle it with a proper network-idle signal in a follow-up.
• P3 'True'/'False' string ambiguity in parse_js_result: acknowledged trade-off of the actor API stringifying booleans as Python-str. Callers that need raw text use element.evaluate directly; every current cdp.evaluate caller wants the decoded form. A typed result envelope is a nice follow-up but felt out of scope for this fix PR.

[cubic-dev-ai]

Thanks for the thorough review @cubic-dev-ai — addressed in a0fda9c:

• P1 text-fallback ignores timeout / single query → rewritten as one page-side pass in the compat layer, polled until timeout_ms.
• P2 40-element cap → gone; the page-side pass scans all candidates in a single evaluation.
• P2 hidden-text/attribute contract → matches rendered innerText plus aria-label/title/placeholder/value.
  ...

@Sangaibisi Looks good — I re-checked each fix in the diff and they match what I flagged:

• Text fallback: find_element_by_text is now a single page-side pass over all candidates (no 40-element cap) that matches rendered innerText plus aria-label/title/placeholder/value, and it polls until the deadline instead of quitting after one query. That resolves the P1.
• XPath JSON-as-dict: the click path goes through cdp.evaluate now, so the decode happens before .get() — no more attribute-trap after the click already fired. Good.
• Marker collision/cleanup: per-lookup secrets.token_hex(6) plus a shared _fetch_marked_element that untags. Overlapping lookups can't grab each other's element. Solid.
• Nested attrs / asyncio_mode / dead helpers: all confirmed in the patch as described.

On the two you left unchanged, your reasoning holds and I'm fine deferring both:

• Load-state timeout → warn-and-continue: you're right that the OLD code was an unconditional sleep(2) with zero signal, so warn-and-continue is strictly more informative. And readyState genuinely can stall below complete on streaming/long-polling documents — failing the action on that would regress pages that were replayable before. A real network-idle signal is the correct fix and belongs in a follow-up, not this port.
• 'True'/'False' ambiguity: acknowledged trade-off. The latent footgun is a JS string like "True" decoding to boolean True, but as long as every cdp.evaluate caller wants decoded output and raw-text callers bypass via element.evaluate, it's contained. A typed result envelope is a reasonable follow-up.

One thing worth keeping visible for that follow-up: parse_js_result also can't distinguish JS "true" from JS true (both round-trip as 'True'), so if anyone later needs a literal JS string 'True', they'll hit it. Not blocking here — just don't let the helper leak into a path that handles arbitrary user text and expects to preserve it verbatim.

The contract test is the real win here — it would have caught this whole class at CI time instead of at replay time. No further changes from me.

_{Tip: get faster answers by chatting with cubic’s review copilot in the review UI}

[cubic-dev-ai]

All reported issues were addressed across 5 files (changes from recent commits).

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[Sangaibisi]

Round-2 review response (c25a7ef): the contract test is fully synchronous, so pytest-asyncio and the [tool.pytest.ini_options] block are gone entirely. Default collection semantics are now byte-identical to upstream - a bare uv run pytest neither executes the pre-existing live-browser scripts in tests/ nor hides the nested workflow_use/*/tests suites. The gate runs with uv run pytest tests/test_browser_use_contract.py (14 passed).