Fix recording pipeline and deterministic replay on browser-use 0.13 / Chrome 137+ - #166
Open
Sangaibisi wants to merge 8 commits into
Open
Conversation
…-use 0.13 Two independent bugs made recording capture zero events: 1. browser-use's default extensions append their own --load-extension flag after the profile args. Chrome only honors the last occurrence, so the recorder extension was silently dropped. Disable default extensions for the recording profile so our flag wins. 2. Branded Google Chrome 137+ removed support for --load-extension entirely. When browser-use picks the installed Chrome, the extension never loads and no events reach the recording server. Prefer a Playwright-bundled Chromium / Chrome for Testing binary when one is available, overridable via WORKFLOW_USE_RECORDER_BROWSER. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
WorkflowDefinitionSchema required the last step to be extract / extract_page_content. The extension streams WORKFLOW_UPDATE events step-by-step while the user records, so virtually every update ended in click/input/navigation and was rejected with 422 by the recording server — the recorder never received any workflow data and create-workflow hung forever after the browser closed. Relax the validator: an extract-terminated workflow cannot be enforced at parse time without breaking recording, and no-AI workflows are valid without a trailing extract step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- _run_deterministic_step stripped cssSelector/xpath from step params,
but the click/input/key_press/select_change action models require
cssSelector — every selector-based deterministic action failed
validation. Keep those fields (extras are ignored via extra='ignore').
- Semantic key_press crashed on named keys: dict.get's default is
evaluated eagerly, so ord('ENTER') raised before the lookup.
- The hand-rolled Input.dispatchKeyEvent call used a stale CDP session
and failed with 'method not found'. Use page.press(), which handles
named keys, combos and session setup.
- The key_press verifier required the pressed element to still be
visible, so keys that navigate (Enter on a search box) always failed
verification and were re-pressed. Treat a URL change as success.
- An empty semantic mapping usually means the page is mid-navigation;
retry extraction a few times before failing the step.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- cli.py crashed with NameError when LLM init failed and the user declined to enter an API key: llm_instance was left undefined. Degrade gracefully — no-AI commands don't need an LLM. - backend imported Browser from the pre-0.13 module path (browser_use.browser.browser) and crashed on startup; and it instantiated ChatBrowserUse eagerly, so the API served 500s without BROWSER_USE_API_KEY even for listing/executing deterministic workflows. Make the LLM optional. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sidebar items only received metadata for the currently selected workflow, so every other row rendered a permanent 'Loading workflow…' placeholder until clicked. Fetch metadata for all listed workflows and pass it through the existing allWorkflowsMetadata prop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Author
|
Pushed one more commit: the GUI sidebar rendered a permanent "Loading workflow…" placeholder for every non-selected workflow (metadata was only fetched for the selected one, and the existing |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
1 issue found across 1 file (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="ui/src/components/workflow-layout.tsx">
<violation number="1" location="ui/src/components/workflow-layout.tsx:174">
P2: This effect fetches the *entire* workflow document (including every step) for all workflows — N parallel GETs on mount and again whenever `workflows` changes — but the Sidebar/WorkflowItem only use the result to render each non-selected row's name and version (the details/edit panel always reads the selected workflow's separate `workflowMetadata`). Fetching a full document per workflow just to display a name is wasteful and can fan out a large number of requests for repos with many workflows. Consider using a lighter metadata/list endpoint (or deriving names from the existing /api/workflows response) instead of downloading each full workflow, and it's worth logging failures in the `catch` so a failed fetch doesn't silently leave the row stuck on the 'Loading workflow…' placeholder.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
| const entries = await Promise.all( | ||
| workflows.map(async (name) => { | ||
| try { | ||
| const { data } = await fetchClient.GET("/api/workflows/{name}", { |
Contributor
There was a problem hiding this comment.
P2: This effect fetches the entire workflow document (including every step) for all workflows — N parallel GETs on mount and again whenever workflows changes — but the Sidebar/WorkflowItem only use the result to render each non-selected row's name and version (the details/edit panel always reads the selected workflow's separate workflowMetadata). Fetching a full document per workflow just to display a name is wasteful and can fan out a large number of requests for repos with many workflows. Consider using a lighter metadata/list endpoint (or deriving names from the existing /api/workflows response) instead of downloading each full workflow, and it's worth logging failures in the catch so a failed fetch doesn't silently leave the row stuck on the 'Loading workflow…' placeholder.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At ui/src/components/workflow-layout.tsx, line 174:
<comment>This effect fetches the *entire* workflow document (including every step) for all workflows — N parallel GETs on mount and again whenever `workflows` changes — but the Sidebar/WorkflowItem only use the result to render each non-selected row's name and version (the details/edit panel always reads the selected workflow's separate `workflowMetadata`). Fetching a full document per workflow just to display a name is wasteful and can fan out a large number of requests for repos with many workflows. Consider using a lighter metadata/list endpoint (or deriving names from the existing /api/workflows response) instead of downloading each full workflow, and it's worth logging failures in the `catch` so a failed fetch doesn't silently leave the row stuck on the 'Loading workflow…' placeholder.</comment>
<file context>
@@ -159,6 +162,37 @@ const WorkflowLayout: React.FC = () => {
+ const entries = await Promise.all(
+ workflows.map(async (name) => {
+ try {
+ const { data } = await fetchClient.GET("/api/workflows/{name}", {
+ params: { path: { name } },
+ });
</file context>
Contributor
There was a problem hiding this comment.
All reported issues were addressed
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
get_service() constructed a fresh WorkflowService per request, so
in-memory task state (active_tasks, cancel_events) was lost immediately:
/tasks/{id}/status always 404'd and cancel never worked.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds a 'Record New Workflow' button to the sidebar. The backend gains recording endpoints (/recordings/start|stop|status) that drive RecordingService in-process, convert the captured recording via convert_recorded_workflow_to_semantic, and save it into ./tmp so it appears in the list immediately. Stopping with zero captured steps cancels the session outright instead of hitting the recorder's wait-forever finalizer path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Author
|
Two more commits:
Happy to split the GUI feature into its own PR if you'd prefer to keep this one fixes-only. |
Contributor
There was a problem hiding this comment.
2 issues found across 6 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="workflows/backend/routers.py">
<violation number="1" location="workflows/backend/routers.py:30">
P1: Concurrent `/execute` requests can run another task’s workflow and close each other’s browser. Keep task/recording state shared, but create workflow/browser execution state per task (or serialize execution).</violation>
<violation number="2" location="workflows/backend/routers.py:30">
P2: Completed task results now accumulate for the backend process lifetime, so repeated executions cause unbounded memory growth. Expire or cap completed `active_tasks` entries after a status-retention window.</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
Fix all with cubic | Re-trigger cubic
| return WorkflowService() | ||
| global _service | ||
| if _service is None: | ||
| _service = WorkflowService() |
Contributor
There was a problem hiding this comment.
P1: Concurrent /execute requests can run another task’s workflow and close each other’s browser. Keep task/recording state shared, but create workflow/browser execution state per task (or serialize execution).
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At workflows/backend/routers.py, line 30:
<comment>Concurrent `/execute` requests can run another task’s workflow and close each other’s browser. Keep task/recording state shared, but create workflow/browser execution state per task (or serialize execution).</comment>
<file context>
@@ -18,9 +19,16 @@
- return WorkflowService()
+ global _service
+ if _service is None:
+ _service = WorkflowService()
+ return _service
</file context>
| return WorkflowService() | ||
| global _service | ||
| if _service is None: | ||
| _service = WorkflowService() |
Contributor
There was a problem hiding this comment.
P2: Completed task results now accumulate for the backend process lifetime, so repeated executions cause unbounded memory growth. Expire or cap completed active_tasks entries after a status-retention window.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At workflows/backend/routers.py, line 30:
<comment>Completed task results now accumulate for the backend process lifetime, so repeated executions cause unbounded memory growth. Expire or cap completed `active_tasks` entries after a status-retention window.</comment>
<file context>
@@ -18,9 +19,16 @@
- return WorkflowService()
+ global _service
+ if _service is None:
+ _service = WorkflowService()
+ return _service
</file context>
- backend: create Browser/WorkflowController per execution instead of sharing one instance, so concurrent /execute runs can't close each other's browser (cubic P1). - backend: prune finished tasks beyond MAX_FINISHED_TASKS so a long-lived process doesn't grow active_tasks unboundedly (cubic P2). - backend: guard RecordingService.browser with getattr — stopping right after start no longer 500s before the browser is assigned (cubic P2). - backend: load-error path now records failed status + error instead of only printing and returning. - recorder: sort Playwright cache dirs by numeric revision, not lexicographically (chromium-1017 must beat chromium-999) (cubic P3). - cli: run-workflow now guards on missing LLM and exits cleanly instead of passing llm=None into the agent path (cubic P2). - ui: add GET /api/workflows/metadata (one request, name/description/ version/input_schema only) and use it for the sidebar instead of fetching every full workflow document (cubic P2). - ui: update the sidebar metadata cache after a metadata save so renamed labels don't revert when another workflow is selected (cubic P2). - ui: record-button stop() decides polling from the response, not the stale render-closure status (cubic P3). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Author
|
Thanks for the review @cubic-dev-ai — addressed all of it in 8defbb4:
Also moved off the shared |
Contributor
@Sangaibisi All eight items check out in the code — this looks fully addressed. Quick pass through each:
One deliberate note, not a blocker: the per-execution No outstanding concerns from my side. Good cleanup. Tip: get faster answers by chatting with cubic’s review copilot in the review UI |
Contributor
There was a problem hiding this comment.
2 issues found across 7 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="workflows/backend/service.py">
<violation number="1" location="workflows/backend/service.py:70">
P2: Cancelling tasks can be pruned before their cancellation handler finishes when another execution starts, losing status and raising `KeyError`; keep `cancelling` entries until they reach a terminal state.</violation>
</file>
<file name="ui/src/components/workflow-layout.tsx">
<violation number="1" location="ui/src/components/workflow-layout.tsx:189">
P2: This replaced the typed OpenAPI client call (previously `fetchClient.GET(...)`) with a raw `fetch` to a hardcoded `http://localhost:8000`, duplicating the base URL that already lives in `ui/src/lib/api/index.ts` and bypassing the `$api`/typed client used everywhere else in this component. Because the response is cast blindly (`as Array<{ file: string } & WorkflowMetadata>`), `data.workflows.map(...)` will throw if the response shape changes or an error body is returned. Since the new backend endpoints aren't in the checked-in `openapi.json`/`apigen`, they were forced to bypass the typed client. Consider regenerating the OpenAPI client (`npm run type-gen-update`) so `/api/workflows/metadata` is represented, then call it through `fetchClient`/`$api` (or a shared base-URL constant) instead of a hardcoded URL, and validate `res.ok` before parsing.</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
Fix all with cubic | Re-trigger cubic
|
|
||
| def _prune_finished_tasks(self) -> None: | ||
| """Cap remembered finished tasks so long-lived processes don't grow unboundedly.""" | ||
| finished = [tid for tid, info in self.active_tasks.items() if info.status not in ('running', 'pending')] |
Contributor
There was a problem hiding this comment.
P2: Cancelling tasks can be pruned before their cancellation handler finishes when another execution starts, losing status and raising KeyError; keep cancelling entries until they reach a terminal state.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At workflows/backend/service.py, line 70:
<comment>Cancelling tasks can be pruned before their cancellation handler finishes when another execution starts, losing status and raising `KeyError`; keep `cancelling` entries until they reach a terminal state.</comment>
<file context>
@@ -64,6 +65,13 @@ def __init__(self) -> None:
+ def _prune_finished_tasks(self) -> None:
+ """Cap remembered finished tasks so long-lived processes don't grow unboundedly."""
+ finished = [tid for tid, info in self.active_tasks.items() if info.status not in ('running', 'pending')]
+ excess = len(finished) - self.MAX_FINISHED_TASKS
+ for tid in finished[:max(0, excess)]: # dict preserves insertion order → oldest first
</file context>
Suggested change
| finished = [tid for tid, info in self.active_tasks.items() if info.status not in ('running', 'pending')] | |
| finished = [tid for tid, info in self.active_tasks.items() if info.status not in ('running', 'pending', 'cancelling')] |
| let cancelled = false; | ||
| (async () => { | ||
| try { | ||
| const res = await fetch("http://localhost:8000/api/workflows/metadata"); |
Contributor
There was a problem hiding this comment.
P2: This replaced the typed OpenAPI client call (previously fetchClient.GET(...)) with a raw fetch to a hardcoded http://localhost:8000, duplicating the base URL that already lives in ui/src/lib/api/index.ts and bypassing the $api/typed client used everywhere else in this component. Because the response is cast blindly (as Array<{ file: string } & WorkflowMetadata>), data.workflows.map(...) will throw if the response shape changes or an error body is returned. Since the new backend endpoints aren't in the checked-in openapi.json/apigen, they were forced to bypass the typed client. Consider regenerating the OpenAPI client (npm run type-gen-update) so /api/workflows/metadata is represented, then call it through fetchClient/$api (or a shared base-URL constant) instead of a hardcoded URL, and validate res.ok before parsing.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At ui/src/components/workflow-layout.tsx, line 189:
<comment>This replaced the typed OpenAPI client call (previously `fetchClient.GET(...)`) with a raw `fetch` to a hardcoded `http://localhost:8000`, duplicating the base URL that already lives in `ui/src/lib/api/index.ts` and bypassing the `$api`/typed client used everywhere else in this component. Because the response is cast blindly (`as Array<{ file: string } & WorkflowMetadata>`), `data.workflows.map(...)` will throw if the response shape changes or an error body is returned. Since the new backend endpoints aren't in the checked-in `openapi.json`/`apigen`, they were forced to bypass the typed client. Consider regenerating the OpenAPI client (`npm run type-gen-update`) so `/api/workflows/metadata` is represented, then call it through `fetchClient`/`$api` (or a shared base-URL constant) instead of a hardcoded URL, and validate `res.ok` before parsing.</comment>
<file context>
@@ -173,30 +179,24 @@ const WorkflowLayout: React.FC = () => {
- Object.fromEntries(entries.filter((e): e is NonNullable<typeof e> => e !== null))
- );
+ try {
+ const res = await fetch("http://localhost:8000/api/workflows/metadata");
+ const data = (await res.json()) as {
+ workflows: Array<{ file: string } & WorkflowMetadata>;
</file context>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #165
On a fresh clone with the pinned browser-use 0.13.4, recording captures zero events and deterministic replay fails on several step types. This PR restores the full record → build → replay (no-AI) pipeline; each commit is self-contained:
1.
fix(recorder)— recorder extension never loads--load-extensionflag which overrides ours (Chrome honors the last one). Disable default extensions for the recording profile.--load-extensionsupport altogether. Prefer a Playwright-bundled Chromium / Chrome for Testing binary when present (macOS/Linux/Windows cache paths), overridable viaWORKFLOW_USE_RECORDER_BROWSER; falls back to browser-use's default with a warning.2.
fix(schema)— validator 422s every incremental recording updatevalidate_ends_with_extractrejected essentially everyWORKFLOW_UPDATEstreamed by the extension mid-recording, so the recording server never stored any data andcreate-workflowhung after the browser closed. The validator is relaxed; no-AI workflows are also valid without a trailing extract step.3.
fix(workflow)— deterministic replay on browser-use 0.13cssSelector/xpathfrom step params: the click/input/key_press/select_change action models requirecssSelector(extras are ignored viaextra='ignore'), so every selector-based deterministic action failed validation.dict.get's default is evaluated eagerly, soord('ENTER')raised even though'Enter'is in the key map.Input.dispatchKeyEvent(stale CDP session →-32601 method not found) withpage.press().4.
fix(cli,backend)— no-AI paths shouldn't requireBROWSER_USE_API_KEYcli.py:NameError: llm_instancewhen LLM init failed; now degrades gracefully.from browser_use.browser.browser import Browserno longer exists in 0.13; and the eagerChatBrowserUseinstantiation made every endpoint 500 without an API key. LLM is now optional.Verified
create-workflow-no-aiproduced a valid semantic workflow from a real user recording.runexecutes navigation/input/key_press/click/scroll steps of the recorded workflow deterministically (no API key), including Enter-triggered navigation.🤖 Generated with Claude Code
Summary by cubic
Restores the full record → build → deterministic replay pipeline on
browser-use0.13, adds a GUI “Record New Workflow” button backed by new recording APIs, and hardens task/recording state so status/cancel work across requests and concurrent runs. No‑AI workflows now run withoutBROWSER_USE_API_KEY.New Features
/recordings/status, and selects the saved file when done./recordings/start|stop|statusrunRecordingService, convert to semantic, and save into./tmp; zero‑step stops cancel cleanly.Bug Fixes
--load-extensionwins; prefer Playwright-bundled Chromium/Chrome for Testing (newest revision) on Chrome 137+; override viaWORKFLOW_USE_RECORDER_BROWSER.WorkflowServiceto preserve in‑memory task/recording state; createBrowser/WorkflowControllerper execution; prune finished tasks; record load errors in task status.cssSelector/xpath; usepage.press()for keys; treat URL change as success; retry semantic mapping during navigation.run-workflowguards on missing LLM; update imports forbrowser-use0.13.GET /api/workflows/metadataso the sidebar shows real names; keep metadata cache in sync after saves; record button stop/polling logic is more robust.Written for commit 8defbb4. Summary will update on new commits.