browser: migrate callers onto the canonical interface (ADR-009 steps 3-6) — closes the #13204 guard gap

Part of #12645 / #12756. Follow-up to #12651, which delivered ADR-009 phases 1–2 (merged a6d7aa1ce, PR #13226).

What exists now

autobot_shared/browser/ holds the canonical contract — Capability, request/result types, capability dispatch with probe-based fallback, and the DNS-resolving URL guard — and autobot-backend/browser_backends/ wraps the three stacks (in_process, worker, container).

Nothing is repointed. Every caller still reaches its stack directly, and register_all() is opt-in.

What this issue covers — ADR-009 steps 3–6

Sequenced so each step is separately reviewable, per the ADR:

content_reach/backends/browser.py — smallest caller, and already guarded, so a regression is easy to see.
web_fetch/fetcher.py render fallback.
chat_workflow/tool_handler.py — delete the hand-rolled cascade (_web_search_via_playwright / _web_search_structured_entries / _web_search_final_fallback / _web_search_via_browser_vm) in favour of capability dispatch.
api/playwright.py, and retire is_url_allowed's regex allowlist in favour of the shared guard plus an explicit, documented internal-host exception list.

The reason this is its own issue, not a rider on #12651

Step 5 is a behaviour change, and it is the one that closes #13204.

Today send_to_browser_vm() performs no URL validation and is reachable from the agent tool path. Routing it through the canonical interface means the DNS-resolving guard starts running on it — so URLs that path accepts today will begin failing: internal addresses, loopback, link-local. That is the intended fix, but it is a production behaviour change on the live chat path and needs its own review and rollout note rather than riding along with an additive interface PR.

api/browser_mcp.py's existing is_url_allowed allowlists loopback and an internal VM range, which suggests at least some internal navigation is deliberate. Enumerate those call sites before step 6 — the exception list needs to be derived from real usage, not guessed.

Done when

content_reach, web_fetch, tool_handler and api/playwright all reach the browser through autobot_shared.browser.get_browser();
no caller names a stack directly;
tool_handler's hand-rolled cascade is gone (the bug class in discovery(chat/search_web): duplicated Playwright service calls in fetch_full path #7478 with it);
#13204 is closed structurally — the guard runs on every navigate/fetch;
research/web-search tests green.

Blocking note

#13204 remains open until step 5 lands. #12651 closing does not close it.

Started step 3 and stopped before changing anything — reading the first two real callers found three gaps in the phase-1/2 contract I shipped in #13226. All three are mine, and all three would have been papered over by migrating anyway.

1. `BrowserResult` cannot carry what `content_reach` returns

content_reach/backends/browser.py maps research_url(extract_content=True) onto a ContentResult with two payloads:

content   = result.get("content") or {}
text      = content.get("text_content", "")
structured = content.get("structured_data") or {}     # <- no home in BrowserResult

BrowserResult has content but no structured field, so migrating as-is silently drops structured_data.

2. `ExtractRequest` has no `url`, so a stateless backend cannot serve it

web_fetch/fetcher.py::_fetch_playwright renders a URL through the container:

result = await svc._post_and_parse("render", {"url": url, "wait": "networkidle", ...})
return result.get("html") or result.get("content")

Two problems. ExtractRequest carries session_id/selector/max_chars but no url — fine for the session-holding stacks, impossible for the stateless container. And my ContainerBrowserBackend declares SCREENSHOT only: I wrapped capture_screenshot and missed that the container also exposes a render endpoint. The capability matrix in ADR-009 under-declares that stack.

3. The bigger one: callers want a specific stack, not "whatever can do X"

This is the design issue, and it is why I stopped.

The capability model assumes callers are indifferent to which backend serves them. The first two real callers are not:

web_fetch uses the container deliberately, so Playwright does not run in the backend's own process. Dispatching to "any EXTRACT-capable backend" could route it in-process — exactly what that fallback exists to avoid.
content_reach wants the research browser specifically: its probe() is rbm.PLAYWRIGHT_AVAILABLE, and its error handling reads blocked_by_guard, which is that stack's security(content_reach): browser + yt-dlp fetches keep a DNS-rebind TOCTOU window that pinning our own connector cannot close #13018 re-check.

Requiring a capability only one stack declares would technically pin the routing, but that is gaming the model, not using it.

The contract needs a way to express process locality or stack affinity as a first-class requirement — something like an OUT_OF_PROCESS capability, or an explicit prefer=/pin= argument on get_browser() — before any caller migrates. Otherwise each migration silently changes which stack serves a request, and the failure only shows up under load or when a backend is down.

What this means for the plan

ADR-009's step ordering should change. It nominated content_reach first as "smallest, already guarded" — but it is the caller with the most stack-specific behaviour (structured extraction plus blocked_by_guard handling). web_fetch's render path is smaller and more self-contained, but it needs gaps 2 and 3 closed first.

Suggested revised order:

Contract fixes first (no caller changes): structured on BrowserResult, url on ExtractRequest, EXTRACT declared and implemented on ContainerBrowserBackend via its render endpoint, and a decision on how affinity is expressed.
Then web_fetch (step 4) as the first migration, not content_reach.
Then content_reach, once structured exists.
Then tool_handler (the security(browser): send_to_browser_vm and playwright_service bypass the SSRF/DNS-rebind guard the other browser paths enforce #13204 fix, and the behaviour change).
Then api/playwright.

I have not written any of this yet — item 1 needs the affinity decision, and I would rather ask than guess, having just watched three assumptions fail on contact with real callers.

#13204 is unaffected by this and stays open; it is still closed only by the tool_handler migration.

feat(browser): close three contract gaps the first real callers expos…

Contract gaps closed — merged as dcb0e3b5d (PR #13256). This issue stays OPEN: no caller is migrated.

What landed

Per your 2026-08-02 decision, locality is now a capability:

browser = await get_browser(requires={
    Capability.EXTRACT,
    Capability.OUT_OF_PROCESS,   # never in the backend's own process
})

Every backend declares exactly one of IN_PROCESS / OUT_OF_PROCESS, asserted by test — a backend declaring neither, or both, would let web_fetch be routed into the process it deliberately avoids.

Also closed, both found the same way:

ExtractRequest.url, guarded by the registry exactly like a navigate URL. A stateless backend holds no current page, and that URL reaches the network the same way, so it cannot skip the check.
BrowserResult.structured + NavigateRequest.extract. content_reach maps research_url's structured_data onto ContentResult.structured; it had no home in the result, so migrating would have silently dropped it. research_url navigates and extracts in one round trip, so asking on the navigate request avoids navigating twice.
ContainerBrowserBackend now declares EXTRACT, implemented against the container's render endpoint — correcting a phase-2 under-declaration where I wrapped only capture_screenshot.

Verified on Dev_new_gui after merge: 36 tests pass (was 30), bandit 0 issues.

Revised step order

ADR-009 nominated content_reach first as "smallest, already guarded". That was wrong — it is the caller with the most stack-specific behaviour (structured extraction plus blocked_by_guard handling). Revised:

~~Contract fixes~~ ✅ done
web_fetch (_fetch_playwright → EXTRACT + OUT_OF_PROCESS) — now the cleanest first migration
content_reach, now that structured exists
tool_handler — the behaviour change, and the step that closes security(browser): send_to_browser_vm and playwright_service bypass the SSRF/DNS-rebind guard the other browser paths enforce #13204
api/playwright, retiring is_url_allowed's regex allowlist

Before step 4

Step 4 will start rejecting internal/loopback URLs that send_to_browser_vm accepts today. is_url_allowed currently allowlists loopback and an internal VM range, which suggests some internal navigation is deliberate. Those call sites need enumerating first so the exception list is derived from real usage rather than guessed — and that step wants your sign-off, since it is a production behaviour change on the live chat path.

#13204 remains open and is unaffected by this PR.

PR #13256 (merged to Dev_new_gui) references this issue with a close keyword.

feat(browser): close three contract gaps the first real callers exposed (#13236)

If this issue is fully resolved, close it manually. If work remains, no action is needed.

feat(browser): ExtractRequest carries a timeout, forwarded by the con…

feat(browser): split EXTRACT by content format so dispatch cannot swa…

PR #13306 (merged to Dev_new_gui) references this issue with a close keyword.

feat(browser): split EXTRACT by content format so dispatch cannot swap HTML for text (#13236)

If this issue is fully resolved, close it manually. If work remains, no action is needed.

refactor(web-fetch): render through the canonical browser interface (#…

refactor(content-reach): route the browser backend through the canoni…

PR #13313 (merged to Dev_new_gui) references this issue with a close keyword.

refactor(content-reach): route the browser backend through the canonical interface (#13236)

If this issue is fully resolved, close it manually. If work remains, no action is needed.

fix(security): resolve hosts instead of prefix-matching URLs in the b…

PR #13317 (merged to Dev_new_gui) references this issue with a close keyword.

fix(security): resolve hosts instead of prefix-matching URLs in the browser allowlist (#13236)

If this issue is fully resolved, close it manually. If work remains, no action is needed.

Status correction — this issue is not close to done, and PR #13317 was mislabelled.

PR #13317 (merged) was titled and described as "ADR-009 step 5". Against the step numbering in this issue and in ADR-009 itself, it delivered the allowlist-retirement half of step 6 — is_url_allowed no longer prefix-matches URLs, and its two call sites were updated. It did not touch chat_workflow/tool_handler.py's cascade, which is what step 5 actually is.

Measured against the "Done when" list on Dev_new_gui at aa7f2c617:

Step	Caller	State
3	`content_reach/backends/browser.py`	done (#13226 / PR #13300)
4	`web_fetch/fetcher.py` render fallback	not started
5	`chat_workflow/tool_handler.py` cascade	not started — `_web_search_via_playwright` and `_web_search_via_browser_vm` are both still present
6	`api/playwright.py` repoint	not started — still imports the research stack directly
6	retire `is_url_allowed` regex allowlist	done (PR #13317)

So three of the four callers still name a stack directly, and tool_handler's hand-rolled cascade is intact.

#13204 therefore remains open on its original terms. Step 5 is the step that closes it structurally, and step 5 has not been done. PR #13317 narrowed the exposure — the guard that path did have was a prefix-match allowlist that accepted lookalike hosts, and that is now a resolving check — but narrowing a weak guard is not the same as routing the path through the interface.

Remaining work is steps 4, 5, and the api/playwright.py half of 6, in that order.

Stale-issue sweep (never auto-closed — Dev_new_gui wasn't GitHub's default branch at merge time). Verified against current origin/main; leaving open.

Met: content_reach and web_fetch now reach the browser via get_browser() — autobot-backend/content_reach/backends/browser.py:26-29,124 (#13313), autobot-backend/web_fetch/fetcher.py:256-265 (#13309). A URL-allowlist regex-prefix bypass was also fixed along the way (#13317, is_url_allowed now resolves hosts via is_public_url_async).

Not met — the issue's core ask ("no caller names a stack directly," the hand-rolled cascade removed): autobot-backend/chat_workflow/tool_handler.py still defines _web_search_structured_entries (:3021), _web_search_via_playwright (:3054), _web_search_final_fallback (:3074), _web_search_via_browser_vm (:3090) — the exact four functions named for deletion — calling send_to_browser_vm directly. No get_browser/autobot_shared.browser reference exists anywhere in that file. autobot-backend/api/playwright.py is also never migrated (from services.playwright_service import (...) directly, no get_browser).

None of #13236's own 5 PRs migrated tool_handler.py or api/playwright.py — both remain unmigrated on current main.

Navigation Menu

Repository navigation

browser: migrate callers onto the canonical interface (ADR-009 steps 3-6) — closes the #13204 guard gap #13236

What exists now

What this issue covers — ADR-009 steps 3–6

The reason this is its own issue, not a rider on #12651

Done when

Blocking note

1. `BrowserResult` cannot carry what `content_reach` returns

2. `ExtractRequest` has no `url`, so a stateless backend cannot serve it

3. The bigger one: callers want a specific stack, not "whatever can do X"

What this means for the plan

What landed

Revised step order

Before step 4

1 remaining item

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

browser: migrate callers onto the canonical interface (ADR-009 steps 3-6) — closes the #13204 guard gap #13236

Description

What exists now

What this issue covers — ADR-009 steps 3–6

The reason this is its own issue, not a rider on #12651

Done when

Blocking note

Activity

mrveiss commented on Aug 1, 2026

1. BrowserResult cannot carry what content_reach returns

2. ExtractRequest has no url, so a stateless backend cannot serve it

3. The bigger one: callers want a specific stack, not "whatever can do X"

What this means for the plan

mrveiss commented on Aug 1, 2026

What landed

Revised step order

Before step 4

github-actions commented on Aug 1, 2026

github-actions commented on Aug 2, 2026

1 remaining item

github-actions commented on Aug 2, 2026

github-actions commented on Aug 3, 2026

mrveiss commented on Aug 3, 2026

mrveiss commented on Sep 12, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

1. `BrowserResult` cannot carry what `content_reach` returns

2. `ExtractRequest` has no `url`, so a stateless backend cannot serve it