Skip to content

feat(browser): split EXTRACT by content format so dispatch cannot swap HTML for text (#13236) - #13306

Merged
mrveiss merged 3 commits into
Dev_new_guifrom
issue-13236-format
Aug 2, 2026
Merged

mrveiss merged 3 commits into
Dev_new_guifrom
issue-13236-format

Conversation

@mrveiss

@mrveiss mrveiss commented Aug 2, 2026 •

Copy link
Copy Markdown
Owner

Closes #13236

Partial — #13236 must stay OPEN. The close keyword is required by the
Validate PR issue link gate; merging to Dev_new_gui does not auto-close.
No caller is migrated — this closes the last contract gap blocking
migration. Steps 2–5 and #13204 remain.

Thinking Path

The fourth gap the real callers exposed, and the only one that would have
produced a silent regression rather than an error.

A single EXTRACT capability meant "return the page's text/markup". That
conflated three different outputs, and the three stacks each produce a
different one:

stack call returns
container render HTML
worker get_text text, markup stripped
in-process extract_content() text + structured

web_fetch/_fetch_playwright needs HTML — it feeds a parser. Both the worker
and the container declared EXTRACT and OUT_OF_PROCESS, and register_all
registers the worker first. So migrating web_fetch onto the interface
would have resolved to the worker and returned text.

Nothing would have raised. Text is a valid BrowserResult.content, so the
parser would simply have received the wrong thing — the failure mode that
only shows up as bad output much later.

What Changed

EXTRACT → EXTRACT_TEXT / EXTRACT_HTML / EXTRACT_STRUCTURED. Each
backend now declares what it actually returns rather than that it "extracts".

ExtractRequest.format (a new ContentFormat, defaulting to TEXT).

The two are tied together. Requiring a capability at get_browser time
and naming a format on the request are separate statements that can disagree,
so Browser.extract checks the resolved backend declares
FORMAT_CAPABILITY[request.format] and raises UnsupportedFormatError rather
than returning the wrong shape.

ADR-009 amended in the same commit, not left to drift — it still
documented the single EXTRACT, and its step order nominated content_reach
as the first migration. The audit showed that is wrong: content_reach is the
caller with the most stack-specific behaviour (structured extraction plus
blocked_by_guard handling). web_fetch is the correct first migration, and
the amendment says so.

A fifth gap, found by checking the contract end-to-end

After the split I checked whether web_fetch could actually be served, and it
still could not. _fetch_playwright sends
timeout: int(timeout * 1000) to the render endpoint, but ExtractRequest
had no timeout field — NavigateRequest and ScreenshotRequest both do,
so it was the odd one out — and container.extract sent no timeout at all.

Migrating would have silently dropped the caller's timeout and fallen back to
the container default. ExtractRequest.timeout_seconds now exists, the
container converts to the milliseconds its endpoint expects, and omits the key
entirely when the caller passes None so the container default still applies
rather than timeout=0.

Verification

autobot_shared/browser + autobot-backend/browser_backends   46 passed  (was 36)
bandit                                                       0 issues
isort · black --line-length=120 · flake8                     clean

The load-bearing test uses the real backends under the real
registration order:

register_all(force=True)
chosen = await resolve_backend({Capability.EXTRACT_HTML, Capability.OUT_OF_PROCESS})
assert chosen.name == "container"   # not "worker", which registers first

The others pin that a mismatched format raises before the backend is
reached (asserted against its call log), that every ContentFormat maps to a
capability so a new format cannot KeyError at dispatch, and that each
stack's declared formats match what its code actually returns.

I also checked no bare Capability.EXTRACT survives anywhere, including the
package docstring example — grep clean.

What this does not do

Model Used

Claude Opus 5 (1M context)

mrveiss added 2 commits August 2, 2026 17:23
…p HTML for text (#13236)

The fourth contract gap the real callers exposed, and the one that would
have caused a silent regression rather than an error.

A single EXTRACT capability meant "return the page's text/markup", which
conflated three different outputs -- and the three stacks each produce a
different one:

  container  render              -> HTML
  worker     get_text            -> text, markup stripped
  in_process extract_content()   -> text + structured_data

web_fetch/_fetch_playwright needs HTML; it feeds a parser. Both the worker
and the container declared EXTRACT and OUT_OF_PROCESS, and register_all
registers the worker FIRST, so migrating web_fetch onto the interface would
have resolved to the worker and returned text. Text is a valid result shape,
so nothing would have raised -- the parser would just have received the
wrong thing.

Replaced with EXTRACT_TEXT / EXTRACT_HTML / EXTRACT_STRUCTURED, and each
backend now declares what it actually returns rather than that it "extracts".

ExtractRequest gains a ContentFormat, defaulting to TEXT. Requiring a
capability at get_browser time and naming a format on the request are two
statements that can disagree, so Browser.extract checks the resolved backend
declares FORMAT_CAPABILITY[request.format] and raises UnsupportedFormatError
instead of returning the wrong shape.

ADR-009 is amended in the same commit rather than left to drift -- it still
documented the single EXTRACT, and its step order nominated content_reach as
the first migration, which the audit showed is wrong (it is the caller with
the most stack-specific behaviour). web_fetch is the correct first migration.

44 tests, up from 36. The load-bearing one asserts that under the REAL
registration order, web_fetch's requirement
{EXTRACT_HTML, OUT_OF_PROCESS} resolves to the container and not the
worker -- the exact swap this prevents.
…tainer (#13236)

A fifth gap, found by checking whether the contract could actually serve
web_fetch end to end after the format split.

_fetch_playwright sends timeout: int(timeout * 1000) to the container's
render endpoint. ExtractRequest had no timeout field -- NavigateRequest and
ScreenshotRequest both do, so it was the odd one out -- and container.extract
sent no timeout at all. Migrating web_fetch would therefore have silently
dropped the caller's timeout and fallen back to the container default,
with nothing failing to signal it.

ExtractRequest gains timeout_seconds; the container converts to the
milliseconds its endpoint expects, and omits the key entirely when the caller
passes None so the container default still applies rather than timeout=0.

46 tests, up from 44.
@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

✅ SSOT Configuration Compliance: Passing

🎉 No hardcoded values detected that have SSOT config equivalents!

@mrveiss
mrveiss merged commit 82c8c9c into Dev_new_gui Aug 2, 2026
1 check was pending
@mrveiss
mrveiss deleted the issue-13236-format branch August 2, 2026 15:07
mrveiss added a commit that referenced this pull request Aug 2, 2026
…13236) (#13309)

ADR-009 step 2 -- the first caller migrated off a directly-named stack, now
that the five contract gaps the audit found are closed.

_fetch_playwright imported services.playwright_service directly. It now
declares what it actually needs and lets dispatch resolve it:

  requires={Capability.EXTRACT_HTML, Capability.OUT_OF_PROCESS}

EXTRACT_HTML because the result feeds a parser, so text will not do.
OUT_OF_PROCESS because the entire reason this fallback exists is that
Playwright must not run inside the backend's own process.

Behaviour-preserving, and that is not incidental. Both requirements are
load-bearing: the worker and the container BOTH satisfy out-of-process
extraction, and register_all registers the worker first, so under the
pre-#13306 single EXTRACT capability this caller would have resolved to the
worker and received get_text output -- plain text where a parser expects
HTML, with nothing raising, because text is a valid result shape.

The inline SSRF check is KEPT rather than deferred to the registry's guard.
It is deliberate defence in depth: the container has network reach to
private IPs, and one extra DNS resolution is cheaper than depending on a
single chokepoint. The docstring says so, so a later reader does not remove
it as redundant.

Six tests pin that this lands on the same stack it called before, sends the
same render payload with the same wait, forwards the caller's timeout, keeps
returning None on failure so the render cascade can fall through, and that
the worker is never selected despite registering first.

186 tests pass across autobot-backend/tests/unit/web_fetch,
test_untrusted_web_content_boundary.py and autobot-backend/web_fetch.

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant