security(content_reach): robots.txt fetch is unguarded and follows redirects; main path validates-then-connects (TOCTOU)

Summary

content_reach's URL guard validates the caller's URL but the robots.txt fetch it performs is itself unguarded, and the main fetch path validates-then-connects rather than pinning. Found while implementing #12625, where the SSRF requirements were tightened; not introduced by that work.

Defect 1 — the robots.txt fetch is completely unguarded and follows redirects

autobot-backend/content_reach/_url_guard.py:54-68:

async def _fetch_robots_text(domain: str) -> str:
    robots_url = f"{domain}/robots.txt"
    async with aiohttp.ClientSession() as session:
        async with session.get(robots_url, timeout=timeout, allow_redirects=True) as resp:

Raw aiohttp.ClientSession() — no pinned_connector, no is_public_url_async, no check of any kind.
allow_redirects=True, so a perfectly public domain can answer the robots request with a 302 to 169.254.169.254, 127.0.0.1, or any internal host, and this follows it.
domain derives from the caller-supplied URL via _extract_domain (_url_guard.py:48).
It is fail-open by design (returns "" on any error), so a successful internal fetch is indistinguishable from a failure in the logs.

Reached via _get_robots_parser → ensure_robots_allowed, which is imported by content_reach/backends/browser.py:21, content_reach/sources/web_page.py:21. So ordering ensure_public_url before ensure_robots_allowed does not help — the redirect happens inside the robots fetch, after validation.

Defect 2 — the main path validates then connects separately (TOCTOU)

ensure_public_url (_url_guard.py:112-119) delegates to autobot_shared.url_safety.is_public_url_async, which resolves the hostname and rejects private addresses — then returns, and the caller opens its own connection afterwards. Between the check and the connect, DNS can change: classic rebinding.

This repo already has the correct primitive and uses it elsewhere. autobot_shared/url_safety.py's own module docstring describes resolve_safe_ip_async as returning "the safe IP literal, enabling callers to connect directly to the resolved" address, and autobot_shared/security/ssrf_guard.py:101 provides pinned_connector, used by api/provider_auth.py, api/marketplace_sources.py, knowledge/connectors/oauth_flow.py and skills/external_importer.py. content_reach is the outlier.

Suggested fix

Route _fetch_robots_text through ssrf_guard.pinned_connector with allow_redirects=False, rejecting any 3xx — the pattern PR feat(research): topic-based source routing + data-driven search sources (#12625) #13016 established for config-declared sources.
Move ensure_public_url's callers onto pinned_connector (or resolve_safe_ip_async + connect-to-literal) so validation and connection cannot diverge.
Reconsider fail-open on the robots fetch, or at least log distinguishably when a robots request is refused by the guard rather than failing naturally.

Verification notes

Defect 1 is read directly off the source above. Defect 2 is the same validate-then-connect shape pinned_connector exists to close; worth confirming with a rebinding test of the kind in api/tests/test_provider_auth_ssrf.py, which mocks real DNS rather than mocking the guard away.

Scope of exposure depends on whether content_reach accepts caller-controlled URLs in the deployed configuration — worth confirming before prioritising.

Phase 1 verified both defects as reported (see PR thinking path for detail):

Defect 1 (robots.txt unguarded + follows redirects): confirmed exactly as filed, real caller-controlled reachability via content_reach/backends/browser.py and content_reach/sources/web_page.py.
Defect 2 (validate-then-connect TOCTOU): confirmed, but the fix shape needed correcting — content_reach's own fetch seam (content_reach/_http.py) is httpx-based, not aiohttp, so pinned_connector isn't directly reusable there; built an httpx-equivalent (_pinned_get) using resolve_safe_ip + Host/SNI-preserving IP-literal connect.
Scope-narrowing discovery: BrowserBackend/BrowserSearchBackend (Playwright) and YtDlpCaptionBackend's extract_info() hand the URL to a different subsystem's own network stack — pinning our own connector doesn't protect those. Filed separately as security(content_reach): browser + yt-dlp fetches keep a DNS-rebind TOCTOU window that pinning our own connector cannot close #13018 (different mechanism, larger blast radius). Also found the identical pattern (out of scope here) in web_fetch/fetcher.py and media/link/pipeline.py — filed as security(web_fetch/media): fetcher.py and link/pipeline.py have the same validate-then-connect SSRF TOCTOU as content_reach (#13017) #13019.

Fix: PR #13020.

fix(content_reach): pin SSRF guard connect target to close robots.txt…

PR #13020 (merged to Dev_new_gui) references this issue with a close keyword.

fix(content_reach): pin SSRF guard connect target to close robots.txt redirect + TOCTOU gaps (#13017)

If this issue is fully resolved, close it manually. If work remains, no action is needed.

Fixed by PR #13020, merged to Dev_new_gui as e33ef8948.

Both defects confirmed real before fixing — I filed this issue from reading source rather than executing it, so each claim was re-verified:

Defect 1 confirmed exactly as reported. _url_guard.py:54-68 used a bare aiohttp.ClientSession() with allow_redirects=True and no guard, and domain derives from a caller-supplied URL via _extract_domain (line 48).
Reachability confirmed as caller-controlled, not inert. _fetch_robots_text → _get_robots_parser → ensure_robots_allowed, imported by content_reach/backends/browser.py:21 and content_reach/sources/web_page.py:21; both pass request.url straight from the caller via TrafilaturaBackend.fetch, JinaReaderBackend.fetch and BrowserBackend.fetch.
Defect 2 confirmed, but my suggested fix was wrong. I proposed reusing ssrf_guard.pinned_connector. That returns an aiohttp.TCPConnector, and content_reach's actual fetch seam (content_reach/_http.py) is httpx-based — so the helper was not reusable there at all. No httpx-compatible pinning helper existed anywhere in the repo.

Fix:

_fetch_robots_text resolves through ssrf_guard.pinned_connector before opening a session, allow_redirects=False, any 3xx rejected outright. Guard rejections now log distinctly ("...rejected outright by SSRF guard") from natural failures ("...fetch failed"), so a blocked internal fetch can no longer hide inside the fail-open path — fail-open is preserved for genuine network errors.
New _pin_host / _pinned_get in content_reach/_http.py: URL rewritten to the resolved IP literal with the original Host header and TLS SNI preserved via httpx extensions={"sni_hostname": ...}, verified against httpcore source plus a live probe to confirm the socket connects to the pinned IP while the certificate still validates against the real hostname. The injected-client test path is untouched.
sources/web_page.py, reddit.py, youtube.py handle the new SSRFError, converting to the existing BackendError contract with a distinguishing warning.

Hostile-test evidence — real DNS mocked at autobot_shared.url_safety.socket.getaddrinfo, the guard itself never mocked away (mirrors api/tests/test_provider_auth_ssrf.py):

tests/content_reach/test_url_guard.py — robots.txt blocked for private/loopback/link-local metadata; a public domain answering robots.txt with a 302 to an internal address is rejected outright and logged distinctly, which is the exact defect-1 vector; a genuine public domain still succeeds.
tests/content_reach/test_http_pinned_get.py (new, 11 tests) — internal-IP/localhost/metadata blocked with zero network calls made; a DNS-rebinding case where the hostname resolves private at connect time; and a success case asserting via httpx.MockTransport that the connect target is the IP literal while Host/SNI keep the original hostname.

Verification: py_compile / flake8 --max-line-length=120 / black --check --line-length=120 clean across all 7 files; cp-swapped baseline 479 passed / 0 failed → 495 passed / 0 failed after (+16 net new), startup-import smoke 350/350 unchanged.

This does NOT close the whole exposure — two follow-ups are filed, deliberately not folded in:

security(content_reach): browser + yt-dlp fetches keep a DNS-rebind TOCTOU window that pinning our own connector cannot close #13018 — BrowserBackend/BrowserSearchBackend (Playwright via research_browser_manager.research_url()) and YtDlpCaptionBackend (yt-dlp's own extract_info()) hand the URL to a different subsystem's network stack. We never open those connections, so pinning our own connector protects nothing there; a different mechanism is required and the TOCTOU window stays open on those paths.
security(web_fetch/media): fetcher.py and link/pipeline.py have the same validate-then-connect SSRF TOCTOU as content_reach (#13017) #13019 — the identical validate-then-connect pattern exists in web_fetch/fetcher.py and media/link/pipeline.py, out of scope for this issue.

Navigation Menu

Repository navigation

security(content_reach): robots.txt fetch is unguarded and follows redirects; main path validates-then-connects (TOCTOU) #13017

Summary

Defect 1 — the robots.txt fetch is completely unguarded and follows redirects

Defect 2 — the main path validates then connects separately (TOCTOU)

Suggested fix

Verification notes

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

security(content_reach): robots.txt fetch is unguarded and follows redirects; main path validates-then-connects (TOCTOU) #13017

Description

Summary

Defect 1 — the robots.txt fetch is completely unguarded and follows redirects

Defect 2 — the main path validates then connects separately (TOCTOU)

Suggested fix

Verification notes

Activity

mrveiss commented on Jul 30, 2026

github-actions commented on Jul 30, 2026

mrveiss commented on Jul 30, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions