fix(web_fetch): add SSRF guard for self-hosted providers by DengY11 · Pull Request #3942 · bytedance/deer-flow

DengY11 · 2026-07-04T15:40:04Z

Summary

This PR hardens self-hosted web_fetch providers against SSRF.

DeerFlow already had URL safety validation for the Browserless web tool path: it rejects loopback, private, link-local, cloud metadata, reserved, multicast, and unresolved targets by default, with an explicit allow_private_addresses: true opt-in for intentional internal access.

However, later-added self-hosted web_fetch providers, especially Crawl4AI, did not reuse that protection. An attacker could potentially guide the agent into calling web_fetch on internal or metadata endpoints such as:

http://169.254.169.254/latest/meta-data/
http://localhost:...
http://10.x.x.x/...
http://*.svc.cluster.local/...

When Browserless/Crawl4AI runs inside the deployment network, this can turn web_fetch into an SSRF primitive and leak sensitive internal information.

What Changed

Extracted the URL safety check into deerflow.community.url_safety.
Applied the shared guard to Browserless web_fetch.
Applied the shared guard to Crawl4AI web_fetch.
Kept the existing explicit allow_private_addresses: true opt-in for trusted internal targets.
Updated config.example.yaml to document the SSRF guard for Browserless and Crawl4AI web_fetch.

Validation

I started the Gateway with make gateway and ran runtime smoke tests against a local fake Browserless/Crawl4AI service.

The smoke test verified that normal public web_fetch still reaches the fetch service, metadata URLs are blocked before reaching the service, and allow_private_addresses: true still permits intentional private fetches. This confirms the feature remains functional and the new validation logic is effective.

Related Browserless/Crawl4AI tests and formatting/lint checks were also run successfully.

CLAassistant · 2026-07-04T15:40:10Z

All committers have signed the CLA.

Copilot

Pull request overview

This PR hardens DeerFlow’s self-hosted web_fetch providers against SSRF by extracting the existing Browserless URL validation into a shared deerflow.community.url_safety utility and applying it consistently to both Browserless and Crawl4AI (with an explicit allow_private_addresses opt-in retained for intentional internal access).

Changes:

Added shared URL screening utilities in deerflow.community.url_safety and reused them in Browserless + Crawl4AI web_fetch.
Added/updated tests to ensure metadata/private/loopback targets are rejected by default and allowed only with explicit opt-in.
Documented the allow_private_addresses SSRF guard in config.example.yaml for both providers.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
config.example.yaml	Documents the SSRF guard (`allow_private_addresses`) for Browserless and Crawl4AI `web_fetch`.
backend/tests/test_crawl4ai_tools.py	Adds SSRF regression tests for Crawl4AI `web_fetch` behavior.
backend/tests/test_browserless_client.py	Adds SSRF regression tests for Browserless `web_fetch` behavior.
backend/packages/harness/deerflow/community/url_safety.py	Introduces shared URL validation + DNS resolution helpers for SSRF screening.
backend/packages/harness/deerflow/community/crawl4ai/tools.py	Applies shared URL validation to Crawl4AI `web_fetch` with config-controlled opt-in.
backend/packages/harness/deerflow/community/browserless/tools.py	Replaces local SSRF validation with shared guard and applies it to Browserless `web_fetch`.

+def is_blocked_address(address: ipaddress._BaseAddress) -> bool:
+    """Return True for addresses web tools should not reach by default."""
+    return address.is_private or address.is_loopback or address.is_link_local or address.is_reserved or address.is_multicast or address.is_unspecified


+
+    normalized_host = hostname.strip().rstrip(".").lower()
+    if normalized_host in _BLOCKED_HOSTNAMES:
+        return f"Error: Refusing to {action} a private or loopback address"


+        wait_for_event = cfg.get("wait_for_event", wait_for_event)
+        raw_wait = cfg.get("wait_for_timeout_ms", wait_for_timeout_ms)
+        wait_for_timeout_ms = int(raw_wait) if not isinstance(raw_wait, int) else raw_wait
+        wait_for_selector = cfg.get("wait_for_selector", wait_for_selector)


+        with patch("deerflow.community.crawl4ai.tools._get_tool_config", return_value={"allow_private_addresses": True}):
+            result = await tools.web_fetch_tool.ainvoke("http://10.0.0.5/dashboard")


+        with patch("deerflow.community.browserless.tools._get_tool_config", return_value={"allow_private_addresses": True}):
+            result = await tools.web_fetch_tool.ainvoke("http://10.0.0.5/dashboard")


fancyboi999

Left one concrete follow-up: the new shared SSRF guard only covers Browserless and Crawl4AI, but fastcrw.web_fetch_tool() is still documented/configured for self-hosted use and still forwards arbitrary URLs without equivalent screening. That leaves the same internal-target fetch primitive available through another current web_fetch provider, so I don't think the hardening is complete yet.

fancyboi999 · 2026-07-04T16:12:49Z

This still leaves one self-hosted web_fetch path unprotected: fastcrw.web_fetch_tool() is documented as supporting base_url: http://localhost:3000 for self-hosting, but it still forwards arbitrary URLs straight into client.scrape(...) without the new SSRF gate. That means the attack this PR is fixing remains reachable whenever users switch to the fastCRW provider, so the hardening is incomplete for the repo's current self-hosted fetch surface. Please either extend the shared validate_public_http_url(...) guard to backend/packages/harness/deerflow/community/fastcrw/tools.py as well (and add matching tests), or narrow the PR/docs claim so it does not imply self-hosted providers are now covered broadly.

DengY11 · 2026-07-04T17:16:26Z

@fancyboi999 Thanks for the catch. I extended the shared SSRF guard to fastCRW web_fetch as well, including metadata/private/DNS-to-private rejection tests and the explicit allow_private_addresses opt-in path. I also updated config.example.yaml to document the fastCRW allow_private_addresses option.

The concern has been addressed.

Conflict resolutions: - docker/docker-compose.yaml: kept local dir-mount hot-reload hardening (103f179) for config.yaml/extensions_config.json; adopted upstream's new Redis stream-bridge service, depends_on, and env wiring. - backend/.../community/crawl4ai/* + backend/tests/test_crawl4ai_tools.py: took upstream (bytedance#3821/bytedance#3942) — our provider plus SSRF guard, timeout coercion, and expanded tests. Module is now upstream-tracked. - csrf_middleware.py/auth.py auto-merged: our bytedance#3872 fix landed upstream. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

) Co-authored-by: Willem Jiang <willem.jiang@gmail.com>

github-actions Bot added risk:medium Medium risk: regular code changes size/M PR changes 100-300 lines labels Jul 4, 2026

DengY11 force-pushed the fix/web-fetch-ssrf-guard branch from 2606e05 to 48b10cb Compare July 4, 2026 15:43

WillemJiang requested a review from Copilot July 4, 2026 16:09

Copilot started reviewing on behalf of WillemJiang July 4, 2026 16:10 View session

Copilot AI reviewed Jul 4, 2026

View reviewed changes

fancyboi999 previously requested changes Jul 4, 2026

View reviewed changes

fix(web_fetch): add SSRF guard for self-hosted providers

52a5b0f

DengY11 force-pushed the fix/web-fetch-ssrf-guard branch from 48b10cb to 52a5b0f Compare July 4, 2026 17:15

github-actions Bot added size/L PR changes 300-700 lines and removed size/M PR changes 100-300 lines labels Jul 4, 2026

WillemJiang added the reviewing A maintainer is reviewing this PR label Jul 5, 2026

WillemJiang added this to the 2.1.0 milestone Jul 5, 2026

Merge branch 'main' into fix/web-fetch-ssrf-guard

7af3bb0

WillemJiang approved these changes Jul 5, 2026

View reviewed changes

WillemJiang merged commit 12eda6c into bytedance:main Jul 5, 2026
11 checks passed

ivan-skvorc mentioned this pull request Jul 11, 2026

Sync fork with upstream bytedance/deer-flow (114 commits) ivan-skvorc/AI-at-home#14

Merged

7 of 9 tasks

marvin9551 pushed a commit to marvin9551/deer-flow that referenced this pull request Aug 21, 2026

fix(web_fetch): add SSRF guard for self-hosted providers (bytedance#3942

dc4fdd8

) Co-authored-by: Willem Jiang <willem.jiang@gmail.com>

jihtsan pushed a commit to jihtsan/dnx-deer-flow that referenced this pull request Aug 29, 2026

fix(web_fetch): add SSRF guard for self-hosted providers (bytedance#3942

d67d3d8

) Co-authored-by: Willem Jiang <willem.jiang@gmail.com>

Navigation Menu

Repository navigation

fix(web_fetch): add SSRF guard for self-hosted providers - #3942

fix(web_fetch): add SSRF guard for self-hosted providers#3942
WillemJiang merged 2 commits into
bytedance:mainfrom
DengY11:fix/web-fetch-ssrf-guard

DengY11 commented Jul 4, 2026

Uh oh!

CLAassistant commented Jul 4, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

fancyboi999 left a comment

Uh oh!

fancyboi999 Jul 4, 2026

Uh oh!

DengY11 commented Jul 4, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

		with patch("deerflow.community.crawl4ai.tools._get_tool_config", return_value={"allow_private_addresses": True}):
		result = await tools.web_fetch_tool.ainvoke("http://10.0.0.5/dashboard")

		with patch("deerflow.community.browserless.tools._get_tool_config", return_value={"allow_private_addresses": True}):
		result = await tools.web_fetch_tool.ainvoke("http://10.0.0.5/dashboard")

Conversation

DengY11 commented Jul 4, 2026

Summary

What Changed

Validation

Uh oh!

CLAassistant commented Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

fancyboi999 left a comment

Choose a reason for hiding this comment

Uh oh!

fancyboi999 Jul 4, 2026

Choose a reason for hiding this comment

Uh oh!

DengY11 commented Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

CLAassistant commented Jul 4, 2026 •

edited

Loading

DengY11 commented Jul 4, 2026 •

edited

Loading