Feature request: opt-in block_internal_urls egress filter for the library layer #2146

Thanks for following up from the email thread, @ntohidi.

Quick correction on my side: the issue body initially only contained a literal file path — I mistakenly relied on @path expansion with gh api, which doesn't expand files (that's a curl / --body-file flag). Pasting the real content here.

Alignment

We're aligned on the classification. The library is a user agent invoked by a trusted caller, so destination filtering should stay opt-in, and the SSRF trust boundary remains at the Docker API server where egress_broker.py already enforces it. The agentic / LLM-chosen-URL case is the scenario that justifies exposing the same primitives to library callers.

Proposed design

• Flag: block_internal_urls: bool (default False, opt-in). Set per-crawl so callers who embed Crawl4AI in an agent can opt in without a global change.
• Chokepoint: a single host-validation call inserted right after the existing scheme allow-list check in AsyncCrawlerStrategy.crawl() (crawl4ai/async_crawler_strategy.py). It must cover both egress paths:
  • HTTP path: AsyncHTTPCrawlerStrategy._handle_http() (aiohttp)
  • Browser path: browser_manager.py → Playwright goto()
• Logic reuse: port the existing validate_url_destination + resolve_and_pin (DNS pinning) + per-hop redirect revalidation from deploy/docker/utils.py / egress_broker.py into a shared helper (e.g. crawl4ai/url_safety.py) so the library and the Docker server share one implementation — no duplicated trust logic.
• Blocked ranges: loopback (127.0.0.0/8, ::1), private (10/8, 172.16/12, 192.168/16, fc00::/7), link-local (169.254/16 incl. cloud metadata 169.254.169.254, fe80::/10), and 0.0.0.0/8. The resolved IP must be checked after DNS (pin) and after every redirect hop (revalidate) to prevent DNS-rebinding / redirect-to-internal bypasses.
• Behavior on block: raise a BlockedURL exception (or return a failed CrawlResult with a clear error) rather than fetching.

Open questions (happy to match maintainer preference)

1. Flag name — block_internal_urls vs deny_private_destinations vs egress_filter?
2. Where the chokepoint sits — shared base crawl() vs per-strategy hooks?
3. Browser redirect following — should the redirect revalidation also cover hops taken by Playwright goto, or only the initial URL?

I'd be happy to draft the PR implementing this once we settle the surface.