kb/web-connector: web_crawler yields 0 docs + opaque 'empty/failed' on Cloudflare/JS-challenged sites — FAST fetch only, no browser-render fallback, error drops status_code #12486

Summary

Adding a web (web_crawler) connector in the Knowledge Base against a site protected by an anti-bot / JS challenge (e.g. Cloudflare) silently yields 0 documents. The sync shows partial, +0 ~0 -0, in ~0.0-0.2s, with a single opaque error: empty/failed: <url>. The operator has no way to tell why nothing was ingested.

Reproduction

Add a web connector for a public URL that sits behind Cloudflare's anti-bot challenge and sync it. A direct fetch from the host confirms the block:

$ curl -A "Mozilla/5.0" -i https://<cloudflare-protected-site>
HTTP/2 403
<title>Just a moment...</title>   # Cloudflare interstitial, ~5 KB, no real content

Result in the KB sync: status partial, added/modified/deleted all 0, 1 err: empty/failed: <url>.

Root cause

knowledge/connectors/web_crawler.py:

1. Fetches with render_mode=RenderMode.FAST — a plain HTTP fetch, no JS/browser rendering. Cloudflare (and any JS-gated site) returns a 403 "Just a moment..." interstitial instead of the page.
2. _fetch_result_to_content() returns None when not result.success or not result.markdown.strip(). The challenge page has no extractable markdown, so it becomes None.
3. _ingest_results_to_kb() then records sync_result.errors.append("empty/failed: %s" % fetch_result.url) — discarding result.status_code / result.error_code, so the report can't distinguish a bot-block (403) from a genuinely empty page, a robots block, or a network error.
4. Retry logic only retries on status_code is None or 429/500/502/503/504 — 403 is not retried and does not trigger any fallback.
5. A browser-render path exists (AUTOBOT_BROWSER_SERVICE_HOST / browser service) but is not used as a fallback when a plain fetch is challenged. (Note: the autobot-browser-worker unit was also observed inactive, so even a fallback would need that worker running.)

Impact

A large fraction of modern public sites sit behind Cloudflare or render content via JS. For all of them the web connector adds zero content and reports only "empty/failed", which reads as a bug with no actionable cause.

Requested (file only — no remediation)

• Surface the real reason in the sync error: include status_code / error_code (e.g. failed: HTTP 403 (anti-bot challenge): <url>), and detect the Cloudflare/JS-interstitial case explicitly.
• Add a browser-render fallback: when a FAST fetch is challenged (403 / interstitial / empty markdown), retry via RenderMode.BROWSER through the browser service so JS-gated and Cloudflare sites can be crawled. Ensure autobot-browser-worker is running for this path (surface a clear message if it is not).
• Consider honouring a per-connector "render mode" option so operators can force browser rendering for known-JS sites.

Notes

• Diagnosed from a real user attempt (web connector -> empty/failed). The target is a public, Cloudflare-protected site; the internal deployment is unaffected. Filed only — no code changes.