In haystack/components/fetchers/link_content.py the methods _fetch (lines 327‑340) and _fetch_async (lines 357‑371) each contain a finally block that does:
self.current_user_agent_idx = 0
The fetcher rotates the User‑Agent header on each retry via _switch_user_agent (line 468). The rotation works within a single request’s retry loop, but after the request finishes the finally block resets the index to 0. Consequently, every new request starts again with the first User‑Agent, so the intended round‑robin rotation across different requests never occurs.
In haystack/components/fetchers/link_content.py the methods _fetch (lines 327‑340) and _fetch_async (lines 357‑371) each contain a finally block that does:
The fetcher rotates the User‑Agent header on each retry via _switch_user_agent (line 468). The rotation works within a single request’s retry loop, but after the request finishes the finally block resets the index to 0. Consequently, every new request starts again with the first User‑Agent, so the intended round‑robin rotation across different requests never occurs.