Research agent P3: topic-based source routing + data-driven source definitions #12625

Activity

1. mrveiss

   added

   enhancementNew feature or requestNew feature or request

   priority: medium

   backend

   knowledge-base

   research

   on Jul 26, 2026
2. mrveiss

   added a commit that references this issue on Jul 30, 2026

   feat(research): topic-based source routing + data-driven search sourc…

   ...

   a41fcb5
3. github-actions commented on Jul 30, 2026

   

   github-actionsbot

   on Jul 30, 2026 – with GitHub Actions

   Contributor

   More actions

   PR #13016 (merged to Dev_new_gui) references this issue with a close keyword.

   feat(research): topic-based source routing + data-driven search sources (#12625)

   If this issue is fully resolved, close it manually. If work remains, no action is needed.
4. mrveiss commented on Jul 30, 2026

   

   mrveiss

   on Jul 30, 2026

   OwnerAuthor

   More actions

   Closed by PR #13016, merged to Dev_new_gui as a41fcb540.

   Delivered: services/research/router.py (topic inference → provider preference), agent_loop/search/config_declared_provider.py (SourceDefinition + ConfigDeclaredSearchProvider), and config/research_sources.yaml / config/research_topics.yaml. SearchProviderRegistry's credential-gating and fallback loop are preserved unchanged; the only registry-side addition is one attribute, WebSearchProvider.supported_categories.

   This issue's "Depends on" line was wrong. It states the merged Planner (#12624 / PR #13014) "produces the topic-tagged sub-questions the router consumes." It does not — planner.py's SubQuestion (line 46) carries only text and expected_value, and the decompose prompt (lines 33-42) never asks for a topic. Rather than reopen the already-merged and tested Planner for a larger blast radius plus an extra LLM round-trip per sub-question — which would work against the efficiency goal Phase 2 exists to serve — the Router infers topic itself via cheap deterministic keyword matching declared in config/research_topics.yaml. Tracked as #13015, low priority, since the keyword router works today.

   Also worth recording: CATEGORY_GENERAL/NEWS/CODE/ACADEMIC and a category= parameter already existed in agent_loop/search/base.py and looked like topic routing. They are not — category only shapes the query sent to whichever provider was already chosen (searxng_provider.py:34-79, brave_provider.py:57-90); no provider declared what it specialises in and the registry never selected on it. Provider selection is what this issue actually needed, and is what was added.

   SSRF — the load-bearing part of this issue. Config-declared sources mean caller-influenced URL templates get fetched by the server, so this reuses autobot_shared.security.ssrf_guard.pinned_connector (the same helper as api/provider_auth.py, knowledge/connectors/oauth_flow.py, skills/external_importer.py) rather than a new validator. Per request: resolve-and-pin fresh every call (never cached), allow_redirects=False with any 3xx rejected outright, and the runtime query passed only via aiohttp's params= dict — never interpolated into the URL string — so it structurally cannot alter scheme, host or port. Deliberately not routed through the pooled autobot_shared.http_client, matching the documented #12278/#12979 carve-out. config/research_sources.yaml ships empty: no source, no fetch, until explicitly opted in.

   18 hostile-case tests in agent_loop/search/config_declared_provider_test.py: internal IP, localhost, link-local metadata (169.254.169.254), DNS-rebind-to-private/loopback/metadata at request time (real DNS mocked the way api/tests/test_provider_auth_ssrf.py does it, not mocked away), redirect-to-internal rejected without following, and a query crafted to look like a scheme/host change confirmed to land only in params.

   Quarantine boundary intact. Config-declared sources only return SearchResult URLs flowing through the unchanged _discover_sources → _fetch_source → _store_source_document/_store_claim path, so every fact still receives collection: research and must pass the #12623 promotion gate. No new KB write path was added.

   Verification: py_compile / flake8 --max-line-length=120 / black --check --line-length=120 clean; cp-swapped baseline (never git stash, restored files diffed byte-identical) 398 passed / 0 failed before → 426 passed / 0 failed after; 10 router tests covering fallback-to-general-web, credential-gated specialist unavailable, erroring specialist, zero wasted general-web calls when a specialist satisfies, and an end-to-end YAML-only source with no Python subclass authored.

   Discovered separately: #13015 (planner emits no topic tag) and #13017 (content_reach robots.txt fetch is unguarded and follows redirects; main path validates-then-connects).
5. mrveiss

   closed this as completedon Jul 30, 2026
6. mrveiss

   added a parent issue on Sep 1, 2026

   • Umbrella: Precision & Efficiency Research Agent (KB-native) #12621