fix: wire three declared-but-not-enforced defects (research budget, link-pipeline cap, stale debug assertion) (#13013, #13021, #13037)

[mrveiss]

Thinking Path

Three issues filed from prior sessions' reading, each in the same defect class -- something declared/asserted that was never actually wired to reality. Verified each premise against current code before touching anything (file:line), per this session's repeated finding that filed premises are sometimes wrong.

• ResearchBudget.truncated_sources field is never populated (dead scaffolding since #12622) #13013: ResearchBudget.truncated_sources (services/research/models.py:88) is a dataclass field, never appended to anywhere -- confirmed via grep across services/research/*.py. The one genuine, existing truncation point in the pipeline is extractor.py:102's content[:max_content_chars] slice, driven by budget.max_content_chars -- but extract_claims() only receives the int, not the budget object, so it can't record into it itself. Wired the check at the call site (orchestrator.py._land_page, which holds both the budget and the pre-slice markdown) instead of duplicating the slicing logic.
• media/link/pipeline.py: _MAX_CONTENT_LENGTH (1MB cap) declared but never enforced in _fetch_and_parse #13021: media/link/pipeline.py:79 declares _MAX_CONTENT_LENGTH = 1_000_000 -- confirmed via grep -n "_MAX_CONTENT_LENGTH" returning only the declaration. _fetch_and_parse's BS4 fallback did await response.text(...) unbounded. Reachability: confirmed via security(web_fetch/media): fetcher.py and link/pipeline.py have the same validate-then-connect SSRF TOCTOU as content_reach (#13017) #13019's own text that LinkPipeline/MediaPipelineManager has zero production callers (api/knowledge_scrape.py and peers call web_fetch.WebFetcher directly); process_url() is only referenced from its own test file. This is a latent/unwired gap, not a live exploitable one -- fixed per "never delete code, wire it in" ahead of any endpoint being wired to this pipeline, not framed as an active incident.
• tool_registry_debug_e2e_test.py asserts Orchestrator.tool_registry, an attribute that doesn't exist #13037: tools/tool_registry_debug_e2e_test.py:25 asserts orchestrator.tool_registry. Confirmed via orchestrator.py (root) that Orchestrator only ever had self.agent_registry (line 159) -- no tool_registry, available_tools, _execute_planned_action, or local_worker either, all also referenced by this same stale script. git log --follow shows the file untouched except repo-wide renames/relicensing since the project's earliest history, predating the refactor(backend): one conductor — single Orchestrator class, rename 6 misnamed '*Orchestrator' classes #5040 "single conductor" orchestrator consolidation. The script's intent (prove the tool-dispatch pipeline is wired and runs a tool) is still valid; its premise about where that lives was wrong -- the real surface is the canonical tools.tool_registry.get_tool_registry() singleton, dispatched from task_handlers/executor.py/api/agent.py, not anything owned by Orchestrator.

All three were fully deliverable -- none needed a scope-changing decision, so nothing was dropped.

What Changed

#13013 (services/research/orchestrator.py, orchestrator_test.py):

• Added ResearchOrchestrator._record_truncation_if_needed(), called from _land_page right after a page is fetched: if len(markdown) > budget.max_content_chars (the same bound extract_claims slices to), appends the URL to budget.truncated_sources and logs it.
• 2 new tests proving population on an oversized source and non-population on an in-budget one.

#13021 (media/link/pipeline.py, pipeline_test.py, autobot_shared/ssot_config.py):

• New link_pipeline_max_content_bytes field on ssot_config (AUTOBOT_LINK_PIPELINE_MAX_CONTENT_BYTES, default 1_000_000) -- replaces the bare literal.
• New _read_bounded_content() streams the fallback fetch response in 64 KiB chunks (mirroring web_fetch/fetcher.py._fetch_bs4's existing pattern) and returns None the instant the cap is crossed -- remaining chunks are never requested, so an oversized/slow-drip body is cut off mid-stream, not read fully then discarded.
• A crossed cap logs "... REJECTED ... body exceeded max_content_length ... (distinct from a fetch/connection failure)" at WARNING, separate from the existing SSRF/connection/HTTP-error log lines.
• 3 new tests: oversized body rejected + never reaches _parse_html; chunk consumption stops well short of the full stream once the cap is crossed (proves the cut, not just a flag); in-budget body still parses normally (regression guard).
• Updated the 3 existing mock-response builders in pipeline_test.py (shared by TestLinkPipelineHttp and TestLinkPipelineJina's BS4-fallback tests, including PR fix(web_fetch): pin SSRF guard connect target in fetcher.py + link/pipeline.py (#13019) #13022's SSRF-pinning-adjacent fixtures) to also supply response.content.iter_chunked(), since the read path moved off response.text(). No assertions in those pre-existing tests changed.

#13037 (tools/tool_registry_debug_e2e_test.py):

• Rewrote the two crash-prone/false-positive functions into 3 real, passing tests: Orchestrator has agent_registry (populated) and never tool_registry; the canonical get_tool_registry() singleton has tools wired (respond_conversationally present in get_available_tools()); and execute_tool("respond_conversationally", ...) actually runs end-to-end (mirrors the original script's _execute_planned_action intent, now via the real dispatch method).

Verification

python3 -m py_compile <all changed .py>                         # OK
python3 -m flake8 --max-line-length=120 <all changed .py>       # 0
python3 -m black --check --line-length=120 <all changed .py>    # unchanged

Per-issue tests:

services/research/orchestrator_test.py::TestTruncatedSourcesRecording   2 passed
media/link/pipeline_test.py::TestLinkPipelineContentLengthCap           3 passed
tools/tool_registry_debug_e2e_test.py                                   3 passed

Full related-scope sweep (services/research, media/link, tools/tool_registry*, api/knowledge_scrape_test.py, ssot_config_test.py):

528 passed (services/research + media/link + tools/tool_registry* + api/knowledge_scrape_test.py + ssot_config_test.py + tests/test_startup_imports.py)

cp-swapped baseline (origin/Dev_new_gui content restored in-place, tests re-run, then my changes restored via cp from a scratch copy -- no git stash):

• Before: tests/test_startup_imports.py -- 350 passed
• After: tests/test_startup_imports.py -- 350 passed (identical ratio, no regression)
• PR fix(web_fetch): pin SSRF guard connect target in fetcher.py + link/pipeline.py (#13019) #13022's media/link SSRF-pinning tests (TestLinkPipelineFetchSSRFPinning, TestLinkPipelineHttp) and fix(knowledge): exclude quarantined research facts at every reachable KB search site (#13009) #13029/fix(knowledge): repair 4 broken KB-search call sites found by #13009's audit #13038's KB-search paths (api/chat_knowledge_test.py, knowledge/quarantine_test.py, services/research/quarantine_boundary_test.py) all still green, same assertions.

Model Used

Claude Sonnet 5

[github-actions]

✅ SSOT Configuration Compliance: Passing

🎉 No hardcoded values detected that have SSOT config equivalents!