Repository navigation
【No.10】feat(deepeyes-v2-agentic): pluggable web-search backends for the search tool - #367
Open
TabsPhasers wants to merge 1 commit into
Open
TabsPhasers wants to merge 1 commit into
TabsPhasers wants to merge 1 commit into
Conversation
…ch tool
Replaces the hardcoded placeholder `search()` with a configurable backend
dispatch (mock / Search-R1 retriever / external web-search API), so the
example keeps running offline by default while real deployments can point
the same tool at a retrieval service or a hosted search API.
# ⭐ Feature
- app/search_backends.py: `SearchConfig` (env-driven config), the three
backends and the `get_search_backend()` factory. `mock` (default) is
deterministic and offline (no randomness, same query -> same results);
`retriever` speaks the Search-R1 wire protocol (`POST {url}` with
`{"queries": [...], "topk": k, "return_scores": false}` -> `result[0]`,
matching examples/search_r1/retrieval_server.py, with `contents` as the
snippet fallback); `external` defaults to the Serper.dev `POST /search`
preset and makes endpoint, auth header, request fields and response field
mapping configurable via env vars. API keys are only ever read from the
environment, never committed.
- Uniform result contract for every backend:
`{"elapsed_time": float, "data": [{"title", "link", "snippet",
"date"|None}, ...]}`; missing title/link/snippet degrade to "" and date to
None, so downstream rendering never sees partial rows.
- Failure semantics follow the env's existing convention: timeouts,
connection errors, 5xx, unparsable JSON and structurally invalid payloads
are retried (linear backoff, configurable) then returned as the literal
`"Error"` string with a logged reason — the agent process keeps running.
Retry semantics mirror the existing
examples/on_policy_distillation/.../retrieval_client.py client; 4xx
statuses fail immediately without burning retries.
- app/search_utils.py: `search()` is now a thin dispatch (explicit `size`
wins over `DEEPEYES_V2_SEARCH_TOP_K`, default 5 as before); the random
sleep retry loop is gone — retries live in the real backends only, mock
stays deterministic. `image_search` is unchanged.
- run_deepeyes_v2_agentic.sh / _klx.sh: forward the search backend env vars
into the Ray workers — runtime-env-json is an explicit whitelist, so
without this a configured backend would silently fall back to `mock` in
real training. The JSON-valued `_FIELD_MAP` is quote-escaped so the
generated JSON stays parseable.
- Zero new dependencies: uses `requests`, already in requirements.txt.
- Config lives in environment variables (documented in env.sh.example) so
endpoints/keys stay in the gitignored env.sh, matching the example's
existing machine-local config pattern.
# ✅ Tests
- tests/examples/deepeyes_v2_agentic/test_search_backends.py: 29 unit tests
against a local stub HTTP service (127.0.0.1, no external network):
mock determinism + shape, config parsing (defaults/overrides/malformed),
retriever request construction + response mapping (Serper preset, custom
field maps, auth header presence/absence), and the failure matrix —
timeout, connection refused, 4xx no-retry, 5xx retry-then-success and
retry-exhausted, invalid JSON, missing/ill-typed fields — plus env-level
integration through DeepEyesV2Env.exec_tool asserting the "Error" →
search_failed observation path, and a guard test asserting both launch
scripts forward every search env var into the Ray workers.
# 📝 Documentation
- README "Text search backends" section: backend table, unified result
contract, failure behavior (including per-request timeout semantics and
worst-case wall time), the full env-var surface, and a one-shot recipe
for verifying a real retriever / external service.
- env.sh.example: optional search-backend config block with a placeholder
key slot in the gitignored template.
TabsPhasers
marked this pull request as ready for review
September 28, 2026 12:25
TabsPhasers
requested review from
Aurelius84,
NINGBENZHE,
Yangruipis and
yxyOo
as code owners
September 28, 2026 12:25
Contributor
Nyanpasu 审查看板审查状态： 💬 已完成 · 有补充意见 审查版本： 9bc1c2c 常规与深度审查均已完成。实现满足任务 No.10 全部显式需求：Search-R1 协议、统一返回结构、"Error" 失败约定、可配置超时/重试/top-k/字段映射、启动脚本透传均验证正确，无阻塞缺陷。共 3 条非阻塞待处理发现：P2 —— 超时/连接错误的重试语义未被测试钉住（变异验证：删除该重试分支后 29 个测试仍全绿）；P3 —— requests 默认 trust_env 使代理环境变量隐式路由搜索流量；P3 —— 重试耗尽消息报告配置次数而非实际次数。生产与测试必要性审计完成，结论为保留现有结构（已检查的更小替代方案及否决理由见精简审查表）。
审查发现待处理
已解决或已取代
提交范围 · 接收 7 · 建议移出 0 · 待确认 0接收 7 个文件 · 建议移出 0 个文件 · 待确认 0 个文件。移出与待确认部分暂停深审，不代表审查通过。
精简审查与验证依据
生产代码的必要性与替代方案
测试的必要性与替代方案
Powered by Nyanpasu with glm-5.3[1m] xhigh, please check the suggestions carefully.
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
rai-studio-bot
left a comment
rai-studio-bot
left a comment
Contributor
There was a problem hiding this comment.
常规审查完成（深度审查进行中）
已核查全部 7 个改动文件：Search-R1 线上协议与 examples/search_r1/retrieval_server.py 一致（return_scores: false 下取 result[0] 平铺语料行 + contents 回退正确）；重试/失败语义与既有 agentic_opd/search_qa retrieval client 对齐（5xx 重试、4xx 立即失败、max(1, max_retries) 次尝试）；"Error" 约定经 search() → _dispatch_search → exec_tool 端到端保持；search() 入口对唯一生产调用方向后兼容；两个启动脚本 13 个搜索环境变量全部透传，FIELD_MAP 引号转义生成的 runtime-env JSON 已验证可解析；README 默认值与超时数学准确。29 个新测试本地全部通过（~7s），未发现阻塞缺陷，已留下 1 条非阻塞建议（P3：代理环境变量对搜索后端的隐式影响，见行内评论）。独立设计对比与测试必要性审计仍在后台进行，结论稍后更新至看板。
审查看板：#367 (comment)
Powered by Nyanpasu with glm-5.3[1m] xhigh, please check the suggestions carefully.
| last_error: str | None = None | ||
| for attempt in range(1, attempts + 1): | ||
| try: | ||
| resp = requests.post(url, headers=dict(headers), json=payload, timeout=timeout_seconds) |
Contributor
There was a problem hiding this comment.
requests.post 默认 trust_env=True，会遵循进程环境里的 HTTP_PROXY / HTTPS_PROXY。在设置了代理的 worker 节点上，指向集群内检索服务的 retriever 请求会被静默路由到代理并失败（重试耗尽后返回 "Error"，训练不中断但搜索工具实际不可用，只有 warning 日志可查）。本 PR 参考的兄弟实现 examples/on_policy_distillation/agentic_opd/search_qa/app/retrieval_client.py 特意默认 trust_env=False（SEARCH_RETRIEVAL_TRUST_ENV 默认 0）；这里沿用了它的重试语义，但没有沿用这一默认值。
复现证据：在设有 HTTP_PROXY 且未设 NO_PROXY 的开发机上直接运行本套件，8/29 个 stub 测试因代理无法回连 127.0.0.1 而得到 "Error"；设 NO_PROXY=127.0.0.1,localhost 后 29/29 通过——差异仅来自代理路由。另外 run_deepeyes_v2_agentic*.sh 的 runtime-env 白名单不透传 NO_PROXY，用户在代理节点上无法只靠 env.sh 规避（klx 脚本第 35–36 行已在提交 shell 中 unset http_proxy https_proxy，说明代理干扰在该部署环境是真实存在的问题）。
非阻塞建议（任选其一）：
- 增加类似
DEEPEYES_V2_SEARCH_TRUST_ENV的开关（默认对齐兄弟实现设0），请求时改用requests.Session并设置trust_env；或 - 至少在 README 的 "Text search backends" 一节注明代理交互与
NO_PROXY配置方法，并把NO_PROXY加入两个启动脚本的透传白名单。
rai-studio-bot
left a comment
rai-studio-bot
left a comment
Contributor
There was a problem hiding this comment.
审查完成（深度审查已合并）
在常规审查基础上完成了深度审查：独立设计对比（在隔离的 merge-base 工作区构建最小设计，32/32 项针对真实调用方 DeepEyesV2Env._dispatch_search 的检查与 12/12 既有示例测试通过）和测试套件审计（11 个定向变异实验 + 精简套件等价性验证，基线 29/29 通过）。结论：实现满足任务 No.10 的全部显式需求——Search-R1 协议、统一返回结构、"Error" 失败约定、可配置超时/重试/top-k/字段映射、启动脚本透传均验证正确，无阻塞缺陷。
本轮新增两条非阻塞发现：P2 —— 模块文档与 PR 描述声明"连接失败、超时会先重试"，但删除 _post_json_with_retries 中对应的重试分支后 29 个测试仍全部通过（这些路径的测试都用 max_retries=1 或只断言 result == "Error"），建议按行内评论加强超时测试钉住该语义；P3 —— 重试耗尽的错误消息在 4xx 提前退出时报告配置次数而非实际次数。加上常规阶段的 trust_env 代理建议（P3），共 3 条待处理意见，均不阻塞合入。
生产结构与测试套件的必要性审计已记录在看板：类/工厂/配置层的更小函数式替代方案经独立验证可行，但其省略了任务显式要求的可配置重试与 top-k 旋钮，且现有测试已锚定当前 API，故结论为保留现有结构。
审查看板：#367 (comment)
Powered by Nyanpasu with glm-5.3[1m] xhigh, please check the suggestions carefully.
Comment on lines
+317
to
+325
| def test_retriever_backend_timeout_returns_error(): | ||
| stub = _StubSearchService() | ||
| url = stub.start() | ||
| try: | ||
| stub.delay_seconds = 1.0 | ||
| result = RetrieverSearchBackend(_retriever_config(url, timeout_seconds=0.2)).search("q", 1) | ||
| assert result == "Error" | ||
| finally: | ||
| stub.stop() |
Contributor
There was a problem hiding this comment.
模块文档与 PR 描述都明确声明"连接失败、超时……会先重试（线性退避）"，且 _post_json_with_retries 的 except (ConnectionError, Timeout) 分支专门实现了这一点；但当前套件没有钉住这条语义。变异验证：在一个一次性工作区里删掉该 except 分支（让超时/连接错误直接冒泡、只尝试 1 次），29 个测试仍然全部通过——因为走这些路径的测试要么用 _retriever_config 默认的 max_retries=1（1 次尝试本来就是期望值），要么只断言 result == "Error"。相对地，5xx 重试语义已被 test_retriever_backend_retries_exhausted_returns_error（len(stub.requests) == 3）牢牢钉住。
建议把超时测试加强为 max_retries=3 + 请求计数断言（顺带可对 connection-refused 做同样处理）；加强后的版本可检出上述变异（1 次 vs 3 次请求）：
Suggested change
| def test_retriever_backend_timeout_returns_error(): | |
| stub = _StubSearchService() | |
| url = stub.start() | |
| try: | |
| stub.delay_seconds = 1.0 | |
| result = RetrieverSearchBackend(_retriever_config(url, timeout_seconds=0.2)).search("q", 1) | |
| assert result == "Error" | |
| finally: | |
| stub.stop() | |
| def test_retriever_backend_timeout_returns_error(): | |
| stub = _StubSearchService() | |
| url = stub.start() | |
| try: | |
| stub.delay_seconds = 1.0 | |
| result = RetrieverSearchBackend( | |
| _retriever_config(url, timeout_seconds=0.2, max_retries=3) | |
| ).search("q", 1) | |
| assert result == "Error" | |
| assert len(stub.requests) == 3 # timeouts are retried before giving up | |
| finally: | |
| stub.stop() |
（retry_delay_seconds 已是 0.0，加强后套件运行时间几乎不变。）
| f"[search_backends] attempt {attempt}/{attempts} failed ({last_error}); retrying in {delay:.1f}s" | ||
| ) | ||
| time.sleep(delay) | ||
| raise RuntimeError(f"search request failed after {attempts} attempts: {last_error}") |
Contributor
There was a problem hiding this comment.
4xx 走 raise_for_status → break 提前退出时，这条消息仍然打印配置的 attempts（如 3），但实际只发了 1 次请求——排查训练日志时会把"不可重试的立即失败"误读成"重试耗尽"。参考的兄弟实现 agentic_opd/search_qa/.../retrieval_client.py 在同场景下报告的是实际次数。最小修复：循环变量 attempt 在循环结束后仍然存活，正常耗尽时等于 attempts，提前 break 时即实际次数，直接改用它即可：
Suggested change
| raise RuntimeError(f"search request failed after {attempts} attempts: {last_error}") | |
| raise RuntimeError(f"search request failed after {attempt} attempts: {last_error}") |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
把 DeepEyes-V2 示例里写死的占位
search()换成可插拔的后端分发：新增两个真实搜索后端，同时保留确定性的离线默认行为。examples/deepeyes_v2_agentic/app/search_backends.py（新增）：mock（默认）— 确定性离线结果：不联网、不需要密钥，同一查询永远返回相同结果。retriever— 调用 Search-R1 兼容的检索服务：向服务发POST {url}，请求体{"queries": [...], "topk": k, "return_scores": false}，取返回的result[0]（协议与
examples/search_r1/retrieval_server.py一致）；语料文档没有 snippet 字段时回退用contents。external— 通用 JSON 网页搜索 API。默认参数对齐 Serper.dev 的POST /search（
X-API-KEY认证头、{"q": …, "num": …}请求体、结果在organic字段下）；endpoint、认证头、请求字段名和响应字段映射全部可用环境变量调整。API 密钥只从环境变量读取，不写入代码。
{"elapsed_time": float, "data": [{"title", "link", "snippet", "date"|None}, ...]}。examples/deepeyes_v2_agentic/app/search_utils.py：search()改为薄分发入口；失败语义不变（返回字面量
"Error"，环境继续运行）。image_search未改动。examples/deepeyes_v2_agentic/env.sh.example、README.md：补齐配置面文档，包括单次请求超时语义和一次性验证真实 retriever/external 服务的方法。
run_deepeyes_v2_agentic.sh、run_deepeyes_v2_agentic_klx.sh：把搜索后端的环境变量透传进 Ray worker（
runtime-env-json是显式白名单，新变量不列进去训练时会静默留在
mock）。JSON 取值的_FIELD_MAP做了引号转义，保证生成的 JSON 可解析。
tests/examples/deepeyes_v2_agentic/test_search_backends.py（新增）：29 个测试。Why
Relates to #321（贡献者计划第二期 No.10）。
改动前
search()是硬编码占位实现，只返回固定文案（search_utils.py@ffa074d），DeepEyes-V2 训练无法接真实的检索服务或网页搜索 API。贡献者计划 No.10 任务（relates to #321）
要求提供可插拔的 mock / Search-R1 retriever / external 三种搜索后端，统一结果结构，
超时和重试可配置，且默认配置离线可跑。
How
（
DEEPEYES_V2_SEARCH_BACKEND、DEEPEYES_V2_SEARCH_TOP_K、_TIMEOUT_SECONDS、_MAX_RETRIES、_RETRY_DELAY_SECONDS、DEEPEYES_V2_RETRIEVER_URL、DEEPEYES_V2_EXTERNAL_SEARCH_ENDPOINT/_API_KEY/_AUTH_HEADER/_QUERY_FIELD/_TOPK_FIELD/ _RESULTS_FIELD/_FIELD_MAP）。这与该示例已有的机器本地env.sh模式一致，endpoint 和密钥留在被 gitignore 的
env.sh里，不进提交。normalize_results()把各后端的原始行映射为title/link/snippet（缺失 →""）和date（缺失 →None），下游渲染不会遇到缺字段的行。Search-R1 语料文档的正文在
contents字段，作为 snippet 的回退；external 后端的行通过
_FIELD_MAP重映射（例如'{"date": "published_date"}'）。"Error"约定：连接失败、超时、5xx、非法 JSON、结构不对的返回体会先重试（线性退避，对齐现有的
agentic_opd/search_qaretrieval client），重试耗尽后返回
"Error"并记录日志。DeepEyesV2Env.exec_tool把它渲染为既有的search_failed观测——agent 进程不会崩溃。配置错误（数值非法、缺少必需的 URL）在工厂函数里抛出，在
search()边界统一转成"Error"加警告日志。4xx 状态立即失败、不消耗重试（只有 5xx 和传输层故障可重试）。
DEEPEYES_V2_SEARCH_TIMEOUT_SECONDS约束的是单次 HTTP 请求；单次搜索调用的最坏耗时约为"超时 × 尝试次数 + 重试间隔"，README 已写明。
requests（已在 requirements.txt 里）。Testing
新增 29 个单元测试，全部打在本地 stub HTTP 服务（127.0.0.1）上——不访问外网、不需要密钥：
pytest tests/examples/deepeyes_v2_agentic/ # 41 passed (29 新增 + 12 既有) in ~7s覆盖范围：mock 确定性与返回结构；配置解析（默认值/覆盖/非法值）；
retriever 请求构造与响应映射（含
return_scores: false请求体断言、空结果成功、contents回退）；external 的 Serper 预设、自定义字段映射、认证头有无；失败矩阵——
超时、连接拒绝、4xx（不重试）、5xx（重试后成功 / 重试耗尽并核对尝试次数）、非法 JSON、字段缺失或类型不对；
经
DeepEyesV2Env.exec_tool的环境层集成（"Error"→search_failed观测路径）；以及一个守卫测试，断言两个启动脚本都把全部搜索环境变量透传进 Ray worker
（
runtime-env-json白名单在加变量时最容易漏）。真实服务验证：README 记录了一次性验证方法——先起
examples/search_r1/run_retriever.sh，把DEEPEYES_V2_RETRIEVER_URL指过去（或配置 Serper endpoint + 密钥），在仓库根目录调用一次
search()。retriever 线上协议与仓库自带的
examples/search_r1/retrieval_server.py一致；开发过程中未连接真实服务（本机没有索引和密钥）。
回归：
pytest tests/examples/—— deepeyes 相关测试全部通过；其余失败在干净的main上同样出现（用
git stash复跑确认过）：tests/examples/nemo_gym_agentic/test_r2e_test_layout.py（5 个，环境问题）和
tests/examples/seq_cls_sft/（需要 torch，本机未安装；本任务不涉及 GPU）。
pre-commit run --all-files --show-diff-on-failure通过（全部钩子）。git diff --check干净。pre-commit run --all-filespassesTests pass (
pytest tests/)New tests added (if applicable)
Documentation updated (if applicable)
Type of Change