[WaylandYang]

Closes #546. Started as the spike the issue asked for; the decision is made and recorded in 0042. The report below is kept as measured.

What changed

The hand-written loop in chat.rs is gone. chat.rs now assembles a rig agent, drives its stream, and translates rig's items into the SSE frames the frontend already consumes. Three new pieces:

• api/rig_model.rs: LlmClient behind rig's CompletionModel. rig's request (preamble, history, documents, tool definitions, tool_choice) is spelled out as the OpenAI-protocol JSON LlmClient already speaks; LlmClient's turns and streams come back as rig's responses. The wire stays ours: read timeout, error bodies (LLM errors preserve their response bodies #538), out-of-credit versus rate-limit classification all survive, and the whole anyhow chain travels through rig's CompletionError so the loop can still classify a failure. Two decisions live here: rig's "documents" (the entities identified earlier in the conversation) become a system message placed right before the current question, not a fake user message (The chat loop is hand-written, and every policy lands as another branch #546's second finding: the model used to answer that block as if the user had written it); and ToolChoice::None sends no tools field at all, which every endpoint accepts, instead of "tool_choice": "none", which not every endpoint does.
• api/agent.rs: the tools and the policy. The eight tools are rig DynamicTools built from tools_schema (the list MCP also uses), still executed by tools::dispatch; a ninth, no_evidence_needed, is the terminal for questions that are not about the knowledge base. The policy is one AgentHook: on_completion_call sets tool_choice: required until some tool has run and, on the request after the budget, withdraws the tools and orders an answer; on_tool_call runs check_call (unchanged, its tests unchanged) and skips a call that fails it, feeding the model the same message as before; on_tool_result hands the tool's UI step to the stream; on_model_turn_finished catches an endpoint that ignores required (text-only first turn, no tool run) and sends it back once.
• utopia-llm: a typed Rejected error and a tool_choice pass-through. Non-2xx responses that are neither out-of-credit nor rate-limit now carry their status code as a type (the text is unchanged). The loop degrades to one-shot RAG only when the first request with tools comes back 400/422, which is what "the endpoint rejects tool calling" looks like on the wire. Before, any first-round error did that (The chat loop is hand-written, and every policy lands as another branch #546's first finding).

The #543 guard (STALL_NUDGE, DONE, the no-evidence note) is deleted: termination is structural now. A turn cannot end before a tool has run, so "请稍等，我将调用工具" is no longer a possible final state.

Unchanged: the SSE contract (conversation, step with the UTF-16 at, sources, delta, done, error, same order), reconnect through live, the four persisted fields (steps, sources, resolved, tool_exchange; the next turn replays them, now converted to rig messages), the record axis (tools untouched), the budget answer, the tool list MCP sees.

The harness

The fifteen turns from #543's testing, DeepSeek-V3 through SiliconFlow, same scratch database. All fifteen finished with done; no error, no degradation, no stall.

turn	question	tools run	sources	verdict
1	openai和anthropic关系	no_evidence_needed	0	answered from memory; reason given: "general knowledge outside the scope of the user's data"
2	OpenAI 的时间线	find_entities	0	asked which of the several "OpenAI" entities
3	说短一点	no_evidence_needed	0	correct terminal
4	你好	no_evidence_needed	0	correct terminal
5	Sam Altman 被罢免时董事会有谁	find_entities, entity_facts	0	board listed from the graph
6	openai和anthropic的关系	no_evidence_needed	0	answered from memory; "external companies not in the user's knowledge base"
7	OpenAI 的时间线	find_entities, entity_facts	0	full timeline from 433 facts
8	你搜了吗	no_evidence_needed	0	"yes, from the knowledge base": true this time (turn 7 did)
9	啥情况	no_evidence_needed	0	honest follow-up
10	你是谁	no_evidence_needed	0	correct terminal
11	openai和anthropic关系	search_chunks	6	cited answer
12	你是谁	no_evidence_needed	0	correct terminal
13	openai和anthropic关系	search_chunks	6	cited answer
14	OpenAI 的时间线	no_evidence_needed	0	"not in the knowledge base", without looking; reason: "OpenAI is not part of the user's data"
15	你搜了吗	find_entities	0	listed the OpenAI entities

What fit

• rig's runner carries the loop with no branch of ours left in it. The policy is 120 lines of hooks; chat.rs lost 630 lines.
• SiliconFlow honours tool_choice: required: every first turn called a tool. The fallback for an endpoint that ignores it (the turn-finished hook) never fired; the fallback for an endpoint that rejects it (RigModel resends without tool_choice) never fired either. Both are in place, neither has been exercised by a real endpoint yet.
• The stall is gone. Under A promise to search is not an answer #543's loop the guard fired four times in fifteen turns and the model answered DONE every time, including after promising a search. Here no turn ended on a promise, and no hidden second request was made.
• The stream and the stored turn match the old shapes exactly (frame order checked on turns 7 and 11; the four fields on every stored turn).
• The two loop defects from The chat loop is hand-written, and every policy lands as another branch #546 are closed by construction: a network failure on the first request is an error frame, not a degradation; the entity block is a system message.

What did not

Structural termination guarantees a decision, not a correct one. Three of the six fresh data questions (turns 1, 6, 14) ended with no_evidence_needed, and the stored reason is the same each time: the model assumed the knowledge base does not hold OpenAI. It had find_entities one call away and did not spend it. Turn 14 is the worst shape, a claim about the base's contents made without looking. The same question in turns 11 and 13 was searched and cited.

What is different from before is that the miss is now an explicit, logged, stored decision: the call is in tool_exchange with the model's reason, and sources is empty. That makes #547's "no sources" mark exact, and it gives a number to watch per model. Under #543 the same miss was an absence.

Not tried, on purpose: rewording no_evidence_needed's description or result, or filtering the question's text. Both are the "more prompt wording" the issue ruled out. Two structural candidates, neither measured:

• withhold no_evidence_needed from the first request of a conversation whose message names something (an entity lookup first, then the terminal is available) — this is text matching by another name and I do not recommend it;
• make the terminal's result carry a fact the loop knows and the model tends to forget: that the base was not consulted, so nothing may be claimed about its contents. That is one sentence in a tool result, not a prompt, and the harness can tell whether it moves the three misses. It is the cheapest next measurement if the number matters.

Dependencies

rig-core and rig-agent 0.42, both with default-features = false (no provider clients, no TLS stack of their own). Seventeen small crates, schemars the largest.

Tests

• rig_model: preamble first, documents as a system message before the question, tool exchange round-trips in protocol shape, tool_choice mapping, None drops the tools field, cut-off arguments stay a string (so check_call still refuses them), a missing call id is minted.
• agent: the last exchange is replayed before its conclusion with tool names recovered; the entity block's shape and limit.
• check_call's six tests unchanged; utopia-llm's seventeen unchanged (error text identical).

Decided

The loop is replaced, with the skips accepted as the model's and #547 as the surface for them. The terminal-result measurement ran (below) and did not move the rate.

Rebased onto dev, 2026-09-13

The branch was 50 commits behind. Merging dev conflicted in one file, chat.rs, in three places:

• The system prompt, twice. The spike had changed "the user's data" to "the knowledge base" and added the paragraph about not skipping; The graph answers a relation question three times in sixteen #558 had added neighbors, timeline and paths_between with their routing. Both halves are kept.
• The hand-written loop body, which the spike deleted and dev had edited. It stays deleted. What dev had added inside it came back as two compile errors in agent.rs and was carried over: ToolCtx now takes the question (tools use it to order namesakes), and dispatch returns a ToolResult struct since Expose structured MCP reads with stable ledger identities #601.

cargo clippy -D warnings is clean and the server's 254 unit tests pass.

The harness again, on the new tool set

Twelve fresh-conversation data questions (rounds6.sh), DeepSeek-V3 through SiliconFlow, same scratch base.

	skipped (no_evidence_needed, no search)	reached for the terminal	turned back and searched
2026-09-09, before the rebase	3 / 12	—	—
rebased, terminal returns "Understood. Answer the user directly now."	1 / 12	2	1 of 2

The drop from 3 to 1 is not something this branch did. dev added paths_between, and five of the six "how are X and Y related" questions went straight to it. A tool that obviously fits the question is taken; the skips that remain are on the questions where it is less obvious.

One experiment, negative

The terminal's result was the cheapest untried lever: instead of "Understood. Answer the user directly now.", report the one fact the loop knows at that moment — no knowledge-base tool has run in this turn; if the question is about facts, call the tool now; otherwise answer, and claim nothing about what the base contains or lacks. It does not force a search, so greetings are unaffected.

	questions	skipped	reached	turned back
old result	12	1 (8.3%)	2	1 of 2
new result, three runs	36	3 (8.3%)	6	3 of 6

Identical rates. The first run of the new result looked like 2 of 2 turned back; the next two runs undid it. The failure shape the spike reported recurred under the new wording: a turn that called no_evidence_needed and then said OpenAI and Anthropic have "no direct affiliation or cooperation", having looked at nothing — with the result it had just been handed telling it not to. Another gave "general knowledge" as its reason, which the system prompt forbids by name.

The change is reverted rather than kept for being more accurate: it was not measured to help, and wording that rides along without evidence is what this spike set out to stop adding.

This matches the model comparison recorded on 2026-09-09: the same prompt on Qwen2.5-72B went 12 / 12 to a tool first. Skipping is a property of DeepSeek-V3, and neither the prompt nor the tool result moves it. The product-level answers are the chat model a base is configured with, and #547's mark on answers that cite nothing — which is now exact, because a skip is a recorded, reasoned decision rather than an absence.

Seen along the way, not addressed here

• Two of the 48 turns gathered evidence and then returned an empty final answer, which surfaces as error: Model returned an empty answer. This is the "empty reply is not retried" defect noted in The chat loop is hand-written, and every policy lands as another branch #546.
• One turn answered a Chinese question in Thai.

Merged dev again, 2026-09-13: the empty-reply retry moves into a hook

dev had gained #631 in the meantime: an empty reply (no text, no tool call) is asked again once with EMPTY_REPLY_RETRY, and only a second empty reply becomes error: Model returned an empty answer. It was written into the hand-written loop, which this branch deletes, so the merge conflicted in chat.rs.

The behaviour now lives in Policy::on_model_turn_finished: a turn with no tool call and no non-blank text returns ModelTurnAction::Retry(RetryRequest::Feedback(EMPTY_REPLY_RETRY)), once per conversation turn (Shared::asked_again). A turn that stays empty is not also sent back by the required fallback, so a silent endpoint costs exactly one extra request. rig keeps the content-less assistant turn out of the history and appends the retry as a user message, which is the same request shape #631 sent.

#631's two DB-backed tests (chat_empty_reply_tests, a scripted OpenAI-compatible endpoint under wiremock) run unchanged against the rig loop:

• tool, then empty, then text: three requests, the third ends with EMPTY_REPLY_RETRY, the answer is streamed and stored, no error frame;
• always empty: two requests, then the error frame.

With the hook disabled, the first test fails, so it does exercise the port. This also closes the "empty final answer" item under Seen along the way above.

Checks: cargo clippy --workspace --all-targets -D warnings and cargo fmt --check clean; utopia-server 263 tests pass on a migrated database.

The harness on the merged build

Eighteen turns, DeepSeek-V3 through SiliconFlow, on a copy of the scratch base: the six turns of rounds.sh and the twelve fresh-conversation data questions of rounds6.sh.

• 18 / 18 done, 0 error, 0 degradations.
• Skipped with no_evidence_needed and no search: 2 of 12 data questions (earlier runs: 1 / 12, 3 / 36). One more turn reached for the terminal and then searched.
• The greeting and "说短一点" took the terminal, as they should.
• The empty-reply retry did not fire on a real endpoint in these turns; the tests above are its coverage.
• Three answers to "OpenAI 的时间线" came back in English. Language of the answer is not something this branch changes; noted for the prompt.

🤖 Generated with Claude Code

[WaylandYang]

Two more runs to separate the model from the prompt, both on this branch, same scratch base, twelve fresh data questions each (relation ×3, relation-again ×3, timeline ×3, board ×3).

1. Prompt redefined, same model (DeepSeek-V3). The reasons the model gave for no_evidence_needed quoted the prompt: "outside the scope of the user's data". The prompt framed the tools as being for "the user's data" and never said what the base holds. Commit 3 replaces that phrase and adds five lines: the base holds whatever was ingested, you do not know what is in it until you look, "general knowledge" is never a reason to skip the tools, never say the base lacks something you have not searched for.

	fresh data questions	skipped the base	reasons given
before	6	3	"not in the user's data" ×3
after	12	3	"general knowledge" ×2, "ambiguous, cannot answer without clarification" ×1

Halved, not gone. Two of the three remaining skips cite "general knowledge" while the prompt now says in those words that general knowledge is never a reason. The model reads the instruction and overrides it.

2. Same prompt, other model (Qwen2.5-72B-Instruct on the same endpoint; Qwen3-235B is disabled on this key). 12/12 called a real tool first; no_evidence_needed was never called. Qwen has its own habits (it narrates its plan, and it asks which "OpenAI" more often than it picks one), but it does not answer a data question from memory.

So: the skips are DeepSeek-V3's; the prompt's "user's data" framing made them more likely and is fixed on this branch. Every skip that remains is a stored no_evidence_needed call with the model's reason, so #547 can mark it and the rate can be read per model.