Repository navigation
A tool call the model writes as text is shown and saved as the answer #937
Copy link
Copy link
Closed
Description
Sometimes the model writes its tool call as text instead of making a structured call. When it does, that text reaches the reader and is stored as the answer. DeepSeek does this with its native markup (<|tool▁calls▁begin|>…). The bench README counts it as 2 of the 35 chat misses; that run was on 2026-09-05, before the loop moved to rig.
What happens on dev (b3919ca)
This was reproduced with a throwaway test against a scripted OpenAI-compatible endpoint that replies with only that markup, as text.
- After a tool has run, the markup is streamed to the reader and stored as the assistant message, and the turn ends with
done. There is no retry and no error; the whole turn takes two model requests. - On the first turn, the
requiredgate works: the turn is sent back once (MUST_CALL), and the model then calls the tool and answers. But the markup has already been streamed, and it stays in the stored answer ahead of the real one.
Why
- A gathering turn's narration is streamed as it arrives (
api/chat.rs). Text that turns out to be a tool call has therefore already reached the reader. on_model_turn_finishedsends back only an empty turn or a text-only first turn. Once a tool has run, any text-only turn is accepted as the answer.finalization_errordoes recognize control output, but it runs only on the budget answer (Hand off exhausted tool runs to an evidence-only final answer #845). It also knows only the DSML spellings, not<|tool▁calls▁begin|>or<|tool▁call▁begin|>.- When a turn is sent back, the route clears that turn's buffer, but text that was already streamed stays in
answer_accand is saved.
Proposal
This follows 0042 (policy lives in a hook) and #845 (a publication guard, never a parser or executor):
- Recognize the leaked-call spellings in one place: the DSML spellings
finalization_erroralready knows plus DeepSeek's native markers. Match them at a line start outside Markdown fences, keeping the existing exceptions for explanations and quoted examples. - In the route, hold a turn's text back while it could still be such markup (the text so far is a prefix of a marker), and publish it as soon as it clearly is not. Markup is then never streamed or added to the stored answer.
- In
on_model_turn_finished, send back once any turn that has no structured call and whose text is markup. The instruction is short: call the tool through the interface, or answer in prose. - If it happens again, end with the error the budget guard already uses ("Model returned tool-control text instead of a final answer") instead of saving the markup.
Questions
- After a second occurrence, should the turn end in that error, or hand over to the evidence-only answer call?
- Which marker families should count: DSML and DeepSeek's native markers only, or also
<tool_call>, which Qwen/Hermes chat templates produce when served without a tool parser?
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Activity