Skip to content

A tool call the model writes as text is shown and saved as the answer #937

Closed

Description

@Maya-Kid

Sometimes the model writes its tool call as text instead of making a structured call. When it does, that text reaches the reader and is stored as the answer. DeepSeek does this with its native markup (<|tool▁calls▁begin|>…). The bench README counts it as 2 of the 35 chat misses; that run was on 2026-09-05, before the loop moved to rig.

What happens on dev (b3919ca)

This was reproduced with a throwaway test against a scripted OpenAI-compatible endpoint that replies with only that markup, as text.

  • After a tool has run, the markup is streamed to the reader and stored as the assistant message, and the turn ends with done. There is no retry and no error; the whole turn takes two model requests.
  • On the first turn, the required gate works: the turn is sent back once (MUST_CALL), and the model then calls the tool and answers. But the markup has already been streamed, and it stays in the stored answer ahead of the real one.

Why

  • A gathering turn's narration is streamed as it arrives (api/chat.rs). Text that turns out to be a tool call has therefore already reached the reader.
  • on_model_turn_finished sends back only an empty turn or a text-only first turn. Once a tool has run, any text-only turn is accepted as the answer.
  • finalization_error does recognize control output, but it runs only on the budget answer (Hand off exhausted tool runs to an evidence-only final answer #845). It also knows only the DSML spellings, not <|tool▁calls▁begin|> or <|tool▁call▁begin|>.
  • When a turn is sent back, the route clears that turn's buffer, but text that was already streamed stays in answer_acc and is saved.

Proposal

This follows 0042 (policy lives in a hook) and #845 (a publication guard, never a parser or executor):

  1. Recognize the leaked-call spellings in one place: the DSML spellings finalization_error already knows plus DeepSeek's native markers. Match them at a line start outside Markdown fences, keeping the existing exceptions for explanations and quoted examples.
  2. In the route, hold a turn's text back while it could still be such markup (the text so far is a prefix of a marker), and publish it as soon as it clearly is not. Markup is then never streamed or added to the stored answer.
  3. In on_model_turn_finished, send back once any turn that has no structured call and whose text is markup. The instruction is short: call the tool through the interface, or answer in prose.
  4. If it happens again, end with the error the budget guard already uses ("Model returned tool-control text instead of a final answer") instead of saving the markup.

Questions

  • After a second occurrence, should the turn end in that error, or hand over to the evidence-only answer call?
  • Which marker families should count: DSML and DeepSeek's native markers only, or also <tool_call>, which Qwen/Hermes chat templates produce when served without a tool parser?

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions