Skip to content

Verify Claude Haiku 5.5 tool_reference / tool search against the live API, then enable #2914

Description

@code-yeongyu

Summary

Claude Haiku 5.5 (claude-haiku-5-5, #2892 / #2911) ships with client-side tool_reference loading and Anthropic native tool search turned off, because its support has not been checked against the live API. This issue tracks verifying both, then enabling them. It also tracks a separate decision: whether the hosted web-search default should move from Claude Haiku 4.5 to 5.5.

Why off, when the docs list it

  • Anthropic's tool-search compatibility table lists Claude Haiku 5.5 for both tool_search_tool_regex_20251119 and tool_search_tool_bm25_20251119: https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool#model-compatibility
  • The same table lists Claude Haiku 4.5 with an identical row. Yet upstream #6474 (3d8f74357) found Haiku 4.5 rejects client-side tool_reference blocks, which is why every Haiku has been excluded since. So the table alone was already wrong once for Haiku.
  • The two switches fail differently:
    • Native tool search (packages/coding-agent/src/core/extensions/builtin/tool-search/native-support.ts, defaultSupportsToolSearch): a 400 disables injection for the session and falls back to local search. Cost: one failed request.
    • supportsToolReferences (packages/ai/src/utils/prompt-cache-ttl.ts, defaultSupportsToolReferences): drives message-anchored deferred loading (defer_loading plus a tool_reference in a tool_result). Its only automatic recovery matches Tool reference '…' not found in available tools (anthropic-messages.ts). Any other rejection message fails the turn.
  • The PR that adds Haiku 5.5 had no sanctioned Anthropic credential, so no probe could be run.

Acceptance criteria

  1. One live probe on claude-haiku-5-5 for each mechanism, recording only the HTTP status and the error type/message, never headers or keys:
    • a request with a client tool_result that carries a tool_reference to a defer_loading tool;
    • a request that includes tool_search_tool_bm25_20251119 (or regex) with deferred tools, and lets the model search.
  2. If both succeed: enable Haiku 5.5+ in both gates. The version-aware check is already written and was reverted in feat(ai): add Claude Haiku 5.5 to the catalog #2911: a Haiku floor of 5.5 in defaultSupportsToolReferences and defaultSupportsToolSearch. Add the cases back to 0000-anthropic-native-tool-search-contract.test.ts and anthropic-haiku-5-5.test.ts, and update packages/ai/src/types.ts and packages/coding-agent/docs/extensions.md.
  3. If either fails: record the status and message here, keep that gate off, and (for tool_reference) decide whether the recovery matcher should also cover the new message.
  4. Optional, same probe session: whether thinking: {type: "disabled"} is a 400 on Haiku 5.5. senpi currently hides off for it, unverified (generate-models.ts adaptive-only comment).

Web-search default model (websearch/search-model.ts)

DEFAULT_SEARCH_MODELS.anthropic stays claude-haiku-4-5. The hosted-search request (websearch/providers/anthropic.ts) sets no thinking, and Haiku 5.5 thinks adaptively by default at effort medium. Moving the default would add thinking latency and tokens to a latency-bound call, even though Haiku 5.5 is 10x cheaper per token than 4.5 (0.1 / 0.5 vs 1 / 5 per MTok).

To decide:

  • Measure p50/p95 latency and cost of one hosted search on Haiku 4.5 vs 5.5. For 5.5, test both as-is and with an explicit low effort, if the search request gains an effort control.
  • Plan for Haiku 4.5's eventual deprecation.

Related

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions