Tool Search Tool (BM25) does not find tools with Japanese descriptions

[DaisukeYoda]

Description

When using the Tool Search Tool (tool_search_tool_bm25_20251119) with tools that have Japanese descriptions, the BM25 search fails to find relevant tools even with exact keyword matches.

Environment

• Beta: advanced-tool-use-2025-11-20
• Model: claude-sonnet-4-5-20250929
• SDK: @anthropic-ai/sdk (TypeScript)

Steps to Reproduce

1. Define tools with defer_loading: true and Japanese descriptions:

const searchLawsTool = {
  name: 'search_laws',
  description: '日本の法令を検索します。憲法、法律、政令、省令などを検索できます。',
  input_schema: { /* ... */ },
  defer_loading: true,
};

2. Send a message containing the exact keyword from the description:

User: "法令を調べて" (Search for laws)

3. The Tool Search Tool fails to find search_laws, even though "法令" appears in the description.

Expected Behavior

The BM25 search should find search_laws since the query "法令" exactly matches the description "日本の法令を検索します".

Actual Behavior

The tool is not found. Claude responds that no relevant tools were found.

Workaround

Setting defer_loading: false for all tools makes them visible without search, but this defeats the purpose of the Tool Search Tool.

Hypothesis

BM25 may not properly tokenize Japanese text (no word boundaries like spaces in English), causing the search to fail.

Additional Context

• Tools with defer_loading: false (always visible) work correctly
• The same tools work fine when all set to defer_loading: false
• This appears to be a CJK (Chinese/Japanese/Korean) language tokenization issue

[Ashutosh0x]

After a comprehensive analysis of the SDK source code, I've confirmed that this library does not contain any client-side implementation of the BM25 algorithm or Tool Search logic. The files present (e.g., ypes/beta/beta_tool_search_tool_bm25_20251119_param.py) are strictly TypeDefinitions for the API interface.

The tokenization issue for CJK languages described here appears to be a server-side behavior within the Anthropic API's backend implementation of the Tool Search beta. It cannot be fixed within this Python SDK.

I recommend routing this feedback to the backend engineering team responsible for the Tool Search beta.

[Ashutosh0x]

Closing as this is a server-side issue, not an SDK bug.