Ollama Qwen 3B and Mistral Small 3.1 cannot handle Toolbox tools

[simonw]

This is odd:

llm chat -T QuickJS -m qwen3:4b --td                              
Chatting with qwen3:4b
Type 'exit' or 'quit' to exit
Type '!multi' to enter multiple lines, then '!end' to finish
Type '!edit' to open your default editor and modify the prompt
Type '!fragment <my_fragment> [<another_fragment> ...]' to insert one or more fragments
> Calcuate 2343242 * 23424323 with quickjs
ToolResult(name='execute_javascript', output='Error: tool "execute_javascript" does not exist', tool_call_id=None, instance=None)
ToolResult(name='execute_javascript', output='Error: tool "execute_javascript" does not exist', tool_call_id=None, instance=None)
ToolResult(name='execute_javascript', output='Error: tool "execute_javascript" does not exist', tool_call_id=None, instance=None)
^C
Aborted!
(llm) (base) simon@Mac llm % llm chat -T QuickJS -m qwen3:4b --td
Chatting with qwen3:4b
Type 'exit' or 'quit' to exit
Type '!multi' to enter multiple lines, then '!end' to finish
Type '!edit' to open your default editor and modify the prompt
Type '!fragment <my_fragment> [<another_fragment> ...]' to insert one or more fragments
> Use QuickQS_execute_javascript to run 23423432 * 23423423
ToolResult(name='execute_javascript', output='Error: tool "execute_javascript" does not exist', tool_call_id=None, instance=None)
ToolResult(name='execute_javascript', output='Error: tool "execute_javascript" does not exist', tool_call_id=None, instance=None)

Makes me suspect that Qwen 3 4B via Ollama cannot handle tools like QuickQS_execute_javascript - it treats them as execute_javascript instead.

If this is just Qwen 3 4B that's not too worrying - if it's all the Qwen models AND any other Ollama model it's a big problem.

[simonw]

Rats, it affects Mistral Small 3.1 on Ollama too:

% llm chat -T QuickJS -m mistral-small3.1:latest --td               
Chatting with mistral-small3.1:latest
Type 'exit' or 'quit' to exit
Type '!multi' to enter multiple lines, then '!end' to finish
Type '!edit' to open your default editor and modify the prompt
Type '!fragment <my_fragment> [<another_fragment> ...]' to insert one or more fragments
> Calcualte 23324233 * 234234234 with quickjs
ToolResult(name='execute_javascript', output='Error: tool "execute_javascript" does not exist', tool_call_id=None, instance=None)
ToolResult(name='execute_javascript', output='Error: tool "execute_javascript" does not exist', tool_call_id=None, instance=None)
I'm sorry, but I currently don't have the tools to execute JavaScript code directly. However, I can guide you on how to calculate `23324233 * 234234234` using JavaScript in your own environment.

Here's a simple script you can run in your browser's console or any JavaScript environment:

```javascript
let result = 23324233 * 234234234;
console.log(result);
```

This script will calculate the product of `23324233` and `234234234` and log the result to the console.

[simonw]

Weirdly, even converting the tool names to lower case did not fix this. Log here: https://gist.github.com/simonw/a5ecc61d4c72c3bc6c1b2e48709bbc93

Mistral was handed quickjs_execute_javascript but still tried to call execute_javascript.

[simonw]

For the moment I'm going to add the function based quickjs tool BACK to llm-tools-quickjs and mention in the announcement that there's a bug.

[simonw]

Yeah, function-based tools are fine:

llm -T quickjs '324324 * 23432423' --td -m qwen3:4b

Tool call: quickjs({'javascript': 'function execute() {\nreturn 324324 * 23432423;\n}\n'})
  7599697157052.0

<think>
Okay, the user asked to multiply 324324 by 23432423. I need to use the quickjs function to execute the JavaScript code. Let me make sure I format the function call correctly. The function requires a JavaScript code snippet with an execute() function that returns the result. I'll write the multiplication inside that function. Then, I'll wrap it in the tool_call tags as specified. Let me double-check the parameters to ensure the JavaScript string is properly escaped. Once that's done, the response should come back with the correct value.
</think>

The result of multiplying 324324 by 23432423 is 7599697157052.0.

[simonw]

Here's the change I made to try lowercase, which did not work:

diff --git a/llm/models.py b/llm/models.py
index 7ee7cf6..758235a 100644
--- a/llm/models.py
+++ b/llm/models.py
@@ -222,7 +222,7 @@ class Toolbox:
             if callable(method) and inspect.ismethod(method):
                 tool = Tool.function(
                     method,
-                    name="{}_{}".format(self.__class__.__name__, method_name),
+                    name="{}_{}".format(self.__class__.__name__.lower(), method_name),
                 )
                 tool.plugin = getattr(self, "plugin", None)
                 yield tool
@@ -930,13 +930,13 @@ class Response(_BaseResponse):
         for tool_call in self.tool_calls():
             tool = tools_by_name.get(tool_call.name)
             if tool is None:
-                tool_results.append(
-                    ToolResult(
-                        name=tool_call.name,
-                        output='Error: tool "{}" does not exist'.format(tool_call.name),
-                        tool_call_id=tool_call.tool_call_id,
-                    )
+                result = ToolResult(
+                    name=tool_call.name,
+                    output='Error: tool "{}" does not exist'.format(tool_call.name),
+                    tool_call_id=tool_call.tool_call_id,
                 )
+                tool_results.append(result)
+                print(result)
                 continue
 
             if before_call:

[ibeex]

hi, I get

❯ llm --tool llm_version "What version?" --td -m qwen3:30b
Error: Ollama: qwen3:30b does not support tools

[mdrio]

Hi, me too, ollama version is 0.7.1.

hi, I get

❯ llm --tool llm_version "What version?" --td -m qwen3:30b
Error: Ollama: qwen3:30b does not support tools

[boegel]

Likewise, I was playing with the ollama example from https://simonwillison.net/2025/May/27/llm-tools/:

$ pip list | grep ^llm
llm                   0.26
llm-ollama            0.10.0

$ ollama --version
ollama version is 0.7.1

$ ollama list
NAME                                                      ID              SIZE      MODIFIED
qwen3:4b                                                  a383baf4993b    2.6 GB    34 minutes ago

$ llm --tool llm_version "What version?" --td -m qwen3:4b
Error: Ollama: qwen3:4b does not support tools

edit: similar issue via gpt4all:

$ pip list | grep ^llm
llm                   0.26
llm-gpt4all           0.4
llm-ollama            0.10.0

$ llm models | grep 'gpt4all.*Meta-Llama.*installed'
gpt4all: Meta-Llama-3-8B-Instruct - Llama 3 8B Instruct, 4.34GB download, needs 8GB RAM (installed)

$ llm --tool llm_version "What version?" --td -m Meta-Llama-3-8B-Instruct
Error: gpt4all: Meta-Llama-3-8B-Instruct - Llama 3 8B Instruct, 4.34GB download, needs 8GB RAM (installed) does not support tools