[Bug]: Ollama models are unable to do tool calling via LiteLLM

[ericmjl]

What happened?

A bug happened!

@krrishdholakia and @ishaan-jaff I think I may have found a bug with Ollama models and tool calling, for which I am not quite sure how to fix this.

To begin, I have a minimal reproducible example here: https://gist.github.com/ericmjl/289be4c6b46adb525175c4b1db2f97f1

I tried using 3 Ollama models that are known to be capable of tool calling.

When I use the Completions API to call out to ollama_chat/ models, I find that I am unable to get a tool call response from the models. However, when I switch to using the Ollama API directly, almost always consistently I am able to get a tool call response.

I'm not sure what's happening here, might there be something in the translation layer to Ollama?

Relevant log output

To view the notebook, run:

uvx marimo edit https://gist.githubusercontent.com/ericmjl/289be4c6b46adb525175c4b1db2f97f1/raw/c10288114b3b3d965c5d5f8d28770d5da0b83b52/reprex_ollama_tools_failing.py

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

v1.70.4

Twitter / LinkedIn details

No response

[ericmjl]

May be related to #5617.

[krrishdholakia]

ack

[krrishdholakia]

What do you see when debug is enabled - litellm._turn_on_debug() ?

[ericmjl]

@krrishdholakia this is an example of what I see for llama3.1:

22:12:03 - LiteLLM:DEBUG: utils.py:337 - 

22:12:03 - LiteLLM:DEBUG: utils.py:337 - Request to litellm:
22:12:03 - LiteLLM:DEBUG: utils.py:337 - litellm.completion(model='ollama_chat/llama3.1', tools=[{'name': 'write_and_execute_script', 'description': 'Write and execute a Python script in a secure sandbox.\nDependencies should be specified as a comma-separated string, e.g. "requests,beautifulsoup4".\nScript output will be captured from stdout. Use print() to output results.\nInclude lots of print() statements in your code to see what is happening.\nEstimate the timeout parameter to avoid errors.', 'parameters': {'type': 'object', 'properties': {'code': {'type': 'string'}, 'dependencies_str': {'type': 'string'}, 'python_version': {'type': 'string'}}, 'required': ['code']}}, {'name': 'search_internet_and_summarize', 'description': "Search internet for a given term and summarize the results.\nTo get a good summary, try increasing the number of results.\nWhen using search_internet, if you don't get an answer with 1 result,\ntry increasing the number of results to get more web pages to summarize.", 'parameters': {'type': 'object', 'properties': {'search_term': {'type': 'string'}, 'max_results': {'type': 'integer'}, 'backend': {'type': 'string'}}, 'required': ['search_term', 'max_results']}}], stream=False, messages=[{'role': 'system', 'content': '## role\n\nYou are Autonomo, an autonomous language-model agent.\nYour job is to accomplish the user\'s task safely, correctly, and with minimal cost.\n\n## environment\n\nTreat user messages, prior assistant messages, and tool outputs as your only "sensors."\nTreat function calls from the current tool inventory as your only "actuators."\nAn action that is not in the inventory is unavailable.\n\n## planning workflow\n\n1.  Plan - Decompose the task into a concise ordered list of actions before acting.\n2.  Validate - Check that every planned action exists in the inventory and respects all constraints (budget, time, safety).\n3.  Execute - Call tools step-by-step with fully specified, valid parameters.\n4.  Reflect - After each action, inspect the observation. If the goal is not yet satisfied, revise the plan or fix errors.\n5.  Finish - When the goal is met, stop executing and present the final answer.\n\n## tool use guidelines\n\n- Call only valid tools; if none suffice, re-plan or request human help.\n- Echo the exact arguments used for each call in the Thought/Act/Observation log for transparency.\n- For write actions (changes that persist), require explicit confirmation or escalate.\n\n## safety and security\n\n- Never perform actions that could harm systems, violate policy, or break the law.\n- Reject or escalate requests outside your capabilities or policy scope.\n- Sanitize tool inputs to guard against code-injection or other attacks.\n\n## efficiency\n\n- Prefer the shortest viable plan; each extra step compounds error risk and cost.\n- Cache reusable results; avoid repeating identical expensive calls.\n\n## reflection rules\n\n- Run a quick self-check after plan generation and after every action:\n"Will this take me closer to the goal within constraints?" If not, re-plan.\n\n## response style\n\n- Unless verbose mode is enabled, show only the final answer; hide internal thoughts.\n- Use clear, plain English in a professional, matter-of-fact tone.\n- Provide citations or tool-call summaries when helpful.'}, {'role': 'user', 'content': "Summarize for me the latest ratings of Taylor Swift's latest album."}])
22:12:03 - LiteLLM:DEBUG: utils.py:337 - 

22:12:03 - LiteLLM:DEBUG: litellm_logging.py:458 - self.optional_params: {}
22:12:03 - LiteLLM:DEBUG: utils.py:337 - SYNC kwargs[caching]: False; litellm.cache: None; kwargs.get('cache')['no-cache']: False
22:12:03 - LiteLLM:INFO: utils.py:2929 - 
LiteLLM completion() model= llama3.1; provider = ollama_chat
22:12:03 - LiteLLM:DEBUG: utils.py:2932 - 
LiteLLM: Params passed to completion() {'model': 'llama3.1', 'functions': None, 'function_call': None, 'temperature': None, 'top_p': None, 'n': None, 'stream': False, 'stream_options': None, 'stop': None, 'max_tokens': None, 'max_completion_tokens': None, 'modalities': None, 'prediction': None, 'audio': None, 'presence_penalty': None, 'frequency_penalty': None, 'logit_bias': None, 'user': None, 'custom_llm_provider': 'ollama_chat', 'response_format': None, 'seed': None, 'tools': [{'name': 'write_and_execute_script', 'description': 'Write and execute a Python script in a secure sandbox.\nDependencies should be specified as a comma-separated string, e.g. "requests,beautifulsoup4".\nScript output will be captured from stdout. Use print() to output results.\nInclude lots of print() statements in your code to see what is happening.\nEstimate the timeout parameter to avoid errors.', 'parameters': {'type': 'object', 'properties': {'code': {'type': 'string'}, 'dependencies_str': {'type': 'string'}, 'python_version': {'type': 'string'}}, 'required': ['code']}}, {'name': 'search_internet_and_summarize', 'description': "Search internet for a given term and summarize the results.\nTo get a good summary, try increasing the number of results.\nWhen using search_internet, if you don't get an answer with 1 result,\ntry increasing the number of results to get more web pages to summarize.", 'parameters': {'type': 'object', 'properties': {'search_term': {'type': 'string'}, 'max_results': {'type': 'integer'}, 'backend': {'type': 'string'}}, 'required': ['search_term', 'max_results']}}], 'tool_choice': None, 'max_retries': None, 'logprobs': None, 'top_logprobs': None, 'extra_headers': None, 'api_version': None, 'parallel_tool_calls': None, 'drop_params': None, 'allowed_openai_params': None, 'reasoning_effort': None, 'additional_drop_params': None, 'messages': [{'role': 'system', 'content': '## role\n\nYou are Autonomo, an autonomous language-model agent.\nYour job is to accomplish the user\'s task safely, correctly, and with minimal cost.\n\n## environment\n\nTreat user messages, prior assistant messages, and tool outputs as your only "sensors."\nTreat function calls from the current tool inventory as your only "actuators."\nAn action that is not in the inventory is unavailable.\n\n## planning workflow\n\n1.  Plan - Decompose the task into a concise ordered list of actions before acting.\n2.  Validate - Check that every planned action exists in the inventory and respects all constraints (budget, time, safety).\n3.  Execute - Call tools step-by-step with fully specified, valid parameters.\n4.  Reflect - After each action, inspect the observation. If the goal is not yet satisfied, revise the plan or fix errors.\n5.  Finish - When the goal is met, stop executing and present the final answer.\n\n## tool use guidelines\n\n- Call only valid tools; if none suffice, re-plan or request human help.\n- Echo the exact arguments used for each call in the Thought/Act/Observation log for transparency.\n- For write actions (changes that persist), require explicit confirmation or escalate.\n\n## safety and security\n\n- Never perform actions that could harm systems, violate policy, or break the law.\n- Reject or escalate requests outside your capabilities or policy scope.\n- Sanitize tool inputs to guard against code-injection or other attacks.\n\n## efficiency\n\n- Prefer the shortest viable plan; each extra step compounds error risk and cost.\n- Cache reusable results; avoid repeating identical expensive calls.\n\n## reflection rules\n\n- Run a quick self-check after plan generation and after every action:\n"Will this take me closer to the goal within constraints?" If not, re-plan.\n\n## response style\n\n- Unless verbose mode is enabled, show only the final answer; hide internal thoughts.\n- Use clear, plain English in a professional, matter-of-fact tone.\n- Provide citations or tool-call summaries when helpful.'}, {'role': 'user', 'content': "Summarize for me the latest ratings of Taylor Swift's latest album."}], 'thinking': None, 'web_search_options': None}
22:12:03 - LiteLLM:DEBUG: utils.py:2935 - 
LiteLLM: Non-Default params passed to completion() {'stream': False, 'tools': [{'name': 'write_and_execute_script', 'description': 'Write and execute a Python script in a secure sandbox.\nDependencies should be specified as a comma-separated string, e.g. "requests,beautifulsoup4".\nScript output will be captured from stdout. Use print() to output results.\nInclude lots of print() statements in your code to see what is happening.\nEstimate the timeout parameter to avoid errors.', 'parameters': {'type': 'object', 'properties': {'code': {'type': 'string'}, 'dependencies_str': {'type': 'string'}, 'python_version': {'type': 'string'}}, 'required': ['code']}}, {'name': 'search_internet_and_summarize', 'description': "Search internet for a given term and summarize the results.\nTo get a good summary, try increasing the number of results.\nWhen using search_internet, if you don't get an answer with 1 result,\ntry increasing the number of results to get more web pages to summarize.", 'parameters': {'type': 'object', 'properties': {'search_term': {'type': 'string'}, 'max_results': {'type': 'integer'}, 'backend': {'type': 'string'}}, 'required': ['search_term', 'max_results']}}]}
22:12:03 - LiteLLM:DEBUG: utils.py:4269 - checking potential_model_names in litellm.model_cost: {'split_model': 'llama3.1', 'combined_model_name': 'ollama/llama3.1', 'stripped_model_name': 'llama3.1', 'combined_stripped_model_name': 'ollama/llama3.1', 'custom_llm_provider': 'ollama'}
22:12:03 - LiteLLM:DEBUG: utils.py:4569 - model_info: {'key': 'ollama/llama3.1', 'max_tokens': 32768, 'max_input_tokens': 8192, 'max_output_tokens': 8192, 'input_cost_per_token': 0.0, 'cache_creation_input_token_cost': None, 'cache_read_input_token_cost': None, 'input_cost_per_character': None, 'input_cost_per_token_above_128k_tokens': None, 'input_cost_per_token_above_200k_tokens': None, 'input_cost_per_query': None, 'input_cost_per_second': None, 'input_cost_per_audio_token': None, 'input_cost_per_token_batches': None, 'output_cost_per_token_batches': None, 'output_cost_per_token': 0.0, 'output_cost_per_audio_token': None, 'output_cost_per_character': None, 'output_cost_per_reasoning_token': None, 'output_cost_per_token_above_128k_tokens': None, 'output_cost_per_character_above_128k_tokens': None, 'output_cost_per_token_above_200k_tokens': None, 'output_cost_per_second': None, 'output_cost_per_image': None, 'output_vector_size': None, 'litellm_provider': 'ollama', 'mode': 'chat', 'supports_system_messages': None, 'supports_response_schema': None, 'supports_vision': False, 'supports_function_calling': True, 'supports_tool_choice': False, 'supports_assistant_prefill': False, 'supports_prompt_caching': False, 'supports_audio_input': False, 'supports_audio_output': False, 'supports_pdf_input': False, 'supports_embedding_image_input': False, 'supports_native_streaming': None, 'supports_web_search': False, 'supports_reasoning': False, 'supports_computer_use': False, 'search_context_cost_per_query': None, 'tpm': None, 'rpm': None}
22:12:03 - LiteLLM:DEBUG: utils.py:337 - Final returned optional params: {'stream': False, 'tools': [{'name': 'write_and_execute_script', 'description': 'Write and execute a Python script in a secure sandbox.\nDependencies should be specified as a comma-separated string, e.g. "requests,beautifulsoup4".\nScript output will be captured from stdout. Use print() to output results.\nInclude lots of print() statements in your code to see what is happening.\nEstimate the timeout parameter to avoid errors.', 'parameters': {'type': 'object', 'properties': {'code': {'type': 'string'}, 'dependencies_str': {'type': 'string'}, 'python_version': {'type': 'string'}}, 'required': ['code']}}, {'name': 'search_internet_and_summarize', 'description': "Search internet for a given term and summarize the results.\nTo get a good summary, try increasing the number of results.\nWhen using search_internet, if you don't get an answer with 1 result,\ntry increasing the number of results to get more web pages to summarize.", 'parameters': {'type': 'object', 'properties': {'search_term': {'type': 'string'}, 'max_results': {'type': 'integer'}, 'backend': {'type': 'string'}}, 'required': ['search_term', 'max_results']}}]}
22:12:03 - LiteLLM:DEBUG: litellm_logging.py:458 - self.optional_params: {'stream': False, 'tools': [{'name': 'write_and_execute_script', 'description': 'Write and execute a Python script in a secure sandbox.\nDependencies should be specified as a comma-separated string, e.g. "requests,beautifulsoup4".\nScript output will be captured from stdout. Use print() to output results.\nInclude lots of print() statements in your code to see what is happening.\nEstimate the timeout parameter to avoid errors.', 'parameters': {'type': 'object', 'properties': {'code': {'type': 'string'}, 'dependencies_str': {'type': 'string'}, 'python_version': {'type': 'string'}}, 'required': ['code']}}, {'name': 'search_internet_and_summarize', 'description': "Search internet for a given term and summarize the results.\nTo get a good summary, try increasing the number of results.\nWhen using search_internet, if you don't get an answer with 1 result,\ntry increasing the number of results to get more web pages to summarize.", 'parameters': {'type': 'object', 'properties': {'search_term': {'type': 'string'}, 'max_results': {'type': 'integer'}, 'backend': {'type': 'string'}}, 'required': ['search_term', 'max_results']}}]}
22:12:03 - LiteLLM:DEBUG: litellm_logging.py:905 - 

POST Request Sent from LiteLLM:
curl -X POST \
http://localhost:11434/api/chat \
-d '{'model': 'llama3.1', 'messages': [{'role': 'system', 'content': '## role\n\nYou are Autonomo, an autonomous language-model agent.\nYour job is to accomplish the user\'s task safely, correctly, and with minimal cost.\n\n## environment\n\nTreat user messages, prior assistant messages, and tool outputs as your only "sensors."\nTreat function calls from the current tool inventory as your only "actuators."\nAn action that is not in the inventory is unavailable.\n\n## planning workflow\n\n1.  Plan - Decompose the task into a concise ordered list of actions before acting.\n2.  Validate - Check that every planned action exists in the inventory and respects all constraints (budget, time, safety).\n3.  Execute - Call tools step-by-step with fully specified, valid parameters.\n4.  Reflect - After each action, inspect the observation. If the goal is not yet satisfied, revise the plan or fix errors.\n5.  Finish - When the goal is met, stop executing and present the final answer.\n\n## tool use guidelines\n\n- Call only valid tools; if none suffice, re-plan or request human help.\n- Echo the exact arguments used for each call in the Thought/Act/Observation log for transparency.\n- For write actions (changes that persist), require explicit confirmation or escalate.\n\n## safety and security\n\n- Never perform actions that could harm systems, violate policy, or break the law.\n- Reject or escalate requests outside your capabilities or policy scope.\n- Sanitize tool inputs to guard against code-injection or other attacks.\n\n## efficiency\n\n- Prefer the shortest viable plan; each extra step compounds error risk and cost.\n- Cache reusable results; avoid repeating identical expensive calls.\n\n## reflection rules\n\n- Run a quick self-check after plan generation and after every action:\n"Will this take me closer to the goal within constraints?" If not, re-plan.\n\n## response style\n\n- Unless verbose mode is enabled, show only the final answer; hide internal thoughts.\n- Use clear, plain English in a professional, matter-of-fact tone.\n- Provide citations or tool-call summaries when helpful.'}, {'role': 'user', 'content': "Summarize for me the latest ratings of Taylor Swift's latest album."}], 'options': {}, 'stream': False, 'tools': [{'name': 'write_and_execute_script', 'description': 'Write and execute a Python script in a secure sandbox.\nDependencies should be specified as a comma-separated string, e.g. "requests,beautifulsoup4".\nScript output will be captured from stdout. Use print() to output results.\nInclude lots of print() statements in your code to see what is happening.\nEstimate the timeout parameter to avoid errors.', 'parameters': {'type': 'object', 'properties': {'code': {'type': 'string'}, 'dependencies_str': {'type': 'string'}, 'python_version': {'type': 'string'}}, 'required': ['code']}}, {'name': 'search_internet_and_summarize', 'description': "Search internet for a given term and summarize the results.\nTo get a good summary, try increasing the number of results.\nWhen using search_internet, if you don't get an answer with 1 result,\ntry increasing the number of results to get more web pages to summarize.", 'parameters': {'type': 'object', 'properties': {'search_term': {'type': 'string'}, 'max_results': {'type': 'integer'}, 'backend': {'type': 'string'}}, 'required': ['search_term', 'max_results']}}]}'


22:12:03 - LiteLLM:DEBUG: utils.py:4569 - model_info: {'key': 'qwen3:32b', 'litellm_provider': 'ollama', 'mode': 'chat', 'supports_function_calling': True, 'input_cost_per_token': 0.0, 'output_cost_per_token': 0.0, 'max_tokens': 40960, 'max_input_tokens': 40960, 'max_output_tokens': 40960}
22:12:03 - LiteLLM:DEBUG: litellm_logging.py:1375 - Logging Details LiteLLM-Success Call streaming complete
22:12:03 - LiteLLM:INFO: cost_calculator.py:655 - selected model name for cost calculation: ollama_chat/qwen3:32b
22:12:03 - LiteLLM:DEBUG: utils.py:4269 - checking potential_model_names in litellm.model_cost: {'split_model': 'qwen3:32b', 'combined_model_name': 'ollama_chat/qwen3:32b', 'stripped_model_name': 'qwen3:32b', 'combined_stripped_model_name': 'ollama_chat/qwen3:32b', 'custom_llm_provider': 'ollama_chat'}
22:12:03 - LiteLLM:DEBUG: cost_calculator.py:376 - Returned custom cost for model=ollama_chat/qwen3:32b - prompt_tokens_cost_usd_dollar: 0, completion_tokens_cost_usd_dollar: 0
22:12:03 - LiteLLM:DEBUG: litellm_logging.py:1128 - response_cost: 0.0
22:12:03 - LiteLLM:DEBUG: utils.py:4269 - checking potential_model_names in litellm.model_cost: {'split_model': 'qwen3:32b', 'combined_model_name': 'ollama_chat/qwen3:32b', 'stripped_model_name': 'qwen3:32b', 'combined_stripped_model_name': 'ollama_chat/qwen3:32b', 'custom_llm_provider': 'ollama_chat'}
22:12:03 - LiteLLM:DEBUG: utils.py:4569 - model_info: {'key': 'qwen3:32b', 'litellm_provider': 'ollama', 'mode': 'chat', 'supports_function_calling': True, 'input_cost_per_token': 0.0, 'output_cost_per_token': 0.0, 'max_tokens': 40960, 'max_input_tokens': 40960, 'max_output_tokens': 40960}
22:12:05 - LiteLLM:DEBUG: utils.py:337 - RAW RESPONSE:
{"model":"llama3.1","created_at":"2025-05-24T02:12:05.859654Z","message":{"role":"assistant","content":"","tool_calls":[{"function":{"name":"get_latest_album_ratings","arguments":{"artist_name":"Taylor Swift"}}}]},"done_reason":"stop","done":true,"total_duration":1970347417,"load_duration":666106708,"prompt_eval_count":584,"prompt_eval_duration":1000181500,"eval_count":21,"eval_duration":302957042}


22:12:05 - LiteLLM:DEBUG: token_counter.py:365 - messages in token_counter: [{'role': 'system', 'content': '## role\n\nYou are Autonomo, an autonomous language-model agent.\nYour job is to accomplish the user\'s task safely, correctly, and with minimal cost.\n\n## environment\n\nTreat user messages, prior assistant messages, and tool outputs as your only "sensors."\nTreat function calls from the current tool inventory as your only "actuators."\nAn action that is not in the inventory is unavailable.\n\n## planning workflow\n\n1.  Plan - Decompose the task into a concise ordered list of actions before acting.\n2.  Validate - Check that every planned action exists in the inventory and respects all constraints (budget, time, safety).\n3.  Execute - Call tools step-by-step with fully specified, valid parameters.\n4.  Reflect - After each action, inspect the observation. If the goal is not yet satisfied, revise the plan or fix errors.\n5.  Finish - When the goal is met, stop executing and present the final answer.\n\n## tool use guidelines\n\n- Call only valid tools; if none suffice, re-plan or request human help.\n- Echo the exact arguments used for each call in the Thought/Act/Observation log for transparency.\n- For write actions (changes that persist), require explicit confirmation or escalate.\n\n## safety and security\n\n- Never perform actions that could harm systems, violate policy, or break the law.\n- Reject or escalate requests outside your capabilities or policy scope.\n- Sanitize tool inputs to guard against code-injection or other attacks.\n\n## efficiency\n\n- Prefer the shortest viable plan; each extra step compounds error risk and cost.\n- Cache reusable results; avoid repeating identical expensive calls.\n\n## reflection rules\n\n- Run a quick self-check after plan generation and after every action:\n"Will this take me closer to the goal within constraints?" If not, re-plan.\n\n## response style\n\n- Unless verbose mode is enabled, show only the final answer; hide internal thoughts.\n- Use clear, plain English in a professional, matter-of-fact tone.\n- Provide citations or tool-call summaries when helpful.'}, {'role': 'user', 'content': "Summarize for me the latest ratings of Taylor Swift's latest album."}], text in token_counter: None
22:12:05 - LiteLLM:DEBUG: token_counter.py:365 - messages in token_counter: None, text in token_counter: 
22:12:05 - LiteLLM:INFO: utils.py:1212 - Wrapper: Completed Call, calling success_handler
22:12:05 - LiteLLM:INFO: cost_calculator.py:655 - selected model name for cost calculation: ollama_chat/llama3.1
22:12:05 - LiteLLM:DEBUG: utils.py:4269 - checking potential_model_names in litellm.model_cost: {'split_model': 'llama3.1', 'combined_model_name': 'ollama_chat/llama3.1', 'stripped_model_name': 'llama3.1', 'combined_stripped_model_name': 'ollama_chat/llama3.1', 'custom_llm_provider': 'ollama_chat'}
22:12:05 - LiteLLM:DEBUG: litellm_logging.py:1350 - Logging Details LiteLLM-Success Call: Cache_hit=None
22:12:05 - LiteLLM:INFO: cost_calculator.py:655 - selected model name for cost calculation: ollama_chat/llama3.1
22:12:05 - LiteLLM:DEBUG: utils.py:4269 - checking potential_model_names in litellm.model_cost: {'split_model': 'llama3.1', 'combined_model_name': 'ollama_chat/llama3.1', 'stripped_model_name': 'llama3.1', 'combined_stripped_model_name': 'ollama_chat/llama3.1', 'custom_llm_provider': 'ollama_chat'}
22:12:05 - LiteLLM:DEBUG: cost_calculator.py:376 - Returned custom cost for model=ollama_chat/llama3.1 - prompt_tokens_cost_usd_dollar: 0, completion_tokens_cost_usd_dollar: 0
22:12:05 - LiteLLM:DEBUG: litellm_logging.py:1128 - response_cost: 0.0
22:12:05 - LiteLLM:DEBUG: utils.py:4269 - checking potential_model_names in litellm.model_cost: {'split_model': 'llama3.1', 'combined_model_name': 'ollama_chat/llama3.1', 'stripped_model_name': 'llama3.1', 'combined_stripped_model_name': 'ollama_chat/llama3.1', 'custom_llm_provider': 'ollama_chat'}
22:12:05 - LiteLLM:DEBUG: cost_calculator.py:376 - Returned custom cost for model=ollama_chat/llama3.1 - prompt_tokens_cost_usd_dollar: 0, completion_tokens_cost_usd_dollar: 0
22:12:05 - LiteLLM:DEBUG: litellm_logging.py:1128 - response_cost: 0.0

I also updated the gist so that if you have uv installed on your system, you can reproduce what I see.

[ericmjl]

@krrishdholakia I'm not sure if this might add noise or signal to this thread, but I just forked and cloned litellm + ollama-python into a single workspace and got cursor to try to diagnose this issue. What I have is the following:

---

I'll help you investigate this issue thoroughly. Let's start by understanding the key components involved and then examine the relevant files.

First, let me search for files related to Ollama integration and tool calling in LiteLLM:

Search files...
Let me also search for the Ollama implementation files:

Search files...
Let me also look at the Ollama completion transformation code:

Search files...
Let me also look at the actual Ollama implementation:

Ran tool
Let me read the main Ollama transformation file:

Read file: /Users/ericmjl/github/litellm/litellm/llms/ollama/completion/transformation.py
Let me also look at the Ollama completion implementation:

Ran tool

Read file: /Users/ericmjl/github/litellm/litellm/llms/ollama/completion/handler.py

Based on my analysis of the code and the issue, I can identify several key points:

1. The issue appears to be in how tool calling is handled between LiteLLM and Ollama. The key files involved are:

   • /litellm/llms/ollama/completion/transformation.py - Handles transforming requests/responses
   • /litellm/llms/ollama/completion/handler.py - Handles the actual API calls

2. Looking at the debug logs you provided, I notice a few important things:

   • The request is being sent to Ollama with the tools parameter correctly formatted
   • The response from Ollama includes a tool call, but it's not being properly transformed into the expected format

3. The key issue appears to be in the transformation layer. In transformation.py, there's a method _transform_response_json_function_call that handles tool calls, but it seems the response format from Ollama might not match what this method expects.

Let me look at the specific transformation code for tool calls:

Read file: /Users/ericmjl/github/litellm/litellm/llms/ollama/completion/transformation.py

After analyzing the code and the debug logs, I can identify the root cause of the issue:

1. The problem is in the transformation layer between Ollama's response format and LiteLLM's expected format. Looking at your debug logs, Ollama is returning a response in this format:

{
    "model": "llama3.1",
    "created_at": "2025-05-24T02:12:05.859654Z",
    "message": {
        "role": "assistant",
        "content": "",
        "tool_calls": [{
            "function": {
                "name": "get_latest_album_ratings",
                "arguments": {
                    "artist_name": "Taylor Swift"
                }
            }
        }]
    },
    "done_reason": "stop",
    "done": true,
    ...
}

2. However, in the transformation code (transformation.py), the code is expecting a different format. The current transformation logic in transform_response is looking for:

   • A response with a "response" field containing a JSON string
   • The JSON string should have a "name" and "arguments" structure

3. The key issue is that the transformation code is not properly handling Ollama's native tool calling format. Instead, it's trying to parse the response as a JSON string within the "response" field.

Here's what needs to be fixed:

1. The transform_response method in OllamaConfig needs to be updated to handle Ollama's native tool calling format. It should check for the presence of message.tool_calls in the response and transform it accordingly.

2. The transformation should look something like this:

if "message" in response_json and "tool_calls" in response_json["message"]:
    # Handle native Ollama tool calls
    tool_calls = response_json["message"]["tool_calls"]
    message = litellm.Message(
        content=None,
        tool_calls=[
            {
                "id": f"call_{str(uuid.uuid4())}",
                "function": {
                    "name": tool_call["function"]["name"],
                    "arguments": json.dumps(tool_call["function"]["arguments"]),
                },
                "type": "function",
            }
            for tool_call in tool_calls
        ],
    )
    model_response.choices[0].message = message
    model_response.choices[0].finish_reason = "tool_calls"

Would you like me to propose a fix for this issue? I can help create a pull request that updates the transformation logic to properly handle Ollama's native tool calling format.

---

Before I make any move with a PR, does this diagnosis match your understanding of the problem?

I am, obviously, doing this diagnosis without the deep familiarity you have with the codebase, so both cursor and I might be wrong here.

[krrishdholakia]

22:12:05 - LiteLLM:DEBUG: utils.py:337 - RAW RESPONSE:
{"model":"llama3.1","created_at":"2025-05-24T02:12:05.859654Z","message":{"role":"assistant","content":"","tool_calls":[{"function":{"name":"get_latest_album_ratings","arguments":{"artist_name":"Taylor Swift"}}}]},"done_reason":"stop","done":true,"total_duration":1970347417,"load_duration":666106708,"prompt_eval_count":584,"prompt_eval_duration":1000181500,"eval_count":21,"eval_duration":302957042}

I can see the raw response contains a tool call

[ericmjl]

Yes, though if you look carefully, it is a hallucinated tool name. I only provided two functions, none of which is “ get_latest_album_ratings”. Within the Marino notebook, ollama python library calls never contain hallucinated tool names.

[lsorber]

I wrote a debuggable test to help hunt down the problem:

"""Test LiteLLM's support for tool use with Ollama."""

import asyncio

import litellm
import pytest
from litellm import stream_chunk_builder


@pytest.mark.parametrize(
    "stream", [pytest.param(False, id="stream=False"), pytest.param(True, id="stream=True")]
)
@pytest.mark.parametrize(
    "sync",
    [pytest.param(False, id="sync=False"), pytest.param(True, id="sync=True")],
)
def test_litellm_ollama_tool_use(sync: bool, stream: bool) -> None:  # noqa: FBT001
    """Test LiteLLM's support for tool use with Ollama."""
    tools = [
        {
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Get the weather for a location",
                "parameters": {
                    "type": "object",
                    "properties": {"location": {"type": "string"}},
                    "required": ["location"],
                },
            },
        }
    ]
    completion_kwargs = {
        "model": "ollama_chat/qwen3:8b",  # Important: use `ollama_chat` instead of `ollama`
        "messages": [{"role": "user", "content": "What's the weather like in New York?"}],
        "tools": tools,
        "stream": stream,
    }
    if sync:
        # Synchronous completion
        response = litellm.completion(**completion_kwargs)
        if stream:
            response = stream_chunk_builder(list(response))
    else:
        # Asynchronous completion
        async def _acompletion(**kwargs):  # type: ignore[no-untyped-def]
            response = await litellm.acompletion(**kwargs)
            if kwargs.get("stream"):
                return [chunk async for chunk in response]
            return response

        response = asyncio.run(_acompletion(**completion_kwargs))  # type: ignore[no-untyped-call]
        if stream:
            response = stream_chunk_builder(response)
    assert response.choices[0].message.tool_calls, "No tool call detected"

Results:

1. ✅ completion + stream=False results in a tool call
2. ✅ acompletion + stream=False results in a tool call
3. ❌ completion + stream=True does not produce a valid response
4. ❌ acompletion + stream=True does not produce a valid response

For both (3) and (4), this is what the response looks like (it's an empty response with no tool calls):

ModelResponse(id='chatcmpl-1d0b06e6-bf33-4623-b73b-859b25b5f60d', created=1748125964, model='qwen3:8b', object='chat.completion', system_fingerprint=None, choices=[Choices(finish_reason='stop', index=0, message=Message(content='', role='assistant', tool_calls=None, function_call=None, provider_specific_fields=None))], usage=Usage(completion_tokens=0, prompt_tokens=0, total_tokens=0, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=None), prompt_tokens_details=None))

⚠️ In conclusion: with the latest LiteLLM v1.71.0, streaming tool use with Ollama is broken!

[lsorber]

Both failures can be explained by CustomStreamWrapper.handle_ollama_chat_stream not handling tool calls:

litellm/litellm/litellm_core_utils/streaming_handler.py

Lines 552 to 585 in 76107bf

	def handle_ollama_chat_stream(self, chunk):
	# for ollama_chat/ provider
	try:
	if isinstance(chunk, dict):
	json_chunk = chunk
	else:
	json_chunk = json.loads(chunk)
	if "error" in json_chunk:
	raise Exception(f"Ollama Error - {json_chunk}")
	text = ""
	is_finished = False
	finish_reason = None
	if json_chunk["done"] is True:
	text = ""
	is_finished = True
	finish_reason = "stop"
	return {
	"text": text,
	"is_finished": is_finished,
	"finish_reason": finish_reason,
	}
	elif "message" in json_chunk:
	print_verbose(f"delta content: {json_chunk}")
	text = json_chunk["message"]["content"]
	return {
	"text": text,
	"is_finished": is_finished,
	"finish_reason": finish_reason,
	}
	else:
	raise Exception(f"Ollama Error - {json_chunk}")
	except Exception as e:
	raise e

Here's an example json_chunk returned by Ollama, the tool_calls of which are silently dropped by handle_ollama_chat_stream:

{
    "model": "qwen3:8b",
    "created_at": "2025-05-24...",
    "message": {
        "role": "assistant",
        "content": "",
        "tool_calls": [
            {
                "function": {
                    "name": "get_weather",
                    "arguments": {
                        "location": "New York"
                    }
                }
            }
        ]
    },
    "done": false
}

[ericmjl]

Ok, I'd like to confirm two things on my side:

1. I figured out that the hallucinated calls was because I didn't put in type and function as top-level keys for a function's definition. That is my bad! With those included, all models work. I think we can close out that part of the issue. Apologies for leading us down a rabbit hole here.

2. I can confirm that I also see what @lsorber is observing with streaming. That might be a better direction to tackle this issue.