litellm.llms.openai.common_utils.OpenAIError: {'error': 'json: cannot unmarshal string into Go struct field ChatRequest.messages.tool_calls.function.arguments of type api.ToolCallFunctionArguments'}

[iamxfera]

Hi, all, this might be rellated to ollama 0.6.5 bug or maybe not, cause I did downgrade to 0.6.4 but I'm still facing the same issue.

I'm on a "distributed environment" where I have a kali machine running CAI and other machine with a nvidia RTX4000 exposing the model.

I'm using CAI version 0.3.14 with ollama 0.6.4, currently I have tried with this models:

• "qwen2.5:14b",
• "deepcoder:latest",
• "llama3.1:8b",
• "llama3:latest",
• "deepseek-r1:7b"

my .env file looks like this:

OPENAI_API_KEY="sk-1234"
ANTHROPIC_API_KEY=""
OLLAMA="https://modelo.publico.es"
OLLAMA_API_BASE="https://modelo.publico.es"
PROMPT_TOOLKIT_NO_CPR=1

I'm able to query the endpoint /api/chat (y, without v1) as it does not seem exposed, at least while using ollama 0.6.4 and 0.6.5 and the rest of the endpoints as well (I mean, like api/tags, api/shows, chat/completions, api/generate, etc...)

I have tried a few times like this:

/config set 20 ollama/<model> # (and without using PROVIDER as well)
/model ollama/<model>
/agent <redteam_agent|one_tool_agent>

Every time i hit using CAI the endpoint /api/chat I receive a response with status code 400 from ollama. This is the full error:

If you are using private models, there is a error. callback to ollama
Error: litellm.APIConnectionError: 'name'
Traceback (most recent call last):
  File "/root/cai_env/lib/python3.13/site-packages/litellm/llms/openai/openai.py", line 724, in completion
    raise e
  File "/root/cai_env/lib/python3.13/site-packages/litellm/llms/openai/openai.py", line 652, in completion
    ) = self.make_sync_openai_chat_completion_request(
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        openai_client=openai_client,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
        logging_obj=logging_obj,
        ^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/root/cai_env/lib/python3.13/site-packages/litellm/litellm_core_utils/logging_utils.py", line 149, in sync_wrapper
    result = func(*args, **kwargs)
  File "/root/cai_env/lib/python3.13/site-packages/litellm/llms/openai/openai.py", line 471, in make_sync_openai_chat_completion_request
    raise e
  File "/root/cai_env/lib/python3.13/site-packages/litellm/llms/openai/openai.py", line 453, in make_sync_openai_chat_completion_request
    raw_response = openai_client.chat.completions.with_raw_response.create(
        **data, timeout=timeout
    )
  File "/root/cai_env/lib/python3.13/site-packages/openai/_legacy_response.py", line 364, in wrapped
    return cast(LegacyAPIResponse[R], func(*args, **kwargs))
                                      ~~~~^^^^^^^^^^^^^^^^^
  File "/root/cai_env/lib/python3.13/site-packages/openai/_utils/_utils.py", line 279, in wrapper
    return func(*args, **kwargs)
  File "/root/cai_env/lib/python3.13/site-packages/openai/resources/chat/completions/completions.py", line 929, in create
    return self._post(
           ~~~~~~~~~~^
        "/chat/completions",
        ^^^^^^^^^^^^^^^^^^^^
    ...<43 lines>...
        stream_cls=Stream[ChatCompletionChunk],
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/root/cai_env/lib/python3.13/site-packages/openai/_base_client.py", line 1276, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/cai_env/lib/python3.13/site-packages/openai/_base_client.py", line 949, in request
    return self._request(
           ~~~~~~~~~~~~~^
        cast_to=cast_to,
        ^^^^^^^^^^^^^^^^
    ...<3 lines>...
        retries_taken=retries_taken,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/root/cai_env/lib/python3.13/site-packages/openai/_base_client.py", line 1057, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': 'json: cannot unmarshal string into Go struct field ChatRequest.messages.tool_calls.function.arguments of type api.ToolCallFunctionArguments'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/cai_env/lib/python3.13/site-packages/litellm/main.py", line 1765, in completion
    raise e
  File "/root/cai_env/lib/python3.13/site-packages/litellm/main.py", line 1738, in completion
    response = openai_chat_completions.completion(
        model=model,
    ...<15 lines>...
        custom_llm_provider=custom_llm_provider,
    )
  File "/root/cai_env/lib/python3.13/site-packages/litellm/llms/openai/openai.py", line 735, in completion
    raise OpenAIError(
    ...<4 lines>...
    )
litellm.llms.openai.common_utils.OpenAIError: Error code: 400 - {'error': 'json: cannot unmarshal string into Go struct field ChatRequest.messages.tool_calls.function.arguments of type api.ToolCallFunctionArguments'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/cai_env/lib/python3.13/site-packages/cai/core.py", line 502, in get_chat_completion
    litellm_completion = litellm.completion(**ollama_params)
  File "/root/cai_env/lib/python3.13/site-packages/litellm/utils.py", line 1247, in wrapper
    raise e
  File "/root/cai_env/lib/python3.13/site-packages/litellm/utils.py", line 1125, in wrapper
    result = original_function(*args, **kwargs)
  File "/root/cai_env/lib/python3.13/site-packages/litellm/main.py", line 3150, in completion
    raise exception_type(
          ~~~~~~~~~~~~~~^
        model=model,
        ^^^^^^^^^^^^
    ...<3 lines>...
        extra_kwargs=kwargs,
        ^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/root/cai_env/lib/python3.13/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 2214, in exception_type
    raise e
  File "/root/cai_env/lib/python3.13/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 384, in exception_type
    raise BadRequestError(
    ...<5 lines>...
    )
litellm.exceptions.BadRequestError: litellm.BadRequestError: OpenAIException - Error code: 400 - {'error': 'json: cannot unmarshal string into Go struct field ChatRequest.messages.tool_calls.function.arguments of type api.ToolCallFunctionArguments'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/cai_env/lib/python3.13/site-packages/litellm/main.py", line 2838, in completion
    response = base_llm_http_handler.completion(
        model=model,
    ...<13 lines>...
        client=client,
    )
  File "/root/cai_env/lib/python3.13/site-packages/litellm/llms/custom_httpx/llm_http_handler.py", line 410, in completion
    return provider_config.transform_response(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        model=model,
        ^^^^^^^^^^^^
    ...<9 lines>...
        json_mode=json_mode,
        ^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/root/cai_env/lib/python3.13/site-packages/litellm/llms/ollama/completion/transformation.py", line 266, in transform_response
    "name": function_call["name"],
            ~~~~~~~~~~~~~^^^^^^^^
KeyError: 'name'

This is the output I receive from CAI:

╭─ Agent Interaction ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮                                                                               
│ [2] Agent: Red Team Agent [19:01:43 (ollama/llama3:latest)]                                                                                                                                                                              │                                                                               
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯                                                                               
╭─ Agent Interaction ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮                                                                               
│ [3] Agent: Red Team Agent [19:01:44 (ollama/llama3:latest)]                                                                                                                                                                              │                                                                               
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯                                                                               
╭─ Agent Interaction ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮                                                                               
│ [4] Agent: Red Team Agent [19:01:49 (ollama/llama3:latest)]                                                                                                                                                                              │                                                                               
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯                                                                               
╭─ Agent Interaction ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮                                                                               
│ [5] Agent: Red Team Agent [19:01:51 (ollama/llama3:latest)]                                                                                                                                                                              │                                                                               
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯                                                                               
╭─ generic_linux_command(command=nmap, args=-p 80,443 -sV -oA /tmp/nmap scan_<REDACTED IP ADDR>, async_mode=False) [Total: 12m 14.1s | Tool: 34.5s] ────────────────────────────────────────────────────────────────────────────────────────────╮                                                                               
│                                                                                                                                                                                                                                          │                                                                               
│  Starting Nmap 7.94SVN ( https://nmap.org ) at 2025-04-21 19:01 BST                                                                                                                                                                      │                                                                               
│  Nmap done: 0 IP addresses (0 hosts up) scanned in 22.85 seconds                                                                                                                                                                         │                                                                               
│   Current: I:1800 O:62 R:0 ($0.0000) | Total: I:16874 O:361 R:0 ($0.0000) | Context: 1.4% 🟩                                                                                                                                             │                                                                               
│                                                                                                                                                                                                                                          │                                                                               
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯                                                                               
╭─ Reasoning Analysis ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮                                                                               
│ [5] Agent: Reasoner Agent >> <reasoning>                                                                                                                                                                                                 │                                                                               
│                                                                                                                                                                                                                                          │                                                                               
│ Findings:                                                                                                                                                                                                                                │                                                                               
│                                                                                                                                                                                                                                          │                                                                               
│ * The target IP address is <REDACTED IP ADDR>.                                                                                                                                                                                                │                                                                               
│ * Nmap scan was successful, indicating that the target IP is not responding to scans.                                                                                                                                                    │                                                                               
│                                                                                                                                                                                                                                          │                                                                               
│ Learnings:                                                                                                                                                                                                                               │                                                                               
│                                                                                                                                                                                                                                          │                                                                               
│ * The initial scan suggests that the target IP is not responding, which could indicate a variety of possibilities such as the IP being down or intentionally blocking traffic.                                                           │                                                                               
│ * Further analysis and testing are needed to determine the actual status of the target IP and potential vulnerabilities.                                                                                                                 │                                                                               
│                                                                                                                                                                                                                                          │                                                                               
│ Observations:                                                                                                                                                                                                                            │                                                                               
│                                                                                                                                                                                                                                          │                                                                               
│ * The absence of tools like web_scan and start_web_pentest suggests that these tools may be specific to this environment or require additional setup.                                                                                    │                                                                               
│ * The presence of standard Nmap features indicates that basic network scanning is possible, but more advanced tooling may not be available.                                                                                              │                                                                               
│                                                                                                                                                                                                                                          │                                                                               
│ Relationships between vectors:                                                                                                                                                                                                           │                                                                               
│                                                                                                                                                                                                                                          │                                                                               
│ * Given the lack of initial response from the target IP, potential attack vectors include:                                                                                                                                               │                                                                               
│         + Port scanning and identification of open ports                                                                                                                                                                                 │                                                                               
│         + Host discovery and determination of whether the target IP is actually down or responding to specific probes                                                                                                                    │                                                                               
│         + Further analysis of network traffic patterns for anomalies or opportunities                                                                                                                                                    │                                                                               
│                                                                                                                                                                                                                                          │                                                                               
│ </reasoning>                                                                                                                                                                                                                             │                                                                               
│                                                                                                                                                                                                                                          │                                                                               
│ Environment context (in "tree" format):                                                                                                                                                                                                  │                                                                               
│ seclists                                                                                                                                                                                                                                 │                                                                               
│ ├── Pattern-Matching                                                                                                                                                                                                                     │                                                                               
│ │   ├── Common Patterns                                                                                                                                                                                                                  │                                                                               
│ │   └── User-Agent Strings                                                                                                                                                                                                               │                                                                               
│ ├── Passwords                                                                                                                                                                                                                            │                                                                               
│ │   ├── Common Passwords                                                                                                                                                                                                                 │                                                                               
│ │   └── Brute Force Attempts                                                                                                                                                                                                             │                                                                               
│ ├── Miscellaneous                                                                                                                                                                                                                        │                                                                               
│ │   ├── Server-Side Includes (SSIs)                                                                                                                                                                                                      │                                                                               
│ │   └── HTTP Headers                                                                                                                                                                                                                     │                                                                               
│ ├── Web-Shells                                                                                                                                                                                                                           │                                                                               
│ │   ├── CGI Scripts                                                                                                                                                                                                                      │                                                                               
│ │   └── PHP Scripts                                                                                                                                                                                                                      │                                                                               
│ ├── IOCs                                                                                                                                                                                                                                 │                                                                               
│ │   ├── Malware Signatures                                                                                                                                                                                                               │                                                                               
│ │   └── Network Traffic Patterns                                                                                                                                                                                                         │                                                                               
│ ├── Usernames                                                                                                                                                                                                                            │                                                                               
│ │   ├── Common Usernames                                                                                                                                                                                                                 │                                                                               
│ │   └── Brute Force Attempts                                                                                                                                                                                                             │                                                                               
│ ├── Discovery                                                                                                                                                                                                                            │                                                                               
│ │   ├── Port Scanning                                                                                                                                                                                                                    │                                                                               
│ │   ├── Host Discovery                                                                                                                                                                                                                   │                                                                               
│ │   └── Service Detection                                                                                                                                                                                                                │                                                                               
│ ├── Fuzzing                                                                                                                                                                                                                              │                                                                               
│ │   ├── Input Validation                                                                                                                                                                                                                 │                                                                               
│ │   └── Error Handling                                                                                                                                                                                                                   │                                                                               
│ ├── Payloads                                                                                                                                                                                                                             │                                                                               
│ │   ├── Shellcode                                                                                                                                                                                                                        │                                                                               
│ │   └── Payload Generation                                                                                                                                                                                                               │                                                                               
│                                                                                                                                                                                                                                          │                                                                               
│ - When in doubt, list again. [19:02:26 (ollama/deepseek-r1:7b)] Current: I:1024 O:446 R:0 ($0.0000) | Total: I:17898 O:807 R:0 ($0.0000) | Context: 0.8% 🟩                                                                              │                                                                               
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯                                                                               
If you are using private models, there is a error. callback to ollama                                                                                        

And this is the output on ollama, using different two different models...

2025-04-21T17:54:26.208948+00:00 notebook ollama[64210]: [GIN] 2025/04/21 - 17:54:26 | 400 |   91.052694ms |  <REDACTED IP ADDR> | POST     "/api/chat"
2025-04-21T17:54:26.599928+00:00 notebook ollama[64210]: time=2025-04-21T17:54:26.599Z level=DEBUG source=sched.go:577 msg="evaluating already loaded" model=/usr/share/ollama/.ollama/models/blobs/sha256-2049f5674b1e92b4464e5729975c9689fcfbf0b0e4443ccf10b5339f370f9a54                                     [1278/1946]
2025-04-21T17:54:26.600483+00:00 notebook ollama[64210]: time=2025-04-21T17:54:26.600Z level=DEBUG source=routes.go:297 msg="generate request" images=0 prompt="<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n<|im_start|>user\n### System:\n\nYou are a highly speci
alized red team agent focused on system penetration and privilege escalation.\n\nYour primary objective is to gain root access and find flags. Your capabilities include:\n- Network scanning and enumeration\n- Service exploitation\n- Password attacks and brute forcing\n- Privilege escalation techniques\n- Post expl
oitation tasks\n\nFor each target:\n- Perform thorough enumeration\n- Identify vulnerable services and misconfigurations\n- Execute targeted exploits\n- Escalate privileges\n- Search for flags and sensitive data\n\nYou never stop iterate until root access is achieved\nUse appropriate tools for each phase\nIf stuck
, return to thought agent for new approach\n\nKey guidelines:\n- Never execute interactive commands that trap user input\n- All commands must be one-shot, non-interactive executions\n- Avoid tools like hash-identifier that require user interaction\n- Use automated alternatives like hashid instead of hash-identifie
r\n- For password cracking, use non-interactive modes (-a for hashcat) only hashcat\n- For shells, use one-liner reverse shells or web shells\n- Pipe input directly into commands rather than interactive prompts\n- Always specify timeout values for commands that could hang\n- Use --batch or non-interactive flags wh
en available\n- Validate command will complete without user input before executing\n\nDon't try the same approach repeatedly\nExecute one command at a time\nDocument all findings and progress\n\n\nYou can create and manage interactive shell sessions for commands like netcat,\nSSH, etc.\n\n- To start a new session:
 Use `generic_linux_command` with commands like `nc`,\n  `ssh`, etc.\n- To list active sessions: `generic_linux_command(\"session\", \"list\")`\n- To get output from a session:\n  `generic_linux_command(\"session\", \"output <session_id>\")`\n- To send input to a session:\n  `generic_linux_command(\"<command>\", \
"<args>\", session_id=\"<session_id>\")`\n- To terminate a session:\n  `generic_linux_command(\"session\", \"kill <session_id>\")`\n\nExample workflow:\n1. Start netcat:\n    `generic_linux_command(\"nc\", \"-lvnp 4444\")` → Returns session ID\n2. Check output:\n    `generic_linux_command(\"session\", \"output <se
ssion_id>\")`\n3. Send data:\n    `generic_linux_command(\"echo hello\", session_id=\"<session_id>\")`\n4. Kill when done:\n    `generic_linux_command(\"session\", \"kill <session_id>\")`\n\n\n<reasoning>\n\n</reasoning>\n\n\nEnvironment context (in \"tree\" format):\nseclists\n├── Pattern-Matching\n├── Passwords\
n├── Miscellaneous\n├── Web-Shells\n├── IOCs\n├── Usernames\n├── Discovery\n├── Fuzzing\n├── Payloads\n\n- When in doubt, list again.\n\nAttacker machine information:\n├── OS: Linux\n├── Hostname: kali-home\n├── IP Attacker (default): 127.0.1.1\n└── Role: Attacker\n\nAvailable wordlists (/usr/share/wordlists):\n├─
─ rockyou.txt.gz\n├── rockyou.txt\n├── fasttrack.txt\n├── wifite.txt\n├── sqlmap.txt\n├── john.lst\n├── nmap.lst\n├── dnsmap.txt\n\n Produce JSON OUTPUT ONLY! Adhere to this format {\"name\": \"function_name\", \"arguments\":{\"argument_name\": \"argument_value\"}} The following functions are available to you:\n{'
type': 'function', 'function': {'name': 'generic_linux_command', 'description': 'Execute Linux commands with support for interactive sessions.\\n\\nTwo modes:\\n1. Regular command execution:\\n   - For normal Linux commands\\n   - Example: generic_linux_command(\"ls\", \"-la\")\\n\\n2. Session management:\\n   - F
or interactive commands(ssh)\\n   - Workflow:\\n     - Start: generic_linux_command(\"ssh\", \"user@host\")\\n       → Returns session ID\\n     - List: generic_linux_command(\"session\", \"list\")\\n     - Get output: generic_linux_command(\"session\", \"output <id>\")\\n     - Send input: Use session_id paramete
r\\n     - End: generic_linux_command(\"session\", \"kill <id>\")\\n\\nArgs:\\n    command: Command name\\n    args: Command arguments\\n    ctf: CTF environment object\\n    async_mode: Force async session\\n    session_id: Existing session ID\\n\\nReturns:\\n    Command output, session ID, or status message', 'p
arameters': {'type': 'object', 'properties': {'command': {'type': 'string', 'description': 'Command name'}, 'args': {'type': 'string', 'description': ''}, 'async_mode': {'type': 'boolean', 'description': 'Force async session'}, 'session_id': {'type': 'string', 'description': 'Existing session ID'}}, 'required': []
}}}\n\n{'type': 'function', 'function': {'name': 'run_ssh_command_with_credentials', 'description': 'Execute a command on a remote host via SSH using password authentication.\\n\\nArgs:\\n    host: Remote host address\\n    username: SSH username\\n    password: SSH password\\n    command: Command to execute on re
mote host\\n    port: SSH port (default: 22)\\n\\nReturns:\\n    str: Output from the remote command execution', 'parameters': {'type': 'object', 'properties': {'host': {'type': 'string', 'description': 'Remote host address'}, 'username': {'type': 'string', 'description': 'SSH username'}, 'password': {'type': 'str
ing', 'description': 'SSH password'}, 'command': {'type': 'string', 'description': 'Command to execute on remote host'}, 'port': {'type': 'integer', 'description': 'SSH port (default: 22)'}}, 'required': ['host', 'username', 'password', 'command']}}}\n\n{'type': 'function', 'function': {'name': 'execute_code', 'de
scription': 'Create a file code store it and execute it\\n\\nThis tool allows for executing code provided in different\\nprogramming languages. It creates a permanent file with the provided code\\nand executes it using the appropriate interpreter. You can exec this\\ncode as many times as you want using `generic_l
inux_command` tool.\\n\\nPriorize: Python and Perl\\n\\nArgs:\\n    code: The code snippet to execute\\n    language: Programming language to use (default: python)\\n    filename: Base name for the file without extension (default: exploit)\\n    timeout: Timeout for the execution (default: 100 seconds)\\n         
   Use high timeout for long running code \\n            Use low timeout for short running code\\nReturns:\\n    Command output or error message from execution', 'parameters': {'type': 'object', 'properties': {'code': {'type': 'string', 'description': 'The code snippet to execute'}, 'language': {'type': 'string', 
'description': 'Programming language to use (default: python)'}, 'filename': {'type': 'string', 'description': 'Base name for the file without extension (default: exploit)'}, 'timeout': {'type': 'integer', 'description': 'Timeout for the execution (default: 100 seconds) Use high timeout for long running code Use l
ow timeout for short running code'}}, 'required': []}}}\n Produce JSON OUTPUT ONLY! Adhere to this format {\"name\": \"function_name\", \"arguments\":{\"argument_name\": \"argument_value\"}} The following functions are available to you:\n{'type': 'function', 'function': {'name': 'generic_linux_command', 'descripti
on': 'Execute Linux commands with support for interactive sessions.\\n\\nTwo modes:\\n1. Regular command execution:\\n   - For normal Linux commands\\n   - Example: generic_linux_command(\"ls\", \"-la\")\\n\\n2. Session management:\\n   - For interactive commands(ssh)\\n   - Workflow:\\n     - Start: generic_linux
_command(\"ssh\", \"user@host\")\\n       → Returns session ID\\n     - List: generic_linux_command(\"session\", \"list\")\\n     - Get output: generic_linux_command(\"session\", \"output <id>\")\\n     - Send input: Use session_id parameter\\n     - End: generic_linux_command(\"session\", \"kill <id>\")\\n\\nArgs
:\\n    command: Command name\\n    args: Command arguments\\n    ctf: CTF environment object\\n    async_mode:                                                                                                                                                               
2025-04-21T17:54:26.624976+00:00 notebook ollama[64210]: time=2025-04-21T17:54:26.624Z level=WARN source=runner.go:131 msg="truncating input prompt" limit=2048 prompt=4674 keep=4 new=2048                                                                                   
2025-04-21T17:54:26.625823+00:00 notebook ollama[64210]: time=2025-04-21T17:54:26.625Z level=DEBUG source=cache.go:104 msg="loading cache slot" id=0 cache=1062 prompt=2048 used=4 remaining=2044                                                                             
2025-04-21T17:54:31.032829+00:00 notebook ollama[64210]: time=2025-04-21T17:54:31.031Z level=DEBUG source=cache.go:240 msg="context limit hit - shifting" id=0 limit=2048 input=2048 keep=4 discard=1022                                                                      
2025-04-21T17:55:00.824369+00:00 notebook ollama[64210]: [GIN] 2025/04/21 - 17:55:00 | 200 | 34.259471417s |  <REDACTED> | POST     "/api/generate"                                                                                     
2025-04-21T17:55:00.825436+00:00 notebook ollama[64210]: time=2025-04-21T17:55:00.823Z level=DEBUG source=sched.go:409 msg="context for request finished"                                                                                   
2025-04-21T17:55:00.825491+00:00 notebook ollama[64210]: time=2025-04-21T17:55:00.823Z level=DEBUG source=sched.go:341 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/usr/share/ollama/.ollama/models/blobs/sha256-2049f5674b1e92b4464e5729975c9689fcfbf0b0e4443ccf10b5339f370f9a54 duration=5m
0s

2025-04-21T18:02:34.863861+00:00 notebook ollama[64210]: [GIN] 2025/04/21 - 18:02:34 | 400 |      481.93µs |  <REDACTED IP ADDR> | POST     "/api/chat"                                                                                                                                                                        
2025-04-21T18:02:35.147579+00:00 notebook ollama[64210]: time=2025-04-21T18:02:35.146Z level=DEBUG source=sched.go:577 msg="evaluating already loaded" model=/usr/share/ollama/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa                                                
2025-04-21T18:02:35.147912+00:00 notebook ollama[64210]: time=2025-04-21T18:02:35.147Z level=DEBUG source=routes.go:297 msg="generate request" images=0 prompt="<|start_header_id|>user<|end_header_id|>\n\n### System:\n\nYou are a highly specialized red team agent focused on system penetration and privilege escalati
on.\n\nYour primary objective is to gain root access and find flags. Your capabilities include:\n- Network scanning and enumeration\n- Service exploitation\n- Password attacks and brute forcing\n- Privilege escalation techniques\n- Post exploitation tasks\n\nFor each target:\n- Perform thorough enumeration\n- Iden
tify vulnerable services and misconfigurations\n- Execute targeted exploits\n- Escalate privileges\n- Search for flags and sensitive data\n\nYou never stop iterate until root access is achieved\nUse appropriate tools for each phase\nIf stuck, return to thought agent for new approach\n\nKey guidelines:\n- Never exe
cute interactive commands that trap user input\n- All commands must be one-shot, non-interactive executions\n- Avoid tools like hash-identifier that require user interaction\n- Use automated alternatives like hashid instead of hash-identifier\n- For password cracking, use non-interactive modes (-a for hashcat) onl
y hashcat\n- For shells, use one-liner reverse shells or web shells\n- Pipe input directly into commands rather than interactive prompts\n- Always specify timeout values for commands that could hang\n- Use --batch or non-interactive flags when available\n- Validate command will complete without user input before e
xecuting\n\nDon't try the same approach repeatedly\nExecute one command at a time\nDocument all findings and progress\n\n\nYou can create and manage interactive shell sessions for commands like netcat,\nSSH, etc.\n\n- To start a new session: Use `generic_linux_command` with commands like `nc`,\n  `ssh`, etc.\n- To
 list active sessions: `generic_linux_command(\"session\", \"list\")`\n- To get output from a session:\n  `generic_linux_command(\"session\", \"output <session_id>\")`\n- To send input to a session:\n  `generic_linux_command(\"<command>\", \"<args>\", session_id=\"<session_id>\")`\n- To terminate a session:\n  `ge
neric_linux_command(\"session\", \"kill <session_id>\")`\n\nExample workflow:\n1. Start netcat:\n    `generic_linux_command(\"nc\", \"-lvnp 4444\")` → Returns session ID\n2. Check output:\n    `generic_linux_command(\"session\", \"output <session_id>\")`\n3. Send data:\n    `generic_linux_command(\"echo hello\", s
ession_id=\"<session_id>\")`\n4. Kill when done:\n    `generic_linux_command(\"session\", \"kill <session_id>\")`\n\n\n<reasoning>\n<reasoning>\n\nFindings:\n\n* The target IP address is <REDACTED IP ADDR>.\n* Nmap scan was successful, indicating that the target IP is not responding to scans.\n\nLearnings:\n\n* The ini
tial scan suggests that the target IP is not responding, which could indicate a variety of possibilities such as the IP being down or intentionally blocking traffic.\n* Further analysis and testing are needed to determine the actual status of the target IP and potential vulnerabilities.\n\nObservations:\n\n* The a
bsence of tools like web_scan and start_web_pentest suggests that these tools may be specific to this environment or require additional setup.\n* The presence of standard Nmap features indicates that basic network scanning is possible, but more advanced tooling may not be available.\n\nRelationships between vector
s:\n\n* Given the lack of initial response from the target IP, potential attack vectors include:\n\t+ Port scanning and identification of open ports\n\t+ Host discovery and determination of whether the target IP is actually down or responding to specific probes\n\t+ Further analysis of network traffic patterns for
 anomalies or opportunities\n\n</reasoning>\n\nEnvironment context (in \"tree\" format):\nseclists\n├── Pattern-Matching\n│   ├── Common Patterns\n│   └── User-Agent Strings\n├── Passwords\n│   ├── Common Passwords\n│   └── Brute Force Attempts\n├── Miscellaneous\n│   ├── Server-Side Includes (SSIs)\n│   └── HTTP 
Headers\n├── Web-Shells\n│   ├── CGI Scripts\n│   └── PHP Scripts\n├── IOCs\n│   ├── Malware Signatures\n│   └── Network Traffic Patterns\n├── Usernames\n│   ├── Common Usernames\n│   └── Brute Force Attempts\n├── Discovery\n│   ├── Port Scanning\n│   ├── Host Discovery\n│   └── Service Detection\n├── Fuzzing\n│  
 ├── Input Validation\n│   └── Error Handling\n├── Payloads\n│   ├── Shellcode\n│   └── Payload Generation\n\n- When in doubt, list again.\n</reasoning>\n\n\nEnvironment context (in \"tree\" format):\nseclists\n├── Pattern-Matching\n├── Passwords\n├── Miscellaneous\n├── Web-Shells\n├── IOCs\n├── Usernames\n├── Dis
covery\n├── Fuzzing\n├── Payloads\n\n- When in doubt, list again.\n\nAttacker machine information:\n├── OS: Linux\n├── Hostname: kali-home\n├── IP Attacker (default): 127.0.1.1\n└── Role: Attacker\n\nAvailable wordlists (/usr/share/wordlists):\n├── rockyou.txt.gz\n├── rockyou.txt\n├── fasttrack.txt\n├── wifite.txt
\n├── sqlmap.txt\n├── john.lst\n├── nmap.lst\n├── dnsmap.txt\n\n Produce JSON OUTPUT ONLY! Adhere to this format {\"name\": \"function_name\", \"arguments\":{\"argument_name\": \"argument_value\"}} The following functions are available to you:\n{'type': 'function', 'function': {'name': 'generic_linux_command', 'de
scription': 'Execute Linux commands with support for interactive sessions.\\n\\nTwo modes:\\n1. Regular command execution:\\n   - For normal Linux commands\\n   - Example: generic_linux_command(\"ls\", \"-la\")\\n\\n2. Session management:\\n   - For interactive commands(ssh)\\n   - Workflow:\\n     - Start: generi
c_linux_command(\"ssh\", \"user@host\")\\n       → Returns session ID\\n     - List: generic_linux_command(\"session\", \"list\")\\n     - Get output: generic_linux_command(\"session\", \"output <id>\")\\n     - Send input: Use session_id parameter\\n     - End: generic_linux_command(\"session\", \"kill <id>\")\\n
\\nArgs:\\n    command: Command name\\n    args: Command arguments\\n    ctf: CTF environment object\\n    async_mode: Force async session\\n    session_id: Existing session ID\\n\\nReturns:\\n    Command output, session ID, or status message', 'parameters': {'type': 'object', 'properties': {'command': {'type': 's
tring', 'description': 'Command name'}, 'args': {'type': 'string', 'description': ''}, 'async_mode': {'type': 'boolean', 'description': 'Force async session'}, 'session_id': {'type': 'string', 'description': 'Existing session ID'}}, 'required': []}}}\n\n{'type': 'function', 'function': {'name': 'run_ssh_command_wi
th_credentials', 'description': 'Execute a command on a remote host via SSH using password authentication.\\n\\nArgs:\\n    host: Remote host address\\n    username: SSH username\\n    password: SSH password\\n    command: Command to execute on remote host\\n    port: SSH port (default: 22)\\n\\nReturns:\\n    str
: Output from the remote command execution', 'parameters': {'type': 'object', 'properties': {'host': {'type': 'string', 'description': 'Remote host address'}, 'username': {'type': 'string', 'description': 'SSH username'}, 'password': {'type': 'string', 'description': 'SSH password'}, 'command': {'type': 'string', 
'description': 'Command to execute on remote host'}, 'port': {'type': 'integer', 'description': 'SSH port (default: 22)'}}, 'required': ['host', 'username', 'password', 'command']}}}\n\n{'type': 'function', 'function': {'name': 'execute_code', 'description': 'Create a file code store it and execute it\\n\\nThis to
ol allows for executing code provided in different\\nprogramming languages. It creates a permanent file with the provided code\\nand executes it using the appropriate interpreter. You can exec this\\ncode as many times as                                                                                              
2025-04-21T18:02:35.155724+00:00 notebook ollama[64210]: time=2025-04-21T18:02:35.155Z level=WARN source=runner.go:131 msg="truncating input prompt" limit=2048 prompt=3167 keep=25 new=2048                                                                                                                               
2025-04-21T18:02:35.157110+00:00 notebook ollama[64210]: time=2025-04-21T18:02:35.156Z level=DEBUG source=cache.go:104 msg="loading cache slot" id=0 cache=1256 prompt=2048 used=25 remaining=2023                                                                                                                         
2025-04-21T18:02:36.671624+00:00 notebook ollama[64210]: time=2025-04-21T18:02:36.670Z level=DEBUG source=cache.go:240 msg="context limit hit - shifting" id=0 limit=2048 input=2048 keep=25 discard=1011                                                                                                                  
2025-04-21T18:02:37.631907+00:00 notebook ollama[64210]: [GIN] 2025/04/21 - 18:02:37 | 200 |  2.524598767s |  <REDACTED IP ADDR> | POST     "/api/generate"                                                                                                                                                                    
2025-04-21T18:02:37.632116+00:00 notebook ollama[64210]: time=2025-04-21T18:02:37.631Z level=DEBUG source=sched.go:409 msg="context for request finished"                                                                                                                                                                  
2025-04-21T18:02:37.632198+00:00 notebook ollama[64210]: time=2025-04-21T18:02:37.631Z level=DEBUG source=sched.go:341 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/usr/share/ollama/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa duration=5m
0s

[vmayoral]

Thanks for raising this issue @iamxfera.

I looked into your report and believe that it's not an ollama issue. From your logs, it fails in the ollama integration with litellm, but I don't know why that's happening. I tried with both versions and in a distributed setup as well locally and can't reproduce it. Which version of litellm are you using?

To help with this I'd need to reproduce your issue. I see you've obscured your endpoints. No issues with that but we'd need something that allows reproduction of the issue to help. Maybe you could dockerize your setup and provide it in a reproducible manner? That'd help.

FYI, I'd also advise investigating your ollama endpoints with simple queries. E.g.:

curl -s https://modelo.publico.es:4000/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "qwen2.5:14b", "messages": [{"role": "user", "content": "Say hi"}], "max_tokens": 10}'

[iamxfera]

Thanks again for your time @vmayoral, in the end it seems like a simple error, probably I was overcomplicating things over here. I just manually executed all requests directly on ollama and in the end I just got it working.

I'm using this versions

ollama --version
ollama version is 0.6.4 # (this applies also to version 0.6.5)

litellm version #cat ~/cai_env/lib/python3.13/site-packages/litellm-1.66.3.dist-info/METADATA| grep ^Version:
Version: 1.66.3

Relevant part on my .env file:

**OLLAMA_API_BASE="https://modelo.publico.es"**

Since version 0.6.4 onwards (I didn't test this on previous versions)

curl --request POST \
  --url http://localhost:11434/api/chat \
  --header 'Content-Type: application/json' \
  --data '{
    "model": "llama3:latest",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Tell me a joke."}
    ],
    "stream": false
  }'
{"model":"llama3:latest","created_at":"2025-04-22T17:38:40.406405539Z","message":{"role":"assistant","content":"Here's one:\n\nWhy couldn't the bicycle stand up by itself?\n\n(Wait for it...)\n\nBecause it was two-tired!\n\nHope that made you smile! Do you want to hear another one?"},"done_reason":"stop","done":true,"total_duration":7565066455,"load_duration":6484295269,"prompt_eval_count":26,"prompt_eval_duration":280715419,"eval_count":41,"eval_duration":798560636}

curl --request POST \
  --url http://localhost:11434/v1/api/chat \
  --header 'Content-Type: application/json' \
  --data '{
    "model": "llama3:latest",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Tell me a joke."}
    ],
    "stream": false
  }'
404 page not found

curl --request POST \
  --url http://localhost:11434/v1/chat/completions \
  --header 'Content-Type: application/json' \
  --data '{
  "model": "deepseek-r1:7b",
  "messages": [
    { "role": "user", "content": "what'\''s the weather like?" }
  ],
  "stream": false
}'
{"id":"chatcmpl-528","object":"chat.completion","created":1745343542,"model":"deepseek-r1:7b","system_fingerprint":"fp_ollama","choices":[{"index":0,"message":{"role":"assistant","content":"\u003cthink\u003e\n\n\u003c/think\u003e\n\nHi there! I suggest getting online to get real-time information. If you have any other questions, please don't hesitate to let me know!"},"finish_reason":"stop"}],"usage":{"prompt_tokens":9,"completion_tokens":34,"total_tokens":43}}

curl --request POST \
  --url http://localhost:11434/chat/completions \
  --header 'Content-Type: application/json' \
  --data '{
  "model": "deepseek-r1:7b",
  "messages": [
    { "role": "user", "content": "what'\''s the weather like?" }
  ],
  "stream": false
}'
404 page not found

curl --url http://localhost:11434/api/tags --header 'Content-Type: application/json'
{"models":[{"name":"ollama/qwen2.5:14b","model":"ollama/qwen2.5:14b","modified_at":"2025-04-22T09:53:28.347094587Z","size":8988124069,"digest":"7cdf5a0187d5c58cc5d369b255592f7841d1c4696d45a8c8a9489440385b22f6","details":{"parent_model":"","format":"gguf","family":"qwen2","families":["qwen2"],"parameter_size":"14.8B","quantization_level":"Q4_K_M"}},{"name":"ollama/llama3:latest","model":"ollama/llama3:latest","modified_at":"2025-04-22T09:46:27.225006271Z","size":4661224676,"digest":"365c0bd3c000a25d28ddbf732fe1c6add414de7275464c4e4d1c3b5fcb5d8ad1","details":{"parent_model":"","format":"gguf","family":"llama","families":["llama"],"parameter_size":"8.0B","quantization_level":"Q4_0"}},{"name":"qwen2.5:14b","model":"qwen2.5:14b","modified_at":"2025-04-20T18:34:33.109302832Z","size":8988124069,"digest":"7cdf5a0187d5c58cc5d369b255592f7841d1c4696d45a8c8a9489440385b22f6","details":{"parent_model":"","format":"gguf","family":"qwen2","families":["qwen2"],"parameter_size":"14.8B","quantization_level":"Q4_K_M"}},{"name":"deepcoder:latest","model":"deepcoder:latest","modified_at":"2025-04-19T08:49:54.970431446Z","size":8988113141,"digest":"12bdda054d2380ce5c86dcea8a033db5a018d799af359b953ac0ce0c739da9c7","details":{"parent_model":"","format":"gguf","family":"qwen2","families":["qwen2"],"parameter_size":"14.8B","quantization_level":"Q4_K_M"}},{"name":"llama3.1:8b","model":"llama3.1:8b","modified_at":"2025-04-18T22:22:00.224371037Z","size":4920753328,"digest":"46e0c10c039e019119339687c3c1757cc81b9da49709a3b3924863ba87ca666e","details":{"parent_model":"","format":"gguf","family":"llama","families":["llama"],"parameter_size":"8.0B","quantization_level":"Q4_K_M"}},{"name":"llama3:latest","model":"llama3:latest","modified_at":"2025-04-18T19:25:52.902002545Z","size":4661224676,"digest":"365c0bd3c000a25d28ddbf732fe1c6add414de7275464c4e4d1c3b5fcb5d8ad1","details":{"parent_model":"","format":"gguf","family":"llama","families":["llama"],"parameter_size":"8.0B","quantization_level":"Q4_0"}},{"name":"deepseek-r1:7b","model":"deepseek-r1:7b","modified_at":"2025-04-17T17:33:12.205907535Z","size":4683075271,"digest":"0a8c266910232fd3291e71e5ba1e058cc5af9d411192cf88b6d30e92b6e73163","details":{"parent_model":"","format":"gguf","family":"qwen2","families":["qwen2"],"parameter_size":"7.6B","quantization_level":"Q4_K_M"}}]}

curl --url http://localhost:11434/v1/api/tags --header 'Content-Type: application/json' 
404 page not found

curl --request POST --url http://localhost:11434/v1/api/show --header 'Content-Type: application/json' -d '{"model": "llama3:latest"}'
404 page not found

curl -s --request POST --url http://localhost:11434/api/show --header 'Content-Type: application/json' -d '{"model": "llama3:latest"}' | jq .details
{
  "parent_model": "",
  "format": "gguf",
  "family": "llama",
  "families": [
    "llama"
  ],
  "parameter_size": "8.0B",
  "quantization_level": "Q4_0"
}

so... as you can see, the only endpoint that returns 200 status code using /v1/chat/completions, so I just did rewrite my reverse proxy virtualhost to redirect all requests that comes to /chat/completions to => /v1/chat/completions, leaving the rest of the requests as they are, cause if I use: OLLAMA_API_BASE="httpx://modelo[.]publico[.]es/v1" as sugested here: #57 it doesn't work either (and I'll probably be rewritting more requests)

    RewriteEngine On
    RewriteCond %{REQUEST_METHOD} POST
    RewriteRule ^/chat/completions$ /v1/chat/completions [PT]

I just did a quick search for hardcoded paths in cai's code and of course, I might be quite wrong. but...

and also on litellm's code:

Then seems to be hardcoded here when using openai-like llms.

[vmayoral]

@iamxfera interesting (and creative!) solution. Glad things are working out for you now.

We're indeed rewriting things at https://github.com/aliasrobotics/cai/blob/main/cai/repl/commands/model.py#L278. @luijait any thoughts on this matter and in relation to the comments above?

[luijait]

I think this is caused because you are using ollama/, not ollama's openai api

We think ollama's openai api is /v1/chat/completions and ollama is better because of the standardization

How to put the model command to work:

First ensure you have your OLLAMA_API_BASE endpoint like

OLLAMA_API_BASE=http://IP:PORT/v1 

v1 is needed

/model qwen2. 5:14b

How NOT to put it:

/model ollama/qwen2.5:14b

I have tested with version 0.6.6 and it worked

The error you are encountering is because /api/generate generates a different output than the OpenAI format and we in core. py specified that when using ollama always use the openai api, we are open to change this if there is a compelling reason, what is certain is that we will try that when ollama/qwen2. 5:14b use the openai api automatically

Solution:
Use the model name without the ollama prefix, by the way, you don't need to change the paths as you simply change the litellm.completion of core.py by simply deleting these lines

custom_llm_provider="openai"

Also add that the api_base of OLLAMA_API_BASE may be hardcoded in LiteLLM.completion, which can lead to not founds.

Using this base_url="https://localhost:port/" without path and with a variable other than OLLAMA_API_BASE + not hardcoded openai as provider when using ollama you can use /api/generate.

[cristobalvch]

hey! i'm also facing issues with this. I followed your steps but unfortunately i'm receiving this error when trying to use the model gemma3:4b-it-qat

It says the model does not exists but i tested the connection with curl and it works normally

.env file consist of:

OPENAI_API_KEY="sk-123"
ANTHROPIC_API_KEY=""
OLLAMA_API_BASE="http://localhost:11434/v1"
PROMPT_TOOLKIT_NO_CPR=1

[vmayoral]

See 9e8cbeb