CodeGate gets stuck on looping

[rdimitrov]

Describe the issue

The issue is that while running the Ollama integration tests locally in a loop (script provided), either CodeGate gets stuck in a loop or the curl is left hanging for response.

This was not reproducible when using larger models (1.5B or 7B).

This does not cause a denial of service to CodeGate.

Steps to Reproduce

1. Run ollama locally
2. Pull the following model - ollama pull qwen2.5-coder:0.5b
3. Start CodeGate
4. Create/activate a venv with the necessary dependencies installed
5. Run the following script -

#!/bin/bash

FAIL_COUNT=0
TOTAL_RUNS=60

for ((i=1; i<=TOTAL_RUNS; i++)); do
    echo "Run #$i"
    CODEGATE_PROVIDERS=ollama python tests/integration/integration_tests.py
    if [ $? -ne 0 ]; then
        ((FAIL_COUNT++))
    fi
    sleep 2
done

echo "Total failures: $FAIL_COUNT out of $TOTAL_RUNS runs."

Operating System

MacOS (Arm)

IDE and Version

None, bash script doing curls

Extension and Version

None, bash script doing curls

Provider

Ollama

Model

qwen2.5-coder:0.5b

Codegate version

Build from main - Commit 7eb57d6

Logs

1. Logs from when the curl is stuck -

• Test:

2025-02-06 14:44:38 [debug    ] Generated embedding            model=all-minilm-L6-v2-q5_k_m.gguf vector_length=384
2025-02-06 14:44:38 [debug    ] Generating embedding           content=["```python\nif __name__ == '__main__':\n    invokehttp.run(call_api)\n```\n"] content_length=70 model=all-minilm-L6-v2-q5_k_m.gguf
2025-02-06 14:44:38 [debug    ] Generated embedding            model=all-minilm-L6-v2-q5_k_m.gguf vector_length=384
2025-02-06 14:44:38 [info     ] Test Ollama FIM passed        
2025-02-06 14:44:38 [debug    ] Creating requester for provider: ollama
2025-02-06 14:44:38 [debug    ] Using requester type: StandardRequester
2025-02-06 14:44:38 [debug    ] Making request to URL: http://127.0.0.1:8989/ollama/chat/completions
2025-02-06 14:44:38 [debug    ] Headers: {'Content-Type': 'application/json'}
2025-02-06 14:44:38 [debug    ] Data: {'max_tokens': 4096, 'messages': [{'content': 'You are a coding assistant.', 'role': 'system'}, {'content': 'Is it safe to use invokehttp?', 'role': 'user'}], 'model': 'qwen2.5-coder:0.5b', 'stream': True, 'temperature': 0}

• Codegate:

2025-02-06T12:44:37.588443Z [debug    ] send_request_body.complete     lineno=87 module=_trace pathname=/Users/dimitrovr/stacklok/codegate/venv/lib/python3.13/site-packages/httpcore/_trace.py
2025-02-06T12:44:37.588578Z [debug    ] receive_response_headers.started request=<Request [b'POST']> lineno=87 module=_trace pathname=/Users/dimitrovr/stacklok/codegate/venv/lib/python3.13/site-packages/httpcore/_trace.py
2025-02-06T12:44:38.313037Z [debug    ] receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Content-Type', b'application/x-ndjson'), (b'Date', b'Thu, 06 Feb 2025 12:44:38 GMT'), (b'Transfer-Encoding', b'chunked')]) lineno=87 module=_trace pathname=/Users/dimitrovr/stacklok/codegate/venv/lib/python3.13/site-packages/httpcore/_trace.py
2025-02-06T12:44:38.313919Z [info     ] HTTP Request: POST http://localhost:11434/api/generate "HTTP/1.1 200 OK" lineno=1786 module=_client pathname=/Users/dimitrovr/stacklok/codegate/venv/lib/python3.13/site-packages/httpx/_client.py
2025-02-06T12:44:38.314582Z [debug    ] receive_response_body.started request=<Request [b'POST']> lineno=87 module=_trace pathname=/Users/dimitrovr/stacklok/codegate/venv/lib/python3.13/site-packages/httpcore/_trace.py
2025-02-06T12:44:38.423121Z [debug    ] receive_response_body.complete lineno=87 module=_trace pathname=/Users/dimitrovr/stacklok/codegate/venv/lib/python3.13/site-packages/httpcore/_trace.py
2025-02-06T12:44:38.423533Z [debug    ] response_closed.started        lineno=87 module=_trace pathname=/Users/dimitrovr/stacklok/codegate/venv/lib/python3.13/site-packages/httpcore/_trace.py
2025-02-06T12:44:38.423923Z [debug    ] response_closed.complete       lineno=87 module=_trace pathname=/Users/dimitrovr/stacklok/codegate/venv/lib/python3.13/site-packages/httpcore/_trace.py
2025-02-06T12:44:38.424442Z [debug    ] No secrets found for session   lineno=115 module=manager pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/pipeline/secrets/manager.py session_id=8f280000-0cd3-4e92-ace6-b7656a1ce6e5
2025-02-06T12:44:38.703034Z [debug    ] User-Agent header received: python-requests/2.32.3 from 127.0.0.1 lineno=56 module=server pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/server.py
2025-02-06T12:44:38.704185Z [info     ] No particilar client detected, using generic client lineno=224 module=detector pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/clients/detector.py
2025-02-06T12:44:38.705078Z [info     ] Chat completion pipeline selected for execution. lineno=158 module=base pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/providers/base.py
2025-02-06T12:44:38.705528Z [info     ] Total secrets redacted since last assistant message: 0 lineno=298 module=secrets pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/pipeline/secrets/secrets.py
2025-02-06T12:44:38.705708Z [debug    ] Generating embedding           content=['Is it safe to use invokehttp?'] content_length=29 lineno=80 model=all-minilm-L6-v2-q5_k_m.gguf module=inference_engine pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/inference/inference_engine.py
2025-02-06T12:44:38.708466Z [debug    ] Generated embedding            lineno=90 model=all-minilm-L6-v2-q5_k_m.gguf module=inference_engine pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/inference/inference_engine.py vector_length=384
2025-02-06T12:44:38.708640Z [debug    ] Performing vector similarity search distance_threshold=0.5 limit=100 lineno=204 module=storage_engine pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/storage/storage_engine.py query='Is it safe to use invokehttp?'
2025-02-06T12:44:38.708896Z [info     ] Adding 1 bad packages to the context. lineno=115 module=codegate pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/pipeline/codegate_context_retriever/codegate.py
2025-02-06T12:44:38.709067Z [debug    ] Found matching packages in sqlite-vec database lineno=52 matched_packages=['invokehttp (pypi)'] module=codegate pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/pipeline/codegate_context_retriever/codegate.py
2025-02-06T12:44:38.709181Z [debug    ] Final context message          context_message='Context: invokehttp is a Python package available on PyPI ecosystem.  However, this package is found to be malicious and must not be used. For additional information refer to https://www.insight.stacklok.com/report/pypi/invokehttp?utm_source=codegate - Package offers this functionality: Invokehttp is a malicious package\n \n\n Query: Is it safe to use invokehttp?' lineno=157 module=codegate pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/pipeline/codegate_context_retriever/codegate.py
2025-02-06T12:44:38.713017Z [info     ] Chat completion pipeline selected for output. lineno=112 module=base pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/providers/base.py
2025-02-06T12:44:38.714148Z [info     ] 127.0.0.1:61870 - "POST /ollama/chat/completions HTTP/1.1" 200 lineno=473 module=h11_impl pathname=/Users/dimitrovr/stacklok/codegate/venv/lib/python3.13/site-packages/uvicorn/protocols/http/h11_impl.py
2025-02-06T12:44:38.714620Z [debug    ] send_request_headers.started request=<Request [b'POST']> lineno=87 module=_trace pathname=/Users/dimitrovr/stacklok/codegate/venv/lib/python3.13/site-packages/httpcore/_trace.py
2025-02-06T12:44:38.714999Z [debug    ] send_request_headers.complete  lineno=87 module=_trace pathname=/Users/dimitrovr/stacklok/codegate/venv/lib/python3.13/site-packages/httpcore/_trace.py
2025-02-06T12:44:38.715093Z [debug    ] send_request_body.started request=<Request [b'POST']> lineno=87 module=_trace pathname=/Users/dimitrovr/stacklok/codegate/venv/lib/python3.13/site-packages/httpcore/_trace.py
2025-02-06T12:44:38.715367Z [debug    ] send_request_body.complete     lineno=87 module=_trace pathname=/Users/dimitrovr/stacklok/codegate/venv/lib/python3.13/site-packages/httpcore/_trace.py
2025-02-06T12:44:38.715451Z [debug    ] receive_response_headers.started request=<Request [b'POST']> lineno=87 module=_trace pathname=/Users/dimitrovr/stacklok/codegate/venv/lib/python3.13/site-packages/httpcore/_trace.py
2025-02-06T12:44:39.624093Z [debug    ] receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Content-Type', b'application/x-ndjson'), (b'Date', b'Thu, 06 Feb 2025 12:44:39 GMT'), (b'Transfer-Encoding', b'chunked')]) lineno=87 module=_trace pathname=/Users/dimitrovr/stacklok/codegate/venv/lib/python3.13/site-packages/httpcore/_trace.py
2025-02-06T12:44:39.625773Z [info     ] HTTP Request: POST http://localhost:11434/api/chat "HTTP/1.1 200 OK" lineno=1786 module=_client pathname=/Users/dimitrovr/stacklok/codegate/venv/lib/python3.13/site-packages/httpx/_client.py
2025-02-06T12:44:39.626668Z [debug    ] receive_response_body.started request=<Request [b'POST']> lineno=87 module=_trace pathname=/Users/dimitrovr/stacklok/codegate/venv/lib/python3.13/site-packages/httpcore/_trace.py

2. Logs from when CodeGate is stuck

• Test:

2025-02-06 14:49:40 [debug    ] Generated embedding            model=all-minilm-L6-v2-q5_k_m.gguf vector_length=384
2025-02-06 14:49:40 [info     ] Test Ollama FIM passed        
2025-02-06 14:49:40 [debug    ] Creating requester for provider: ollama
2025-02-06 14:49:40 [debug    ] Using requester type: StandardRequester
2025-02-06 14:49:40 [debug    ] Making request to URL: http://127.0.0.1:8989/ollama/chat/completions
2025-02-06 14:49:40 [debug    ] Headers: {'Content-Type': 'application/json'}
2025-02-06 14:49:40 [debug    ] Data: {'max_tokens': 4096, 'messages': [{'content': 'You are a coding assistant.', 'role': 'system'}, {'content': 'Is it safe to use invokehttp?', 'role': 'user'}], 'model': 'qwen2.5-coder:0.5b', 'stream': True, 'temperature': 0}

• CodeGate

...
2025-02-06T12:50:25.417862Z [info     ] Found 0 libraries in the storage engine lineno=58 module=output pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/pipeline/extract_snippets/output.py
2025-02-06T12:50:26.199536Z [debug    ] Searching by package names and ecosystem ecosystem=pypi lineno=161 module=storage_engine packages=['flask'] params=['flask', 'pypi'] pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/storage/storage_engine.py sql='\n                    SELECT name, type, status, description\n                    FROM packages\n                    WHERE LOWER(name) IN (?)\n                    AND LOWER(type) = ?\n                '
2025-02-06T12:50:26.199793Z [info     ] Found 0 libraries in the storage engine lineno=58 module=output pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/pipeline/extract_snippets/output.py
2025-02-06T12:50:26.990139Z [debug    ] Searching by package names and ecosystem ecosystem=pypi lineno=161 module=storage_engine packages=['flask'] params=['flask', 'pypi'] pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/storage/storage_engine.py sql='\n                    SELECT name, type, status, description\n                    FROM packages\n                    WHERE LOWER(name) IN (?)\n                    AND LOWER(type) = ?\n                '
2025-02-06T12:50:26.990398Z [info     ] Found 0 libraries in the storage engine lineno=58 module=output pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/pipeline/extract_snippets/output.py
2025-02-06T12:50:27.738928Z [debug    ] Searching by package names and ecosystem ecosystem=pypi lineno=161 module=storage_engine packages=['flask'] params=['flask', 'pypi'] pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/storage/storage_engine.py sql='\n                    SELECT name, type, status, description\n                    FROM packages\n                    WHERE LOWER(name) IN (?)\n                    AND LOWER(type) = ?\n                '
2025-02-06T12:50:27.739193Z [info     ] Found 0 libraries in the storage engine lineno=58 module=output pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/pipeline/extract_snippets/output.py
2025-02-06T12:50:28.386815Z [debug    ] Searching by package names and ecosystem ecosystem=pypi lineno=161 module=storage_engine packages=['flask'] params=['flask', 'pypi'] pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/storage/storage_engine.py sql='\n                    SELECT name, type, status, description\n                    FROM packages\n                    WHERE LOWER(name) IN (?)\n                    AND LOWER(type) = ?\n                '
2025-02-06T12:50:28.387055Z [info     ] Found 0 libraries in the storage engine lineno=58 module=output pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/pipeline/extract_snippets/output.py
2025-02-06T12:50:29.059227Z [debug    ] Searching by package names and ecosystem ecosystem=pypi lineno=161 module=storage_engine packages=['flask'] params=['flask', 'pypi'] pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/storage/storage_engine.py sql='\n                    SELECT name, type, status, description\n                    FROM packages\n                    WHERE LOWER(name) IN (?)\n                    AND LOWER(type) = ?\n                '
2025-02-06T12:50:29.059469Z [info     ] Found 0 libraries in the storage engine lineno=58 module=output pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/pipeline/extract_snippets/output.py
2025-02-06T12:50:29.756715Z [debug    ] Searching by package names and ecosystem ecosystem=pypi lineno=161 module=storage_engine packages=['flask'] params=['flask', 'pypi'] pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/storage/storage_engine.py sql='\n                    SELECT name, type, status, description\n                    FROM packages\n                    WHERE LOWER(name) IN (?)\n                    AND LOWER(type) = ?\n                '
2025-02-06T12:50:29.756984Z [info     ] Found 0 libraries in the storage engine lineno=58 module=output pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/pipeline/extract_snippets/output.py
2025-02-06T12:50:30.457167Z [debug    ] Searching by package names and ecosystem ecosystem=pypi lineno=161 module=storage_engine packages=['flask'] params=['flask', 'pypi'] pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/storage/storage_engine.py sql='\n                    SELECT name, type, status, description\n                    FROM packages\n                    WHERE LOWER(name) IN (?)\n                    AND LOWER(type) = ?\n                '
2025-02-06T12:50:30.457502Z [info     ] Found 0 libraries in the storage engine lineno=58 module=output pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/pipeline/extract_snippets/output.py
2025-02-06T12:50:31.165392Z [debug    ] Searching by package names and ecosystem ecosystem=pypi lineno=161 module=storage_engine packages=['flask'] params=['flask', 'pypi'] pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/storage/storage_engine.py sql='\n                    SELECT name, type, status, description\n                    FROM packages\n                    WHERE LOWER(name) IN (?)\n                    AND LOWER(type) = ?\n                '
2025-02-06T12:50:31.165673Z [info     ] Found 0 libraries in the storage engine lineno=58 module=output pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/pipeline/extract_snippets/output.py
2025-02-06T12:50:31.896884Z [debug    ] Searching by package names and ecosystem ecosystem=pypi lineno=161 module=storage_engine packages=['flask'] params=['flask', 'pypi'] pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/storage/storage_engine.py sql='\n                    SELECT name, type, status, description\n                    FROM packages\n                    WHERE LOWER(name) IN (?)\n                    AND LOWER(type) = ?\n                '
2025-02-06T12:50:31.897134Z [info     ] Found 0 libraries in the storage engine lineno=58 module=output pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/pipeline/extract_snippets/output.py
2025-02-06T12:50:32.651781Z [debug    ] Searching by package names and ecosystem ecosystem=pypi lineno=161 module=storage_engine packages=['flask'] params=['flask', 'pypi'] pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/storage/storage_engine.py sql='\n                    SELECT name, type, status, description\n                    FROM packages\n                    WHERE LOWER(name) IN (?)\n                    AND LOWER(type) = ?\n                '
2025-02-06T12:50:32.652048Z [info     ] Found 0 libraries in the storage engine lineno=58 module=output pathname=/Users/dimitrovr/stacklok/codegate/src/codegate/pipeline/extract_snippets/output.py
...

Additional Context

No response

[rdimitrov]

After debugging this it turns out that this issue was limited to Ollama (I was not able to reproduce this with the other providers I've tested it - llamacpp, openai, copilot, anthropic).

The issue is that the Ollama Server is sometimes hanging and when that happens CodeGate had a 5min timeout which was not resulting in a good fail fast behavior. The solution is to reduce the timeout to 30s.

[rdimitrov]

After further debugging the 2nd issue described here where CodeGate is stuck looping seems to be a bug with Ollama.

The reasons for this are:

• In process_stream() CodeGate keeps receiving chunks from that stream without any finish_reason being set to stop which is what causes the endless loop.
• I've tried the same use case with other providers (openai, copilot, anthropic, llamacpp) and I was not able to reproduce it.
• Re-setting my local environment affects how often I can reproduce the issue which again makes me feel it's some performance issue with Ollama.

That said I think it's alright to close the issue and move forward.