Assertion Error with OpenAI API Server on Linux #75
Description
The following tests run with no issue on my Mac (M2). However, an assertion error occurs running these same API calls on a Linux Ubuntu 22.04 box (using CPU only and with a GTX 3090 GPU).
Test - Download https://huggingface.co/jartine/mistral-7b.llamafile/blob/main/mistral-7b-instruct-v0.1-Q4_K_M-server.llamafile
# Run Server
chmod +x mistral-7b-instruct-v0.1-Q4_K_M-server.llamafile
./mistral-7b-instruct-v0.1-Q4_K_M-server.llamafile
# Run API Test
curl -i http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer no-key" \
-d '{
"model": "gpt-3.5-turbo",
"messages": [
{
"role": "system",
"content": "You are ChatGPT, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests."
},
{
"role": "user",
"content": "Write a limerick about python exceptions"
}
]
}'
Error
loading weights...
{"timestamp":1702100777,"level":"INFO","function":"main","line":3039,"message":"HTTP server listening","hostname":"127.0.0.1","port":8080}
all slots are idle and system prompt is empty, clear the KV cache
llama.cpp/server/json.h:21313: assert(it != m_value.object->end()) failed (cosmoaddr2line /data/ai/llamafile/mistral-7b 4247a4 42ba67 42ce5b 45da
Ref: #24