can never get by: "localai" error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:43369: connect: connection refused\""

[leonbollerup]

Hey,

Help me out a bit here.. i'm i an 100% idiot or what is going on ?

--

LocalAI version:
LocalAI, Native, no docker

Environment, CPU architecture, OS, and Version:
VM on proxmox with 1070 TI passtrough and Mac M4, latest

Describe the bug
Clean install, load (any) model and get this:

"localai" error="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:43369: connect: connection refused""

No matter how i set it up (havent tested docker yet) - i always come back to the same error.

I load it up, i choose a model, i goto chat and type something.. and get this:

May 08 12:19:18 localai local-ai[1333]: 12:19PM INF Trying to load the model 'qwen3-32b' with the backend '[llama-cpp llama-cpp-fallback bark-cpp piper silero-vad stablediffusion-ggml whisper huggingface]'
May 08 12:19:18 localai local-ai[1333]: 12:19PM INF [llama-cpp] Attempting to load
May 08 12:19:18 localai local-ai[1333]: 12:19PM INF BackendLoader starting backend=llama-cpp modelID=qwen3-32b o.model=Qwen_Qwen3-32B-Q4_K_M.gguf
May 08 12:19:18 localai local-ai[1333]: 12:19PM INF [llama-cpp] attempting to load with CUDA variant
May 08 12:19:56 localai local-ai[1333]: 12:19PM ERR failed starting/connecting to the gRPC service error="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:43369: connect: connection refused""
May 08 12:19:58 localai local-ai[1333]: 12:19PM ERR [llama-cpp] Failed loading model, trying with fallback 'llama-cpp-fallback', error: failed to load model with internal loader: grpc service not ready
May 08 12:20:37 localai local-ai[1333]: 12:20PM INF [llama-cpp] Loads OK

Same hardware works without issues with GPT4All.

[leonbollerup]

did alot more testing using docker aio instead (cpu and gpu) - and it works with CPU (dreadfull slow logically)

With GPU it fails with similar or other errors..

so.. extremely cool project.. but seems very unstable..

[gmicroul]

3:19AM INF BackendLoader starting backend=stablediffusion-ggml modelID=sd-1.5-ggml o.model=stable-diffusion-v1-5-pruned-emaonly-Q4_0.gguf
3:20AM ERR failed starting/connecting to the gRPC service error="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:43477: connect: connection refused""
3:20AM ERR Server error error="failed to load model with internal loader: grpc service not ready" ip=192.168.2.202 latency=40.318460567s method=POST status=500 url=/v1/images/generations

[leonbollerup]

8:55PM INF Trying to load the model 'qwen2.5-coder-14b' with the backend '[llama-cpp llama-cpp-fallback whisper bark-cpp piper silero-vad stablediffusion-ggml huggingface /build/backend/python/exllama2/run.sh /build/backend/python/diffusers/run.sh /build/backend/python/kokoro/run.sh /build/backend/python/coqui/run.sh /build/backend/python/vllm/run.sh /build/backend/python/rerankers/run.sh /build/backend/python/bark/run.sh /build/backend/python/transformers/run.sh /build/backend/python/faster-whisper/run.sh]'
8:55PM INF [llama-cpp] Attempting to load
8:55PM INF BackendLoader starting backend=llama-cpp modelID=qwen2.5-coder-14b o.model=Qwen2.5-Coder-14B.Q4_K_M.gguf
8:55PM ERR [llama-cpp] Failed loading model, trying with fallback 'llama-cpp-fallback', error: failed to load model with internal loader: could not load model: rpc error: code = Canceled desc =

[leonbollerup]

9:01PM INF Trying to load the model 'deepseek-coder-v2-lite-instruct' with the backend '[llama-cpp llama-cpp-fallback piper silero-vad stablediffusion-ggml whisper bark-cpp huggingface /build/backend/python/rerankers/run.sh /build/backend/python/vllm/run.sh /build/backend/python/exllama2/run.sh /build/backend/python/transformers/run.sh /build/backend/python/diffusers/run.sh /build/backend/python/kokoro/run.sh /build/backend/python/bark/run.sh /build/backend/python/faster-whisper/run.sh /build/backend/python/coqui/run.sh]'
9:01PM INF [llama-cpp] Attempting to load
9:01PM INF BackendLoader starting backend=llama-cpp modelID=deepseek-coder-v2-lite-instruct o.model=DeepSeek-Coder-V2-Lite-Instruct-Q4_K_M.gguf
9:01PM INF Success ip=127.0.0.1 latency="27.166µs" method=GET status=200 url=/readyz
9:01PM ERR [llama-cpp] Failed loading model, trying with fallback 'llama-cpp-fallback', error: failed to load model with internal loader: could not load model: rpc error: code = Canceled desc =
9:01PM INF [llama-cpp] Fails: failed to load model with internal loader: could not load model: rpc error: code = Canceled desc =
9:01PM INF [llama-cpp-fallback] Attempting to load
9:01PM INF BackendLoader starting backend=llama-cpp-fallback modelID=deepseek-coder-v2-lite-instruct o.model=DeepSeek-Coder-V2-Lite-Instruct-Q4_K_M.gguf
9:01PM INF [llama-cpp-fallback] Fails: failed to load model with internal loader: could not load model: rpc error: code = Canceled desc =
9:01PM INF [piper] Attempting to load
9:01PM INF BackendLoader starting backend=piper modelID=deepseek-coder-v2-lite-instruct o.model=DeepSeek-Coder-V2-Lite-Instruct-Q4_K_M.gguf
9:01PM INF [piper] Fails: failed to load model with internal loader: could not load model: rpc error: code = Unknown desc = unsupported model type /build/models/DeepSeek-Coder-V2-Lite-Instruct-Q4_K_M.gguf (should end with .onnx)
9:01PM INF [silero-vad] Attempting to load
9:01PM INF BackendLoader starting backend=silero-vad modelID=deepseek-coder-v2-lite-instruct o.model=DeepSeek-Coder-V2-Lite-Instruct-Q4_K_M.gguf
9:01PM INF [silero-vad] Fails: failed to load model with internal loader: could not load model: rpc error: code = Unknown desc = create silero detector: failed to create session: Load model from /build/models/DeepSeek-Coder-V2-Lite-Instruct-Q4_K_M.gguf failed:Protobuf parsing failed.
9:01PM INF [stablediffusion-ggml] Attempting to load
9:01PM INF BackendLoader starting backend=stablediffusion-ggml modelID=deepseek-coder-v2-lite-instruct o.model=DeepSeek-Coder-V2-Lite-Instruct-Q4_K_M.gguf
9:01PM INF [stablediffusion-ggml] Fails: failed to load model with internal loader: could not load model: rpc error: code = Unknown desc = could not load model
9:01PM INF [whisper] Attempting to load
9:01PM INF BackendLoader starting backend=whisper modelID=deepseek-coder-v2-lite-instruct o.model=DeepSeek-Coder-V2-Lite-Instruct-Q4_K_M.gguf
9:01PM INF [whisper] Fails: failed to load model with internal loader: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF
9:01PM INF [bark-cpp] Attempting to load
9:01PM INF BackendLoader starting backend=bark-cpp modelID=deepseek-coder-v2-lite-instruct o.model=DeepSeek-Coder-V2-Lite-Instruct-Q4_K_M.gguf
9:01PM INF [bark-cpp] Fails: failed to load model with internal loader: could not load model: rpc error: code = Unknown desc = inference failed
9:01PM INF [huggingface] Attempting to load

[leonbollerup]

i'v tested like 20 different models and maybe 1+2 works... is something totally broken ?

[leonbollerup]

SOLVED - its GPU vram problem, if the model is to big it wont load..

Might i suggest that a real error or atleast a check is preformed!??!

[Fr0d0Beutl1n]

There was a check in the latest stable version, but apparently there is a regression in the master branch?

[leonbollerup]

Seems like it.. as a newcommer to LocalAI it would have saved me alot of time..

Maybe it would also be smart to actually re-format the information about a model to include more structured information about the model.. such as requirements.

A dream scenario would be to be able to filter on models based on requirements

[raokrutarth]

yup. Facing a similar error when trying to use text-to-image models:

localai  | 3:47AM ERR failed starting/connecting to the gRPC service error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:40529: connect: connection refused\""
localai  | 3:47AM DBG GRPC Service NOT ready
localai  | 3:47AM ERR Server error error="failed to load model with internal loader: grpc service not ready" ip=172.21.0.1 latency=40.037790276s method=POST status=500 url=/v1/images/generations

[maniek335]

I had the same problem on an M2 Mac after installing via Brew or running the downloaded binary from releases. I managed to solve the problem by installing protobuf via Brew.