Getting started example not working

[wouterverduin]

This project got my interest and wanted to give it a shot. Was attempting the getting started docker example and ran into issues:

LocalAI version:
Latest image

Environment, CPU architecture, OS, and Version:
Running in an ubuntu 22.04 VM with docker. Got an Xeon E3 with 4 cores+8threads; All threads are available for this VM with 16gig ram.

Describe the bug/reproduce
Ran this command:
docker run -p 8888:8080 -ti --rm quay.io/go-skynet/local-ai:latest --models-path /home/wouter/docker/LocalAI/models --context-size 700 --threads 4

Was assuming this would get my container up and running but ran into bugs as seen below is the output of the files.
Also attempted to pre-download a model using this:
wget https://gpt4all.io/models/ggml-gpt4all-j.bin -O models/ggml-gpt4all-j

Expected behavior
A running container :)

Logs
`CMake Error at tests/CMakeLists.txt:145 (set_target_properties):
set_target_properties called with incorrect number of arguments.

-- Configuring incomplete, errors occurred!
See also "/build/go-ggml-transformers/build/CMakeFiles/CMakeOutput.log".
See also "/build/go-ggml-transformers/build/CMakeFiles/CMakeError.log".
make[1]: *** [Makefile:144: ggml.o] Error 1
make[1]: Leaving directory '/build/go-ggml-transformers'
make: *** [Makefile:181: go-ggml-transformers/libtransformers.a] Error 2`

Additional context
Hope somebody knows the answer!

[finger42]

Hello @wouterverduin,
I run maybe in similar problems when I started to play around with this docker containers.

Have you tried to add the env variable
-e "REBUILD=false"
this prevent the rebuilding of the application inside of the container.
Currently I am not sure why this is possible, but my expectation is to just pull and run the container without recompile
all the stuff.

Maybe you can test:
docker run -e "REBUILD=false" -p 8888:8080 -ti --rm quay.io/go-skynet/local-ai:latest --models-path /home/wouter/docker/LocalAI/models --context-size 700 --threads 4

Also you can try to run it with:
docker run -v models:/build/models -e "REBUILD=false" -p 8888:8080 -ti --rm quay.io/go-skynet/local-ai:latest

If you want to pass more options take a look at: https://github.com/go-skynet/LocalAI/blob/master/.env
there you can see also the comment on

## Uncomment and set to false to disable rebuilding from source
# REBUILD=false

Hope that helps

[wouterverduin]

@finger42

Thanks for the reply! I ran a docker command (havent triede a compose file yet) and is running now.

This is my output:

`Starting LocalAI using 4 threads, with models path: /home/wouter/docker/LocalAI/models
unexpected end of JSON input
7:34AM ERR error loading config files: open /home/wouter/docker/LocalAI/models: no such file or directory

┌───────────────────────────────────────────────────┐
│ Fiber v2.47.0 │
│ http://127.0.0.1:8080 │
│ (bound on host 0.0.0.0 and port 8080) │
│ │
│ Handlers ............ 31 Processes ........... 1 │
│ Prefork ....... Disabled PID ................. 6 │
└───────────────────────────────────────────────────┘
`

So i made progress. When i go to my link it the webrowser it shows:
{"error":{"code":404,"message":"Cannot GET /","type":""}}

But when i curl http://localhost:8888/v1/models it says:

{"error":{"code":500,"message":"open /home/wouter/docker/LocalAI/models: no such file or directory","type":""}}

The things is when i cd into '/home/wouter/docker/LocalAI/models' i have two files:

-rw-r--r-- 1 root root 3785248281 Apr 15 15:17 ggml-gpt4all-j -rw-r--r-- 1 root root 180 Jul 5 09:26 ggml-gpt4all-j.tmpl

Are these the wrong files? Or is there another reason it cant acces this folder?

Yet again; Any help would be greatly appreciated!

[vankeer]

@wouterverduin I haven't tried it without docker-compose, but it seems like your local models folder (/home/.../models) has to be mapped as a volume (/models), and then the models-path env variable should point to that volume ?

I'm having a similar problem getting the GPT4ALL-J model running locally using Docker on a 2021 MacBook M1 Pro.

First I noticed the unexpected end of JSON input error in the Docker logs, but that got fixed after uncommenting more env variables in .env (I think it was the GALLERIES that fixed it).

Now I'm having no errors:

$ docker-compose up --pull always
[+] Running 1/1
 ✔ api Pulled                                                                                                                                                                                  1.6s 
[+] Running 1/1
 ✔ Container localai-api-1  Created                                                                                                                                                            0.0s 
Attaching to localai-api-1
localai-api-1  | Starting LocalAI using 4 threads, with models path: /models
localai-api-1  | <nil>
localai-api-1  | 
localai-api-1  |  ┌───────────────────────────────────────────────────┐ 
localai-api-1  |  │                   Fiber v2.47.0                   │ 
localai-api-1  |  │               http://127.0.0.1:8080               │ 
localai-api-1  |  │                                                   │ 
localai-api-1  |  │ Handlers ............ 33  Processes ........... 1 │ 
localai-api-1  |  │ Prefork ....... Disabled  PID ................. 7 │ 
localai-api-1  |  └───────────────────────────────────────────────────┘ 
localai-api-1  | 
localai-api-1  | [127.0.0.1]:43294  200  -  GET      /readyz
localai-api-1  | [127.0.0.1]:39004  200  -  GET      /readyz
localai-api-1  | [127.0.0.1]:43076  200  -  GET      /readyz
localai-api-1  | [127.0.0.1]:34776  200  -  GET      /readyz
localai-api-1  | [127.0.0.1]:35052  200  -  GET      /readyz
localai-api-1  | [127.0.0.1]:54114  200  -  GET      /readyz

but, I get no response from the API:

$ curl http://127.0.0.1:8080/v1/models
curl: (52) Empty reply from server

Did anyone get the GPT4ALL-J example working using Docker?

[finger42]

-v models:/build/models
      ^             ^
      |               path inside of your container
      path in your host

The host path should be something like: /home/wouter/docker/LocalAI/models
The path inside of the container should be /build/models

I get also these errors at the start of the container:
unexpected end of JSON input
But I ignore it currently.

@wouterverduin Can you post your complete docker command + the output ?

My current way to spin it up is:

docker run -it --rm -v ~/LLM/models:/build/models:cached -p 8080:8080 -e "ADDRESS=0.0.0.0:8080" -e "DEBUG=true" -e "CORS=true"  -e "THREADS=8" -e "REBUILD=false"  quay.io/go-skynet/local-ai:v1.20.1

You can play arrount with the -e options

Starting LocalAI using 8 threads, with models path: /build/models
unexpected end of JSON input

 ┌───────────────────────────────────────────────────┐ 
 │                   Fiber v2.47.0                   │ 
 │               http://127.0.0.1:8080               │ 
 │       (bound on host 0.0.0.0 and port 8080)       │ 
 │                                                   │ 
 │ Handlers ............ 33  Processes ........... 1 │ 
 │ Prefork ....... Disabled  PID ................. 7 │ 
 └───────────────────────────────────────────────────┘ 
[172.17.0.1]:44416  200  -  GET      /v1/models

Also to get the list of loaded models:

curl http://localhost:8080/v1/models
{"object":"list","data":[{"id":"ggml-nous-gpt4-vicuna-13b.bin","object":"model"}]}

[vankeer]

For me the issue is fixed. I just restarted the container with the default env vars and waited quite a bit longer, then suddenly the API started responding, with no change in the logs.

[Aisuko]

Thank you guys' feedback.

The binary inside the image is rebuilt at the start of the container to enable CPU optimizations for the execution environment, you can set the environment variable REBUILD to false to prevent this behavior.

And about the model path @finger42 already given a clear answer.

[localai-bot]

⚠️⚠️⚠️⚠️⚠️

Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

but.... I can also be funny or helpful 😸 and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don't engage in conversation with me,
I don't support (yet) replying!

⚠️⚠️⚠️⚠️⚠️

I am sorry but I do not know how to resolve this issue. The error message indicates that there is an issue with CMake configuration, specifically that there was an incorrect number of arguments passed to set_target_properties. Additionally, it appears that there is an issue building the ggml-transformers library, with an error occurring when trying to build the ggml.o file and another error when creating the libtransformers.a file. This could potentially be due to missing or incorrect dependencies. Without more information about the environment and any other steps taken, it is difficult to determine the cause of this issue. I suggest checking the logs for more information on the errors and possibly reviewing the installation guide and relevant documentation.

Sources:

• https://github.com/go-skynet/LocalAI/tree/master/examples/langchain-python/docker-compose.yaml
• https://github.com/go-skynet/LocalAI/tree/master/examples/rwkv/docker-compose.yaml
• https://localai.io/basics/getting_started/index.html
• https://github.com/go-skynet/LocalAI/tree/master/examples/langchain/docker-compose.yaml

[gitwittidbit]

Because the whole thing does not work for me in docker, I am now trying to build it locally.

However, I am running into the same error as OP (just outside of docker, on the host machine).

Cmake throws a ton of warnings before it stops with

`CMake Error at tests/CMakeLists.txt:145 (set_target_properties):
set_target_properties called with incorrect number of arguments.'

Any ideas?

[gitwittidbit]

Okay, I figured it out.

Mind you, OP and I stumbled across

CMake Error at tests/CMakeLists.txt:145 (set_target_properties): set_target_properties called with incorrect number of arguments.

Which happens when you rebuild local-ai (either in docker or outside) and which is set as the default behaviour.

The solution presented here so far is to avoid rebuilding local-ai. Unfortunately, that didn't work for me as I needed to rebuild local-ai without avx2 support (which is active per default).

So, without further ado, the solution:

There is a set of double quotes missing in go-ggllm/ggllm.cpp/CMakeLists.txt in line 145:

It reads in the repo
set(LLAMA_EXTRA_LIBS ${LLAMA_EXTRA_LIBS} ${ACCELERATE_FRAMEWORK})

but should read
set(LLAMA_EXTRA_LIBS "${LLAMA_EXTRA_LIBS} ${ACCELERATE_FRAMEWORK}")

With this change, for me the build process goes through without the error.

Not being a software developer, unfortunately, I don't know yet how to make a PR to correct this myself. I am going to learn that but I thought it would be better to provide the solution faster. @mudler will have this fixed in no time whereas I would probably need a couple of days to set up an IDE, git, etc. just to make the PR.

Thanks @mudler for this wonderful project! I really admire what you have created here.

[Aisuko]

Hi, @gitwittidbit , thanks for your feedback. I wonder if this line would work if you do not rebuild the project? Here, go-ggllm/ggllm.cpp is an bounding project, and it was loaded by make command. It should not have this kind of issue.

[gitwittidbit]

Hi, @gitwittidbit , thanks for your feedback. I wonder if this line would work if you do not rebuild the project? Here, go-ggllm/ggllm.cpp is an bounding project, and it was loaded by make command. It should not have this kind of issue.

Yes, if you don't rebuild the project, an error in the source/build code won't be a problem. For me it was a problem because I needed to rebuild the project.

And if you do rebuild it and the build process stop with the error

CMake Error at tests/CMakeLists.txt:145 (set_target_properties): set_target_properties called with incorrect number of arguments.

the you can fix this by correcting line 145 in CMakeLists.txt.

[mudler]

Okay, I figured it out.

Mind you, OP and I stumbled across

CMake Error at tests/CMakeLists.txt:145 (set_target_properties): set_target_properties called with incorrect number of arguments.

Which happens when you rebuild local-ai (either in docker or outside) and which is set as the default behaviour.

The solution presented here so far is to avoid rebuilding local-ai. Unfortunately, that didn't work for me as I needed to rebuild local-ai without avx2 support (which is active per default).

So, without further ado, the solution:

There is a set of double quotes missing in go-ggllm/ggllm.cpp/CMakeLists.txt in line 145:

It reads in the repo set(LLAMA_EXTRA_LIBS ${LLAMA_EXTRA_LIBS} ${ACCELERATE_FRAMEWORK})

but should read set(LLAMA_EXTRA_LIBS "${LLAMA_EXTRA_LIBS} ${ACCELERATE_FRAMEWORK}")

With this change, for me the build process goes through without the error.

Not being a software developer, unfortunately, I don't know yet how to make a PR to correct this myself. I am going to learn that but I thought it would be better to provide the solution faster. @mudler will have this fixed in no time whereas I would probably need a couple of days to set up an IDE, git, etc. just to make the PR.

Thanks @mudler for this wonderful project! I really admire what you have created here.

I'll try to have a look soon - however you can skip building the gglm backend if you don't need it like this:

make BUILD_TYPE=... GRPC_BACKENDS="backend-assets/grpc/langchain-huggingface backend-assets/grpc/falcon-ggml backend-assets/grpc/bert-embeddings backend-assets/grpc/bloomz backend-assets/grpc/llama backend-assets/grpc/gpt4all backend-assets/grpc/dolly backend-assets/grpc/gpt2 backend-assets/grpc/gptj backend-assets/grpc/gptneox backend-assets/grpc/mpt backend-assets/grpc/replit backend-assets/grpc/starcoder backend-assets/grpc/rwkv backend-assets/grpc/whisper" build