Long response time , gpu support?

[drmetro09]

Hi , im new to this i have installed this pdf chat bot by docker compose on unraid , but its not using gpu and taking 15 mins to 20mis to generate response any help and guidance on this ? I want to ask if this supports gpu nividia

[amithkoujalgi]

Hello @drmetro09. I have added a docker-compose setup for enabling GPU support for Ollama. Give that a try.

If you're having a Nvidia GPU and you've installed the correct drivers, you should be able to use the GPU with the docker-compose setup from this repo.

PS: If you're setting up and running the setup for the very first time, you will need to wait for the model to download and the embedding model to load. Once these steps are done, the responses should be pretty fast.

Also, I've added benchmarks in the readme file for reference.

Happy exploring!

[drmetro09]

Hi , thanks for quick response but when gpu support is enabled it gives cublas error 15 and the bot doesn’t not give any response i have sent you a request on insta could you please guide me with this am new to this

[drmetro09]

[amithkoujalgi]

The error is originating from the Ollama server module. Do you have the correct Nvidia drivers installed? If you have, could you share some screenshots for the same? And maybe share the details of your execution environment to get a better understanding about the issue.

You could have a look at the Ollama docker setup documentation to see if it helps you.

There seems to be a similar issue reported in Ollama repo. See if you can correlate your issue to that.

Also,

i have sent you a request on insta

I didn't quite understand this. :)

[drmetro09]

Have tried various ollama models like orca-mini , llama2, starling-lm all are behaving same

[drmetro09]

Update: Its finally working now , i downgraded from to v 0.1.10 and everything working fine its using the gpu without errors, tried using LLAMA2 model haven’t tried other models.
Thanks for your amazing work , i have few suggestions listed below , hope you would consider them for this project.

1. Increase size of pdf to be uploaded to more than 200mb maybe 600mb .
2. Option to include docx ,ppt , CSV , jpeg , jpg and other formats which are commonly used.
3. Option to give the bot an icon/Avatar of our choice .
4. Instead of rebuilding the vector database again when the same pdf is uploaded , the bot should have ability to store the previous database and use it again subsequently . This would save time and makes the output fast.
5. Ability to download and select various ollama models from the web UI of pdf bot
6. using the bot for general chat besides the docs QnA.
   ( There is an amazing repo Private GPT for inspiration which satisfies the above points but its very complex to install and run from a perspective of non-IT guy )

[deepakitkar]

hi @drmetro09 could you please help me with setting this up. I have installed Docker-Desktop and have pulled the repository. I see this error in the log: requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=11434): Max retries exceeded with url: /api/pull (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd87933c070>: Failed to establish a new connection: [Errno 111] Connection refused'))
I checked the ollama is running on localhost:11434.
Under bindmounts not sure why the Network Settings has different local ip and gateway?
Any help?

[drmetro09]

Use the local ip like 192.168.0. whatever you have:11434 or use host.docker.internal:11434 in ollama configuration in docker compose file of pdf bot

[amithkoujalgi]

Great to hear that you were able to get it working! I personally have tried different models and found Llama2 to be the best suited for my needs as the responses were more apt.

About the feature/enhancement suggestions:

• I started this project just as an exploration work, but ended up creating a functional chatbot for PDFs. :)
• I'd love to take up these enhancements but I do not have enough bandwidth to work on these at the moment.
• I'd be happy to have collaborators for the repo who can contribute and build more features.

Cheers!