Skip to content

Add simple ollama chat example #336

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 9, 2025
Merged

Add simple ollama chat example #336

merged 1 commit into from
Mar 9, 2025

Conversation

matthewhaynesonline
Copy link
Contributor

Fixes #191.

I used the code examples from #173 as a starting point and tried to simplify them to match the existing chat example.

@matthewhaynesonline matthewhaynesonline changed the title #191: Add simple ollama chat example Add simple ollama chat example Feb 17, 2025
@matthewhaynesonline
Copy link
Contributor Author

matthewhaynesonline commented Feb 24, 2025

Hi @64bit, I've updated my PR with your feedback, though the Docker setup has some caveats I didn't have elegant solutions for.

  1. Build context: since Docker restricts where you can copy files from to the current directory branch (parents / siblings are not allowed), I adjusted the example to pull the async_openai crate from crates.io, not the parent dir.
    1. The only way I could think to use the crate from the parent project dir was to place the docker-compose.yml / Dockerfile in the project root, but that would no longer make the example self contained.
  2. First time Ollama model pull: Ollama requires you to pull a model before you can use it and, understandably, the Ollama docker image doesn't come preloaded with any models. As such, a custom entrypoint is use to pull the model on startup. A healthcheck is then used to wait to start the Rust container until after the Ollama container is fully ready.

What do you think?

@64bit
Copy link
Owner

64bit commented Mar 3, 2025

Thanks for update, I just tried it and looks good!

I think Dockerfile and ollam_chat containers in docker-compose are not required. My initial reason for docker-compose was to bring up Ollama and then cargo run example which would point to "localhost:ollama-port".

Your entrypoint stuff to download model looks good to me.

So user has to do two steps:

# bring ollama up with model
docker compose up -d

# once model is downloaded and ollama is up
cargo run 

@matthewhaynesonline
Copy link
Contributor Author

@64bit Sounds good! I updated my branch with your feedback to run the Rust code natively and rebased / cleaned up the commit history off latest upstream main.

Copy link
Owner

@64bit 64bit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good, thank your for your contributions!

@64bit 64bit merged commit 75a7c7a into 64bit:main Mar 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Ollama AI Chat Completion Example / OpenAI Compatibility
2 participants