Support native macOS (Apple Silicon) execution without Docker

[davidfowl]

Right now, running Ollama via Docker on macOS doesn’t use the GPU because Docker can’t access Apple Silicon GPUs. That’s a pretty big performance hit, especially when working with larger models.

This blog post lays it out clearly: https://chariotsolutions.com/blog/post/apple-silicon-gpus-docker-and-ollama-pick-two/

When you run Ollama natively, it detects and uses the Metal backend:

level=INFO msg="Dynamic LLM libraries [metal]"

But in Docker:

level=INFO msg="no GPU detected"

It’d be great if Ollama could run natively on macOS without Docker, so we can take advantage of Apple’s GPU acceleration. Performance and dev experience would be much better.

[aaronpowell]

Would you expect it to download/install Ollama if it's not already installed?

Can you specify the port of the Ollama host via the CLI? That'd be important for ensuring you don't have port conflicts on startup.

A current workaround would be to use the ConnectionStringResource and provide the endpoint for Ollama run from outside of Aspire.

[davidfowl]

Ran into this again, feel like we should do this.

Would you expect it to download/install Ollama if it's not already installed?

It wouldn't. It would fail like node / python apps do when you don't have them installed.

Can you specify the port of the Ollama host via the CLI? That'd be important for ensuring you don't have port conflicts on startup.

Should be no different from containers. We can choose random ports (this is what happens when you use node as well).

A current workaround would be to use the ConnectionStringResource and provide the endpoint for Ollama run from outside of Aspire.

Yes this works but model downloading is one of the nice things the resouce does.

[github-actions]

We have noticed this issue has not been updated within 21 days. If there is no action on this issue in the next 14 days, we will automatically close it. You can use /stale-extend to extend the window.

[JudeVajira]

/stale-extend

[github-actions]

We have noticed this issue has not been updated within 21 days. If there is no action on this issue in the next 14 days, we will automatically close it. You can use /stale-extend to extend the window.