✨ feat(llm): support Ollama AI Provider (local llm)

[sjy]

💻 变更类型 | Change Type

• ✨ feat support local llm via ollama (openai compatible)

🔀 变更说明 | Description of Change

📝 补充信息 | Additional Information

Refs #1283

• Need to wait for the ollma PR merged this https://github.com/ollama/ollama/pull/2376/files
• Which open source models are listed under Ollama by default ( currently is Mistral and Llama2 and Qwen)
• Add visible control via server config (env var control)
• Add form field so that user can select customized model (leverage the capability of Ollama)

[vercel]

Someone is attempting to deploy a commit to the LobeHub Team on Vercel.

A member of the Team first needs to authorize it.

[lobehubbot]

👍 @sjy

Thank you for raising your pull request and contributing to our Community
Please make sure you have followed our contributing guidelines. We will review it as soon as possible.
If you encounter any problems, please feel free to connect with us.
非常感谢您提出拉取请求并为我们的社区做出贡献，请确保您已经遵循了我们的贡献指南，我们会尽快审查它。
如果您遇到任何问题，请随时与我们联系。

[arvinxx]

@sjy A great PR! 👏👏

I think there is two issue we should discuss:

1. Ollama only support local LLM, it means that we shouldn't display it when user deploy with Docker/Vercel or other platform.So I think we should hide Ollama setttings when there is no ENV of Ollama. Or we need to implement this feature: [RFC] 024 - 浏览器端请求方式改造 #1257 , call LLM from browser.
2. I find that ollama has it's own dev kit: ollama-js. why not use this?

[arvinxx]

I have tried with my MacBook Pro, the speed is really amazing. cc @cocobeach

Ollama.mp4

[sjy]

@sjy A great PR! 👏👏

I think there is two issue we should discuss:

1. Ollama only support local LLM, it means that we shouldn't display it when user deploy with Docker/Vercel or other platform.So I think we should hide Ollama setttings when there is no ENV of Ollama. Or we need to implement this feature: [RFC] 024 - 浏览器端请求方式改造 #1257 , call LLM from browser.
2. I find that ollama has it's own dev kit: ollama-js. why not use this?

1. A really good point. I will add some visible logic then. For the 1257 feat, I think I need more context
2. We can use ths js sdk either. But the latest version introduced the opai compatible feature and when looking around the existing implemention of existing none-openai modle providers are also implemented with the openai js sdk, I dont know if we need to follow some guiline when add provider which can implemented by multiple ways just like such case here. BTW. the curent ollama api are only partial compatible with openai api, bit the js dev kit is announced containing all features

[cocobeach]

I have tried with my MacBook Pro, the speed is really amazing. cc @cocobeach

Ollama.mp4

That is truly impressive, can't wait to test it, I only have a i9 8 cores and 32gb or ram so I doubt it will run that fast, I can run Solar reasonably well, I find the open source models less "lazy" than gpt4 that seems unbothered to write code without placeholders. What spec is your MacBook, it is indeed mighty fast?
Well done thanks!!! 🙏

[arvinxx]

For the 1257 feat, I think I need more context

I think this PR doesn't need to consider #1257, I will do it later. This PR just need to focus on the Ollama Provider.

Beside the visible control, I think there should be another field for the custome models in Ollama. It may be the same as OpenAI's custom models. So then users can add their own local models.

As for the ollama sdk, I prefer your current implement after investigating the ollama js sdk, it current don't support browser by now. So we can't use it when working on #1257.
Besides, the all LLM providers are working on the way to be compatible with OpenAI's API schema. Beside Ollama, there are Zhipu AI（智谱)、 Moonshot Ai(月之暗面) and so on. So we can support them very fast.

Therefore, I think it's better to just use OpenAI sdk.

[arvinxx]

What spec is your MacBook, it is indeed mighty fast?

My Macbook Pro is a M1 Max 16", 64GB RAM.

[cocobeach]

What spec is your MacBook, it is indeed mighty fast?

My Macbook Pro is a M1 Max 16", 64GB RAM.

It's is impressive in speed, the x86 is a bit slower.
Well done!
Thanks for implementing Ollama so quickly! 😀😀😀

[tisfeng]

Looking forward to this new feature to be able to use Ollama local models.

[sjy]

💻 变更类型 | Change Type

• ✨ feat support local llm via ollama (openai compatible)
• 🐛 fix
• ♻️ refactor
• 💄 style
• 🔨 chore
• ⚡️ perf
• 📝 docs

🔀 变更说明 | Description of Change

Support Ollama as an AI Model Provider

• close [Request] Local LLM SUPPORT #1281
• close [Request] local models #1048
• close [Request] LLM 适配 #151

📝 补充信息 | Additional Information

Refs #1283

• Need to wait for the ollma PR merged this https://github.com/ollama/ollama/pull/2376/files
• Update i18n translation
• Which open source models are listed under Ollama by default ( currently is Mistral and Llama2, add Qwen to the default list)

[sjy]

For the 1257 feat, I think I need more context

I think this PR doesn't need to consider #1257, I will do it later. This PR just need to focus on the Ollama Provider.

Beside the visible control, I think there should be another field for the custome models in Ollama. It may be the same as OpenAI's custom models. So then users can add their own local models.

As for the ollama sdk, I prefer your current implement after investigating the ollama js sdk, it current don't support browser by now. So we can't use it when working on #1257. Besides, the all LLM providers are working on the way to be compatible with OpenAI's API schema. Beside Ollama, there are Zhipu AI（智谱)、 Moonshot Ai(月之暗面) and so on. So we can support them very fast.

Therefore, I think it's better to just use OpenAI sdk.

@arvinxx, updated, BTW, Qwen model is added to the default list when enable Ollam, please help review the changes ~

[arvinxx]

@sjy It's great 👍. And let me take over the rest task~

[lobehubbot]

❤️ Great PR @sjy ❤️

The growth of project is inseparable from user feedback and contribution, thanks for your contribution! If you are interesting with the lobehub developer community, please join our discord and then dm @arvinxx or @canisminor1990. They will invite you to our private developer channel. We are talking about the lobe-chat development or sharing ai newsletter around the world.
项目的成长离不开用户反馈和贡献，感谢您的贡献! 如果您对 LobeHub 开发者社区感兴趣，请加入我们的 discord，然后私信 @arvinxx 或 @canisminor1990。他们会邀请您加入我们的私密开发者频道。我们将会讨论关于 Lobe Chat 的开发，分享和讨论全球范围内的 AI 消息。