Possible to add support for OpenAI compatible third party providers?

[peixotorms]

Hi, is it possible to add support for OpenAI compatible third party providers?
Some providers like together.ai for example, return OpenAI compatible replies, so I would like to use some models from them like LLama 3 70B, Mixtral 8x22B, Qwen 2, WizardLM... or whatever.

I don't really do rails, but since the format is the same as OpenAI API, I suppose it should be possible to add some feature, to allow adding a new service, as long as it follows the openai format. Ideally, we would choose the endpoint, api key, model name, temperature, etc and a name, then it would be added to the sidebar, like GPT and Claude.

If it's not too time consuming, please do, as that would be a game changer.

That would also help #366 and possibly other requests, although not directly, but rather via any compatible service provider with OpenAI API compatibility (hence an option to choose the endpoint).

Apparently it's in the works, so if it's going to have the option to add only specific models like I described it above, then you have my thanks.

[krschacht]

Hi @peixotorms we’re almost done with that! @stephan-buckmaster has been taking the lead on it. We merged in the first half last week and he has the second half tee’d up in a PR that we’ve been going back and forth on. I’m going to dig into it again tomorrow so it should be very soon.

[krschacht]

@peixotorms if you want to take a look it’s #389 . The plan is that you’ll be able to create a new backend (arbitrary URL and port, so it could be remote like Groq or local if you’re running a Llamafile or something. Then you specify the “driver” as OpenAI and assuming it follows the same API spec it should “just work”. We haven’t tested extensively yet so it’ll be great if you can try it out after we merge in.

I’d love to hear more about your interest in this project? What brought you to it? And what’s motivating you to want to run these alternate models?

[peixotorms]

Hi,

It sounds like you are saying that we could just replace the endpoint used by the existing OpenAI option, and it would just work... but would we still be able to use OpenAI and Claude, or would that replacement redirect all traffic through the driver and OpenAI and Claude stop working (as in, it would use the driver instead of the official api)?

What I would like to see, is the ability to create additional provider/models on the sidebar (although maybe for multiple models, a dropdown is needed there), while preserving the ability to continue using OpenAI and Claude.
https://share.raisercdn.com/raul/MeFuHevI67.png

Running those alternative models in my case, is mostly for testing during development, but some of them are actually quite good and faster at certain types of tasks, and at a lower cost too.

For example, I am developing an app in PHP, that let's the user select different providers and api models to generate profile descriptions for academic authors, institutions, etc. With a few million rows, we might need to use different cheaper alternative models for example, which would then need testing.

I can always use the playground or command line, but having the possibility to just add whatever model we want and chat with it as a chatgpt style, is much better. Furthermore, if I need to share with someone that is not a dev, that makes things easier.

As OpenAI GPT-4o is getting to my nerves for coding with so many error, wrong answers, incapable of detecting even basic things like wrong syntax for example, (GPT-4 is still fine for now) I been looking for alternatives. Claude so far fits the bill for me, but I still would like the freedom to use other models.

Thanks

[krschacht]

Just in the past week I’ve been feeling like GPT-4o was giving me worse answers than I used to get from GPT-4 a month ago. It’s funny that you mention that!

You’re about to get everything you want. In fact, I would love to have you try out the branch. Stephen did a lot of work on it, I did some reviewing of it today, and from my limited testing it’s working well. But if you try it out you may be able to let us know if there are any issues with it:

It’s PR #389. Pull the branch locally and then:

• Go to your profile (bottom left) and then settings
• Create a new API. This is where you enter a URL, port, and a key if you need one. For example, you might put the details for Groq here
• Create a new language model (e.g. Llama 3) and indicate that this new Groq API is the backend for this. In this case the “driver” would be OpenAI since it conforms to that spec
• Then create a new assistant, called anything, which uses the Llama 3 language model you just created.

It’ll appear in the sidebar just like you want and will leave all the others untouched.

[stephan-buckmaster]

Thanks for your feedback @peixotorms -- I have the same impression, this stuff can be useful even if for super-quick try-outs. Here, one could set up an API Service which is stable. One sets up a Language Model pointing to that API Service. Then an Assistant for that Language Model. Where it gets quick is when one just changes the name of the Language Model, and everything else stays the same. You can try out your new model by switching the name you choose on the server.

In terms of other backend protocols, I'm thinking this can be accomplished by allowing classes under the AIBackend namespace as drivers for APIService records. But for now, we're starting with just Anthropic, and OpenAI.

When I use it, Ollama is already compatible with OpenAI. I find the discussions don' t get the title assigned, but that seems marginal, in terms of of rate of change in this area in general.

[peixotorms]

I would rather wait for stable version so I can just run it on render, because at the moment I am a bit busy. Perhaps next week, I will be able to download #389 and try doing some tests. Thanks!