Model providers should be community driven

[isidorn]

@ggerganov makes a good point here https://x.com/ggerganov/status/1925587681519579594

In short the Ollama option should not be called Ollama but "Custom endpoint"

So ideally:

• We finalize the LanguageModelProvider API in June
• We decide which of the built-in providers should not be built-in (Cerebras, Groq)
• We reach out to all the built-in providers that should not be built-in and give them a heads up that they can now publish an extension
• We change the "Ollama" one to be called "Custom endpoint"
• Make discovery of extension language model providers dynamic and easy Manage models needs to expose additional providers #251267

fyi @sbatten @pierceboggan @jrieken

[ngxson]

Looking forward to having this feature supported on vscode! We will try our best on our side - llama.cpp - to make the setup as friction-less as possible for end-users, cc @ggerganov @slaren

[ggerganov]

Note that the "Custom endpoint" should expect that the standard OpenAI APIs are being served (e.g. /v1/chat/completions). This could be configurable to support additional APIs, but the default should be the OAI API.

[SomeOddCodeGuy]

Adding in that making it a Chat Completions endpoint will not only make the endpoint nearly universal across all LLM applications, as the vast majority of local and proprietary hosting solutions expose such an endpoint, but it would also simplify any confusion around the sampler settings that the endpoints will accept.

In terms of usability, the community will definitely get more bang for the buck by adding compatibility to an industry standard schema like chat completions, rather than focusing on a proprietary schema used by only a small handful of applications.

[pierceboggan]

If I go to set BYOK provider, but don't see what I'm looking for, there should be a More Providers option that populates based on what we see from extensions, similar to what we have for our theme configuration that's populated from marketplace. Obviously not a P0, but would be good UX to have.

[insilications]

It is nice to support standard OpenAI APIs (e.g. the good old /v1/chat/completions). It guarantees wide support. But I wonder if there will be future native API support for OpenAI's Response API? (cc @lramos15 @jrieken) The Response API can increase tool calling performance by 6-10% and decrease token costs by increasing cache hit by up to 80% (claims from OpenAI's cookbook I can't find right now):

For the best results with this change, we recommend using the Responses API with the store parameter set to true, and passing in all reasoning items from previous requests (either using previous_response_id, or by taking all the output items from an older request and passing them in as input items for a new one). OpenAI will automatically include any relevant reasoning items in the model's context and ignore any irrelevant ones. In more advanced use‑cases where you’d like to manage what goes into the model's context more precisely, we recommend that you at least include all reasoning items between the latest function call and the previous user message. Doing this will ensure that the model doesn’t have to restart its reasoning when you respond to a function call, resulting in better function‑calling performance and lower overall token usage.
https://platform.openai.com/docs/guides/reasoning-best-practices#how-to-keep-costs-low-and-accuracy-high

https://platform.openai.com/docs/guides/responses-vs-chat-completions

[solomatov]

It would be great if it is possible to have AI features without setting up a copilot account. I.e. completely offline mode.