Discussion Piece: Allow env based openAI compatibilty.

[jasperblues]

Problem Statement

Long tail LLM providers have a unique value proposition that gives them a portion of the market without being mainstream. For this reason, rather than offer their own API they use OpenAI as the lingua di franco compatibility layer.

Embabel handles this well, allowing registration of a bean. But what about:

• Pre-packaged agents (eg in docker hub) where users want to use an openAI compatible LLM, but don't have access to the code.
• Other situations where it makes sense to have a purely properties based approach.

Proposed Solution

Most of the pieces are in place already to offer this. However there is a registry of possible openAI models to use. We need to be able to plug in the long-tail vendor's model on the fly, so Embabel can hook it up.

We can do that something like (in this pr):

export OPENAI_BASE_URL="https://api.groq.com/openai/"
export OPENAI_API_KEY="GROQ_KEY"
export OPENAI_CUSTOM_MODELS="llama-3.3-70b-versatile,mixtral-8x7b-32768,openai/gpt-oss-120b"
export EMBABEL_MODELS_DEFAULT_LLM="llama-3.3-70b-versatile"

I used this approach to spin up the hub powered by models provided by Groq. (Not the flashiest of models, but Groq lets them run blisteringly fast).

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[alexheifetz]

@jasperblues the idea is very good and useful. As far as implementation, i think it would be better to introduce dedicated Autoconfiguraiton and corresponding Starter modules.

Something like embabel-agent-openaicompatible-autoconfigure and keep existing openai module dedicated to OpenAi Provider only.

That would give better separation of concerns (OpenAi vs OpenAiCompatible)

[alexheifetz]

Custom Model for OpenAI has been added via separate autoconfiguration and starter.

See https://github.com/embabel/embabel-agent/tree/main/embabel-agent-starters/embabel-agent-starter-openai-custom

@jasperblues please give it a try, as a side note all variables contain CUSTOM as part of it name. For example:

export OPENAI_CUSTOM_BASE_URL="https://api.groq.com/openai/"
export OPENAI_CUSTOM_API_KEY="GROQ_KEY"