Use other models compatible with the OpenAI library

[CNFeffery]

Such as DeepSeek ( https://api-docs.deepseek.com/ ), :

By supporting the configuration of parameters such as api_key, base_url, and model, various large model services compatible with the OpenAI library call methods can be used.

[srdas]

Definitely worth looking into. For the moment note that several deepseek models are also available through Ollama: https://ollama.com/library/deepseek-llm.

You may use these models with Jupyter AI:

[CNFeffery]

Definitely worth looking into. For the moment note that several deepseek models are also available through Ollama: https://ollama.com/library/deepseek-llm. You may use these models with Jupyter AI:

Although DeepSeek V3 has released an open-source model, the model parameters are quite large, making self-deployment based on Ollama impractical. Typically, people use DeepSeek's official online API service.

[zgpnuaa]

Deepseek is comparable in capability to Claude and ChatGPT, but its price is much cheaper, offering great value for money.

[dlqqq]

Thank you all for the additional context! I think there is a way to bring Deepseek support into Jupyter AI, even though it lacks a LangChain partner package:

• Subclass ChatOpenAI like so:

class DeepseekProvider(BaseProvider, ChatOpenAI):
  def __init__(self, *args, **kwargs):
    super().__init__(*args, **kwargs)
    self.openai_api_base = "https://api.deepseek.com"

• Set auth_strategy = EnvAuthStrategy(name="DEEPSEEK_API_KEY") as a class attribute to give this provider its own unique API key to save to the config.

@srdas I've outlined some steps to try above. I'll assign this issue to you, so feel free to work on this issue if you have spare time.

[srdas]

I was working on adding the class DeepseekProvider as suggested above, and realized that it is already provided for via the OpenRouter provider, which was written to accommodate cases where langchain_openai is used with class ChatOpenAI. See: this code

The approach uses deepseek-chat in Jupyter AI using the OpenRouter provider in Jupyter AI as follows:

Simply enter the Deepseek API key in the OPENROUTER_API_KEY.

I tested to see that this works:

This enables the use of Deepseek models in Jupyter AI.

We can surely add a new DeepseekProvider class which would mimic the code in openrouter.py but this would be redundant? Though the UX would be better. Wondering what folks think?

[srdas]

Another approach would be to enhance langchain_openai explicitly with a new module deepseek.py here along the lines of azure.py.

[CNFeffery]

@srdas is it possible to add a general API Provider OpenAl Compatible like cline, to achieve compatibility with all large model interfaces that conform to the OpenAI calling method.

[dlqqq]

@CNFeffery Thank you for opening this issue. As @srdas indicated with his latest comments, we actually already have a provider which allows usage of any model with an API conforming to the OpenAI API specification. This is the OpenRouter provider, contributed by @akaihola in #996.

I will summarize the steps to take to use this with Deepseek:

1. Install langchain-openai and restart JupyterLab. This is necessary as it provides the SDK for accessing any OpenAI API.
2. Select the OpenRouter :: * model in the Jupyter AI settings. If you don't see this, please upgrade to the latest version of Jupyter AI v2.
3. Type in the API base URL & local model ID corresponding to the model you wish to use. For Deepseek, you should use https://api.deepseek.com as the API base URL, and use deepseek-chat as the local model ID.
4. Click "Save Changes" at the bottom to save your settings. You should now be able to use Deepseek! 🤗

@srdas has opened an issue to improve documentation on using OpenRouter to use any model providing an OpenAI API: #1190.

@CNFeffery I don't Cline is available for use through this, since Cline isn't an LLM by itself but rather an agent + set of agent tools powered by a Claude 3.5 Sonnet model. This is stated in their README.

(cc @zgpnuaa)

[dlqqq]

Closing as a solution exists for the user's original issue. Feel free to continue commenting here or open another issue as needed. 👍

[CNFeffery]

@dlqqq The openai compatible provider in in the settings of cline: