Integrate opensource LLMs into autogen

[LeoLjl]

Currently, autogen.oai only supports OpenAI models. I am planning to integrate open source LLMs into autogen. This would require Hugging Face transformers library.

Some of the open source LLMs I have in mind include LLaMa, Alpaca, Falcon, Vicuna, WizardLM, Starcoder, Guanaco. Most of their inference codes are integrated into Hugging Face transformers library, which can be unified into something like this.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "<name-of-model>"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16)
inputs = tokenizer("The world is", return_tensors="pt")
sample = model.generate(**inputs, max_length=128)
print(tokenizer.decode(sample[0]))

Tasks

Preview Give feedback

1. Could you also consider adding example on how to add custom API's? I think this approach might be generalizable to other kind of API's, such as search API's. FLAML#1017
   +0
   
2. The possibility of integrating other LLMs into Auto Generation FLAML#1037
   enhancement +1
3. Issues with Adding Custom APIs in Auto Generation #15
   feature good first issue +2

[LeoLjl]

The current goal is to support open-source LLMs in LLM-based agents with as little modification as possible.

The core code that needs modification is L41 in autogen/agent/assistant_agent.py. It only supports making completions using OpenAI models.

        responses = oai.ChatCompletion.create(messages=self._conversations[sender.name], **self._config)

There are two ways to do so. One is to create a utility function that checks the model name and call the regarding inference APIs.

openai_models = ["gpt-4", "gpt-3"]
open_source_models = ["llama", "alpaca"]
google_models = ["bard"]

def create(model, **kwargs):
    if model in openai_models:
        # generate response using  "oai.ChatCompletion.create"
        pass
    elif model in open_source_models:
        # generate response using related Hugging Face APIs
        pass
    elif model in google_models:
        # generate response using related Google APIs
        pass

This is quick to implement and easy to use. However it may introduce some unsolicited problems when developing other applications.

Another is to inherit the autogen.oai.Completion class and rewrite the methods.

class OpensourceCompletion(autogen.oai.Completion):
    @classmethod
    def create(**kwargs):
        # Make completion using related APIs
        pass

This method is not consistent with folder name oai.

I am researching what other libraries are handling with this situation and find potential solutions, e.g. LangChain, JARVIS.

[sonichi]

The naming consistency is not an issue because we can interpret it. For example, oai offers an inference API compatible with openai. It doesn't prevent supporting non-openai models using the same inference API.

[sonichi]

Related issue: microsoft/FLAML#1037 #15 microsoft/FLAML#1017

[sonichi]

related: FastChat/docs/openai_api.md at main · lm-sys/FastChat · GitHub

[LeoLjl]

FastChat is a useful tool as a local drop-in replacement for OpenAI APIs. This greatly simplifies integration.
ToDo:

• Update docs on how to spin up endpoints using FastChat.
• Write an automated code to start OpenAI-compatible API server.

[sonichi]

Thanks for creating a task list. Could you edit the "tasks" in the main body of the issue? There you can track sub issues. For example, the two items you listed here correspond to two existing issues microsoft/FLAML#1017 and microsoft/FLAML#1037. You can comment on those existing issues about the approach you'll take to solve them. Also add #15 to the task list.

[ishaan-jaff]

Hi @sonichi @LeoLjl I’m the maintainer of LiteLLM (abstraction to call 100+ LLMs)- we allow you to create a proxy server to call 100+ LLMs, and I think it can solve your problem (I'd love your feedback if it does not)

Try it here: https://docs.litellm.ai/docs/proxy_server
https://github.com/BerriAI/litellm

Using LiteLLM Proxy Server

import openai
openai.api_base = "http://0.0.0.0:8000/" # proxy url
print(openai.ChatCompletion.create(model="test", messages=[{"role":"user", "content":"Hey!"}]))

Creating a proxy server

Ollama models

$ litellm --model ollama/llama2 --api_base http://localhost:11434

Hugging Face Models

$ export HUGGINGFACE_API_KEY=my-api-key #[OPTIONAL]
$ litellm --model claude-instant-1

Anthropic

$ export ANTHROPIC_API_KEY=my-api-key
$ litellm --model claude-instant-1

Palm

$ export PALM_API_KEY=my-palm-key
$ litellm --model palm/chat-bison

[yhyu13]

@LeoLjl There are projects like textgen webui that allows openai api adapter to local llms.
And there are proejcts like one-api that servers as middleware for a lot of different remote llm backends

All of them simply redirect the openai api without introducing new code or modifying code

[LeoLjl]

Hi @yhyu13, thanks for the valuable information. This is really helpful. I will include it in the next docs update!

[LeoLjl]

Hi @ishaan-jaff, liteLLM looks like just the solution we need. I will try it out and include it in the new docs update!

[YaswanthDasamandam]

Hi ,
can someone answer my question #89 .
I am overriding the create class to generate the answers .

[ishaan-jaff]

@TheCompAce I can help you on our discord: https://discord.com/invite/wuPM9dRgDw