[Issue]: How to add additional data params to the completion API - Autogenstudio

[satnair]

Describe the issue

I am able to host and load the local vllm endpoint. The issue is in adding additional data payload params to /chat/completions endpoint, I need to send few additional params .

I used the Autogen studio UI to set the api endpoint, api key etc. But there are no options to add additional data payload.

Steps to reproduce

For example, I will need to send the data payload like this to the vllm endpoint. How to configure this in autopen studio model endpoint.

===Example
import requests

-API endpoint
url = "http://localhost:8001/v1/chat/completions"

-Headers for the request
headers = {"Content-Type": "application/json"}

-Data payload
data = {
"messages": [{"content": "list of songs in 1900"}],
"use_context": True, # Set to True if needed
"context_filter": None, # Provide context filter if required
"include_sources": False, # Set to True to include sources
"stream": False # Set to True for streaming
}

-Making the POST request
response = requests.post(url, headers=headers, json=data)

==========end of example======

Please suggest ways of how it can be done in autogen studio. Any custom code can be added to support this.

Screenshots and logs

No response

Additional Information

No response

[rohlay]

Check out:
openai_compatible_server
examples/offline_inference_chat

One solution is creating a compatible "OpenAI Chat Completion Client". Some of those args, such as 'stream' should go in the API.
Find the chat template of your model at: https://github.com/vllm-project/vllm/tree/main/examples. I think that will take care of the headers for the request.
The other args you can configure when you run vllm serve, which for a compatible OpenAI API you should use python3 -m vllm.entrypoints.openai.api_server.

For example (add args here):
python3 -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-3.2-1B --host 0.0.0.0 --port 8000" --api-key token-abc123

Check the documentation, some args could go in the terminal command line, others may go in the get model API.

This might help:

from openai import OpenAI

# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "token-abc123"
openai_api_base = "http://localhost:4000/v1"

# Create OpenAI client
try: 
    client = OpenAI(
        # defaults to os.environ.get("OPENAI_API_KEY")
        api_key=openai_api_key,
        base_url=openai_api_base,
        stream=True,
    )
    print("Client created successfully")
except Exception as e:
    print("Error:", e)

try:
    models = client.models.list()
    model = models.data[0].id
    print(models)
    print()
    print(model)
except Exception as e:
    print("Error:", e)


with open('tool_chat_template_llama3.2_json.jinja', "r") as f:
    chat_template = f.read()


# Test chat completion
try:
    completion = client.chat.completions.create(
        model="meta-llama/Llama-3.2-1B",
        messages=[
            {"role": "user", "content": "write a haiku about AI"}
        ],
        extra_body={
        "chat_template": chat_template,
    }
    )
    print(completion.choices[0].message.content)
    print("Completion Result:", completion)
except Exception as e:
    print("Error:", e)

[rysweet]

please feel free to try the above solution or try on autogen 0.4.5 or later and reopen if needed