Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: return streaming errors as an event formatted for openai's client #2668

Merged

Conversation

drbh
Copy link
Collaborator

@drbh drbh commented Oct 18, 2024

This PR adds the functionality to emit errors as formatted events.

Note the changes look large because of a new indent however the main logic that changed is the addition of

Err(err) => {
    let error_event: ErrorEvent = err.into();
    let event = Event::default().json_data(error_event).unwrap_or_else(|e| {
        InferError::StreamSerializationError(e.to_string()).into()
    });
    yield Ok::<Event, Infallible>(event);
    break;
}

this change enables the openai client to pull the error message from the stream and correctly surface it to the user

example script (throws with change included)

import openai
from openai import OpenAI

client = OpenAI(
    api_key="key",
)

client.base_url = "http://127.0.0.1:3000/v1"

try:
    completion = client.chat.completions.create(
        messages=[
            {
                "role": "user",
                "content": "test " * 32_000,
            }
        ],
        model="tgi",
        stream=True,
    )

    for index, message in enumerate(completion):
        print(f"Message {index}: {message}")
except openai.APIError as e:
    print(e)

@Narsil Narsil force-pushed the return-streaming-error-in-openai-client-compatible-format branch from 2993191 to d8f1203 Compare November 15, 2024 13:48
@Narsil Narsil merged commit 6489f85 into main Nov 15, 2024
7 of 8 checks passed
@Narsil Narsil deleted the return-streaming-error-in-openai-client-compatible-format branch November 15, 2024 13:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants