Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(chatgpt): Resolve async response issue with Azure GPT-4o API #1772

Merged
merged 2 commits into from
Aug 5, 2024

Conversation

JxQg
Copy link
Contributor

@JxQg JxQg commented Aug 2, 2024

Description

Fixed an issue where the Azure GPT-4o API's asynchronous responses contained empty choices arrays, leading to incorrect handling of empty data.

How Has This Been Tested?

This is my configuration.

LOCAL_DB_HOST=mysql
LOCAL_DB_PASSWORD=******
ALLOWLISTED_PLUGINS=db_dashboard
LLM_MODEL=proxyllm
PROXY_API_KEY=*
PROXY_API_BASE=https://*.openai.azure.com/
PROXY_API_TYPE=azure
PROXY_SERVER_URL=https://*.openai.azure.com/openai/deployments/gpt-4o/chat/completions?api-version=2024-02-15-preview
PROXY_API_VERSION=2024-02-15-preview
PROXYLLM_BACKEND=gpt-4o

When I successfully deploy with docker or a local conda environment, no matter what questions are raised on the front end, the output will be disconnected at the end and an exception will be thrown.As follows:

Snipaste_2024-08-02_17-35-25

Logs:

 messages:
[{'role': 'system', 'content': '你是一个有用的 AI 助手。'}, {'role': 'user', 'content': '测试正常AzureOpenAPI响应'}]
llm_adapter: <OpenAIProxyLLMModelAdapter model_name=proxyllm model_path=chatgpt_proxyllm>

model prompt: 

system: 你是一个有用的 AI 助手。
human: 测试正常AzureOpenAPI响应

async stream output:

INFO:     172.19.0.1:58758 - "POST /api/v1/chat/mode/params/list?chat_mode=chat_normal HTTP/1.1" 200 OK
INFO:     172.19.0.1:58774 - "GET /api/v1/feedback/select HTTP/1.1" 200 OK
2024-08-02 09:34:57 a71ebe4fcaf1 dbgpt.model.cluster.worker.default_worker[1] INFO is_first_generate, usage: None
要测试 Azure OpenAPI 的正常响应,你可以按照以下步骤进行:

1. **创建 Azure 资源**:确保你有一个 Azure 订阅,并创建一个 API Management 服务或 Azure Function App 来托管你的 API。

2. **创建 OpenAPI 定义**:编写或生成你的 OpenAPI (Swagger) 定义文档。你可以使用 Swagger Editor 或其他工具来创建这个文档。

3. **部署 API**:将你的 API 部署到 Azure 上。你可以使用 Azure 门户、Azure CLI 或 Azure DevOps 等工具来进行部署。

4. **获取 API 的 URL**:部署完成后,你会得到 API 的 URL,这个 URL 将用于测试。

5. **测试 API**:使用工具如 Postman、cURL 或者编写一个简单的脚本来发送请求到你的 API,并检查响应。

下面是一个使用 cURL 测试 API 的简单示例:


curl -X GET "https://your-api-url/endpoint" -H "accept: application/json"


如果你想使用 Postman,请按照以下步骤:

1. 打开 Postman。
2. 创建一个新的请求。
3. 设置请求方法(如 GET、POST 等)和 URL。
4. 添加必要的头信息(如 `accept: application/json`)。
5. 点击“Send”按钮发送请求。
6. 检查响应内容。

### 示例 OpenAPI 定义
这是一个简单的 OpenAPI 定义示例:


openapi: 3.0.0
info:
  title: Sample API
  description: A sample API to illustrate OpenAPI concepts
  version: 1.0.0
paths:
  /hello:
    get:
      summary: Returns a greeting message
      responses:
        '200':
          description: A successful response
2024-08-02 09:35:09 a71ebe4fcaf1 dbgpt.model.cluster.worker.default_worker[1] ERROR Model inference error, detail: Traceback (most recent call last):
  File "/app/dbgpt/model/cluster/worker/default_worker.py", line 248, in async_generate_stream
    async for output in generate_stream_func(
  File "/app/dbgpt/model/proxy/llms/chatgpt.py", line 43, in chatgpt_generate_stream
    async for r in client.generate_stream(request):
  File "/app/dbgpt/model/proxy/llms/chatgpt.py", line 230, in generate_stream
    async for r in self.generate_stream_v1(messages, payload):
  File "/app/dbgpt/model/proxy/llms/chatgpt.py", line 265, in generate_stream_v1
    if r.choices[0].delta.content is not None:
AttributeError: 'NoneType' object has no attribute 'content'

Snapshots:

Include snapshots for easier review.

Checklist:

  • My code follows the style guidelines of this project
  • I have already rebased the commits and make the commit message conform to the project standard.
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • Any dependent changes have been merged and published in downstream modules

Fixed an issue where the Azure GPT-4o API's asynchronous responses contained empty choices arrays, leading to incorrect handling of empty data.
@github-actions github-actions bot added the fix Bug fixes label Aug 2, 2024
@csunny
Copy link
Collaborator

csunny commented Aug 2, 2024

@JxQg Well, it's a very useful PR. But could you please change the comments to en?

Updated comments in the async response handling code to English for better readability and understanding. Ensures the empty 'choices' array in Azure GPT-4o API responses is correctly handled.
@JxQg
Copy link
Contributor Author

JxQg commented Aug 2, 2024

@csunny I'm so sry for comment format that does not conform to the code specification. I have modified it and push it .

Copy link
Collaborator

@csunny csunny left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

r+

Copy link
Collaborator

@fangyinc fangyinc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@fangyinc fangyinc merged commit 5bd946f into eosphoros-ai:main Aug 5, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fix Bug fixes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants