fix: set index of tool calls in stream mode #1468

hustxiayang · 2025-10-30T21:02:40Z

Description

There is an index field in the tool calls definition in stream mode: https://github.com/openai/openai-python/blob/4e8856576211064b09c0cc4a1ed35b82b169abe2/src/openai/types/chat/chat_completion_chunk.py#L48, and it's not optional. Third party libs might implement based on this assumption, for example: https://github.com/open-telemetry/opentelemetry-python-contrib/blob/447aac2b0fbe88998e970941c1160147f355ec73/instrumentation-genai/opentelemetry-instrumentation-openai-v2/src/opentelemetry/instrumentation/openai_v2/patch.py#L285. Internally, this is not set and may break codes. Thus, set it as 0 to make it compatible with OpenAI API.

Also, in the previous implementation, ChatCompletionMessageToolCallParam was reused in the response, and has an index field, which is not correct: https://github.com/openai/openai-python/blob/4e8856576211064b09c0cc4a1ed35b82b169abe2/src/openai/types/chat/chat_completion_message_function_tool_call_param.py#L23

Discuss/suggestion:

It seems that many definitions are reused. I think it's a very bad practice and very hard to maintain/debug. Would suggest to have separate definitions.

For both converse api and anthropic api, the stream chunks have strong dependencies. We should use complete responses to write unit tests.

codecov-commenter · 2025-10-30T21:05:07Z

Codecov Report

❌ Patch coverage is 84.61538% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.68%. Comparing base (c87e367) to head (e5509c3).

Files with missing lines	Patch %	Lines
internal/extproc/translator/openai_awsbedrock.go	22.22%	7 Missing ⚠️
...l/extproc/translator/openai_gcpanthropic_stream.go	94.73%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1468      +/-   ##
==========================================
- Coverage   83.99%   83.68%   -0.31%     
==========================================
  Files         142      142              
  Lines       12555    12588      +33     
==========================================
- Hits        10545    10534      -11     
- Misses       1400     1443      +43     
- Partials      610      611       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

yuzisun · 2025-10-31T09:33:08Z

internal/extproc/translator/gemini_helper.go

+				Name:      part.FunctionCall.Name,
+				Arguments: string(args),
+			},
+			Index: 0,


here should handle multiple tool calls to ensure that multiple tool calls in a single streaming response receive proper incremental indices

Signed-off-by: yxia216 <yxia216@bloomberg.net>

…ion (envoyproxy#1418) **Description** This PR implement adds a translator from the Anthropic API to AWS Bedrock for Anthropic. **Related Issues/PRs (if applicable)** Closes envoyproxy#1371 --------- Signed-off-by: secustor <sebastian@poxhofer.at> Signed-off-by: yxia216 <yxia216@bloomberg.net>

envoyproxy#1466) **Description** This is to fix issue: envoyproxy#1465 After this change, the empty `input: {}` is removed in `arguments` **Special notes for reviewers (if applicable)** Example: Before this change if user make a streaming request with tool use for claude: - data: {"choices":[{"index":0,"delta":{"content":"I'll check","role":"assistant"}}],"object":"chat.completion.chunk"} - data: {"choices":[{"index":0,"delta":{"content":" the delivery date for your order right"}}],"object":"chat.completion.chunk"} - data: {"choices":[{"index":0,"delta":{"content":" away."}}],"object":"chat.completion.chunk"} - data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":1,"id":"toolu_vrtx_013xZtkBipjVEQqLAdRtLyES","function":{"arguments":"{}","name":"get_delivery_date"},"type":"function"}]}}],"object":"chat.completion.chunk"} - data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":1,"id":null,"function":{"arguments":"","name":""}}]}}],"object":"chat.completion.chunk"} - data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":1,"id":null,"function":{"arguments":"{\"","name":""}}]}}],"object":"chat.completion.chunk"} - data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":1,"id":null,"function":{"arguments":"orde","name":""}}]}}],"object":"chat.completion.chunk"} - data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":1,"id":null,"function":{"arguments":"r_id\"","name":""}}]}}],"object":"chat.completion.chunk"} - data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":1,"id":null,"function":{"arguments":": \"order_12","name":""}}]}}],"object":"chat.completion.chunk"} - data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":1,"id":null,"function":{"arguments":"345\"}","name":""}}]}}],"object":"chat.completion.chunk"} - data: {"choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}],"object":"chat.completion.chunk"} - data: {"object":"chat.completion.chunk","usage":{"prompt_tokens":458,"completion_tokens":73,"total_tokens":531,"prompt_tokens_details":{}}} - data: [DONE] After this change: - data: {"choices":[{"index":0,"delta":{"content":"I'll help you check","role":"assistant"}}],"object":"chat.completion.chunk"} - data: {"choices":[{"index":0,"delta":{"content":" the delivery date for your order. Let"}}],"object":"chat.completion.chunk"} - data: {"choices":[{"index":0,"delta":{"content":" me look that up for you."}}],"object":"chat.completion.chunk"} - data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":1,"id":"toolu_vrtx_01CrVxrmRpMn5DgBTvy2cEFX","function":{"arguments":"","name":"get_delivery_date"},"type":"function"}]}}],"object":"chat.completion.chunk"} - data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":1,"id":null,"function":{"arguments":"","name":""}}]}}],"object":"chat.completion.chunk"} - data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":1,"id":null,"function":{"arguments":"{\"o","name":""}}]}}],"object":"chat.completion.chunk"} - data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":1,"id":null,"function":{"arguments":"rde","name":""}}]}}],"object":"chat.completion.chunk"} - data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":1,"id":null,"function":{"arguments":"r_i","name":""}}]}}],"object":"chat.completion.chunk"} - data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":1,"id":null,"function":{"arguments":"d\"","name":""}}]}}],"object":"chat.completion.chunk"} - data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":1,"id":null,"function":{"arguments":": \"order_12","name":""}}]}}],"object":"chat.completion.chunk"} - data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":1,"id":null,"function":{"arguments":"345\"}","name":""}}]}}],"object":"chat.completion.chunk"} - data: {"choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}],"object":"chat.completion.chunk"} - data: {"object":"chat.completion.chunk","usage":{"prompt_tokens":458,"completion_tokens":81,"total_tokens":539,"prompt_tokens_details":{}}} - data: [DONE] --------- Signed-off-by: ydu208 <YDU208@bloomberg.net> Co-authored-by: ydu208 <YDU208@bloomberg.net> Co-authored-by: Dan Sun <dsun20@bloomberg.net> Signed-off-by: yxia216 <yxia216@bloomberg.net>

**Description** The extproc test's envoy config was missing TLS config for AWS Bedrock sicne envoyproxy#1418. Since the test is using credentials, hence only running on the main branch, this was not caught before merging it. **Related Issues/PRs (if applicable)** Follow up on envoyproxy#1418 Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com> Signed-off-by: yxia216 <yxia216@bloomberg.net>

Signed-off-by: yxia216 <yxia216@bloomberg.net>

Signed-off-by: Dan Sun <dsun20@bloomberg.net> Signed-off-by: yxia216 <yxia216@bloomberg.net>

Signed-off-by: yxia216 <yxia216@bloomberg.net>

hustxiayang · 2025-10-31T19:25:53Z

/retest

Signed-off-by: yxia216 <yxia216@bloomberg.net>

hustxiayang requested a review from a team as a code owner October 30, 2025 21:02

hustxiayang marked this pull request as draft October 30, 2025 21:02

hustxiayang changed the title ~~fix index of tool calls in stream mode~~ fix: set index of tool calls in stream mode Oct 30, 2025

hustxiayang marked this pull request as ready for review October 30, 2025 21:53

yuzisun reviewed Oct 31, 2025

View reviewed changes

yuzisun force-pushed the tool-call-index branch from a27b5cd to a7ded26 Compare October 31, 2025 14:30

mathetake assigned mathetake and yuzisun and unassigned mathetake Oct 31, 2025

hustxiayang marked this pull request as draft October 31, 2025 16:09

hustxiayang and others added 8 commits October 31, 2025 12:12

fix-index

c3b9d46

Signed-off-by: yxia216 <yxia216@bloomberg.net>

fix_test

14063fc

Signed-off-by: yxia216 <yxia216@bloomberg.net>

handle multiple tool calls in the same stream and add tests

dcf48e5

Signed-off-by: yxia216 <yxia216@bloomberg.net>

add tests for anthropic stream events

c30ec2b

Signed-off-by: Dan Sun <dsun20@bloomberg.net> Signed-off-by: yxia216 <yxia216@bloomberg.net>

counting

26b609e

Signed-off-by: yxia216 <yxia216@bloomberg.net>

hustxiayang force-pushed the tool-call-index branch from ab6f765 to 26b609e Compare October 31, 2025 16:12

Merge branch 'main' into tool-call-index

b80aa7f

fix-tests

475dd3c

Signed-off-by: yxia216 <yxia216@bloomberg.net>

hustxiayang force-pushed the tool-call-index branch from 327858e to 475dd3c Compare October 31, 2025 19:39

Merge branch 'main' into tool-call-index

e5509c3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: set index of tool calls in stream mode #1468

fix: set index of tool calls in stream mode #1468

Uh oh!

hustxiayang commented Oct 30, 2025 •

edited

Loading

Uh oh!

codecov-commenter commented Oct 30, 2025 •

edited

Loading

Uh oh!

yuzisun Oct 31, 2025

Uh oh!

hustxiayang commented Oct 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

fix: set index of tool calls in stream mode #1468

Are you sure you want to change the base?

fix: set index of tool calls in stream mode #1468

Uh oh!

Conversation

hustxiayang commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

yuzisun Oct 31, 2025

Choose a reason for hiding this comment

Uh oh!

hustxiayang commented Oct 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

hustxiayang commented Oct 30, 2025 •

edited

Loading

codecov-commenter commented Oct 30, 2025 •

edited

Loading