[Feature]: return Usage info for streaming request for each chunk in ChatCompletion

[yecohn]

🚀 The feature, motivation and pitch

in entrypoints.openai.serving_completions.py I see OpenAIServingCompletion holds the method completion_stream_generator that can return usage info for each chunk by using StreamOptions continuous_usage_stats.
line 297.

``if (request.stream_options
and request.stream_options.include_usage):
if (request.stream_options.continuous_usage_stats
or output.finish_reason is not None):
prompt_tokens = len(res.prompt_token_ids)
completion_tokens = len(output.token_ids)
usage = UsageInfo(
prompt_tokens=prompt_tokens,
completion_tokens=completion_tokens,
total_tokens=prompt_tokens + completion_tokens,
)

Somehow, this is not the case in entrypoints.openai.serving_chat.py. I propose to add this feature for OpenAIServingChat.

What do you think ?

Alternatives

No response

Additional context

No response

[tdoublep]

@yecohn I guess it makes sense. We added it to the completions API because that is the API we are primarily using for benchmarking. However, if it's helpful to have it in the chat API too, I don't see any reason not to add it. Shouldn't be a big change.

[yecohn]

Perfect then I'll open a PR