Log probabilities for chat completion output tokens #625
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #625 +/- ##
=======================================
Coverage 98.44% 98.44%
=======================================
Files 24 24
Lines 1354 1354
=======================================
Hits 1333 1333
Misses 15 15
Partials 6 6 ☔ View full report in Codecov by Sentry. |
|
@sashabaranov would it be possible to get a review? This would unblock us without having to maintain a fork of |
|
@danai-antoniou, thank you for the PR, and sorry for the sluggish review — I was away for the holidays! :D |
chat.go
Outdated
| // LogProbs indicates whether to return log probabilities of the output tokens or not. | ||
| // If true, returns the log probabilities of each output token returned in the content of message. | ||
| // This option is currently not available on the gpt-4-vision-preview model. | ||
| LogProbs *bool `json:"logprobs,omitempty"` |
There was a problem hiding this comment.
If we use *bool and *int here, we'll be unable to set those inline like
req := &openai.ChatCompletionRequest{
LogProbs: true,
TopLogProbs: 3,
}See example: https://go.dev/play/p/YZzhc0nEOzf
Using just bool and int would be more ergonomic for developers and we'll still be able to omitempty as top_logprobs=0 is the same as null.
There was a problem hiding this comment.
Ah yes good shout ✅
|
Thank you! ❤️ |
Describe the change
OpenAI finally released log probabilities for chat completions. This PR implements the required changes to be able to request log probabilities in chat completion requests and receive them in the chat completion response.
Provide OpenAI documentation link
https://platform.openai.com/docs/api-reference/chat/object
Describe your solution
Added
logprobsandtop_logprobsin the chat completion request and response.Tests
Manually tested by making chat completions
Issues
Closes #626