support batching

[go-dockly]

It would be nice if the chatCompletion method could accept an array of requests optionally as described below:

Batching requests
The OpenAI API has separate limits for requests per minute and tokens per minute.

If you're hitting the limit on requests per minute, but have available capacity on tokens per minute, you can increase your throughput by batching multiple tasks into each request. This will allow you to process more tokens per minute, especially with our smaller models.

Sending in a batch of prompts works exactly the same as a normal API call, except you pass in a list of strings to the prompt parameter instead of a single string.

[sashabaranov]

Done in #220

[animalnots]

Hi, @sashabaranov,
I noticed that this feature seems different from the batch API as described in the Batch API FAQ. Could you clarify how it supports batch processing, including statuses, job checks, and JSONL logic?

I've also noticed a PR which mentions support for it.
And an issue asking support for it.