Dynamic embedding batch sizes

[lsorber]

OpenAI puts a limit of 8192 tokens per embedding request. We should dynamically adjust the batch sizes to fit within that limit.

[ThomasDelsart]

Regarding the issue: you want to modify the batch size used when embedding text with an OpenAI model. Currently, the batch size is set to 128, meaning up to 128 strings are embedded per request. This has consistently worked well.

While it's true that OpenAI enforces a limit of 8192 tokens per individual input string, this typically isn't a concern at the sentence level. In my understanding, that token cap applies per string, not across the entire batch.

I tested larger batch sizes by manually increasing the number of strings sent in a single request. It failed with a batch of 2760 strings, returning the following error:

   raise BadRequestError(
litellm.exceptions.BadRequestError: litellm.BadRequestError: OpenAIException - Error code: 400 - {'error': {'message': 'Requested 415282 tokens, max 300000 tokens per request', 'type': 'max_tokens_per_request', 'param': None, 'code': 'max_tokens_per_request'}}

I'm opening a PR now to implement what you suggested: dynamic embedding batch sizes for OpenAI models. However, as shown above, it appears that LiteLLM enforces a hard limit of 300,000 tokens per request at the batch level. Given that, I'm not entirely convinced this change adds much value—we could simply hardcode the batch size to a safe, known working value.

Maybe I am missing something or misunderstood your intent.