[IDEA] allow setting batch size for embedding model

[King-of-Infinite-Space]

Describe the feature you'd like

Currently the embedding model sends up to 1000 entries to api each time. However, third-party embedding APIs may impose different maximum batch size (despite being OpenAI compatible). For better compatibility, I suggest adding a setting for embedding batch size either from the admin panel or as an environment variable.

khoj/src/khoj/processor/embeddings.py

Line 96 in 0ddb6a3

def embed_documents(self, docs):