feat(embedderconfig): add embedding_openai_endpoint argument to EmbedderConfig for custom OpenAI deployments

[dutchfarao]

Summary

This pull request adds a embedding_openai_endpoint argument to the EmbedderConfig class, allowing users to set a custom base_url when using an OpenAI client. This is essential for custom LLM deployments that follow the OpenAI API convention.

Details

• Added embedding_openai_endpoint argument to EmbedderConfig class.
• Ensured backward compatibility by providing a default value for base_url.

Testing

I have verified that the changes work for our custom deployment. I have also verified that the change does not break Pipeline if no embedding_openai_endpoint is provided.

from unstructured_ingest.v2.processes.embedder import EmbedderConfig
from unstructured_ingest.embed.openai import OpenAIEmbeddingConfig, OpenAIEmbeddingEncoder

embedder_config=EmbedderConfig(
        embedding_provider='openai',
        embedding_api_key="your_api_key",
        embedding_openai_endpoint='https://custom-openai-instance.com'
        )
# Initialize the OpenAI client with the custom config as done in EmbedderConfig.get_openai_embedder()
config_kwargs = {
            "api_key": embedder_config.embedding_api_key,
            "base_url": embedder_config.embedding_openai_endpoint,
        }
        
if model_name := embedder_config.embedding_model_name:
            config_kwargs["model_name"] = model_name

client = OpenAIEmbeddingEncoder(config=OpenAIEmbeddingConfig.model_validate(config_kwargs))
# Verify that the base URL is correctly set
assert client.config.base_url == "https://custom-openai-instance.com"

[mingfang]

Please merge. I need this.

[micmarty-deepsense]

@dutchfarao , @mingfang base_url support was introduced some time ago, please see this line:

unstructured-ingest/unstructured_ingest/embed/openai.py

Line 32 in 23ada7c

base_url: Optional[str] = Field(default=None, description="optional override for the base url")

You can check that by rebasing/updating this PR's branch