Support adding model name path segments to non-Azure URLs

[devUndef1ned]

In my company we use APIM for requests to LLM. We have similar endpoints structure for that:

https://BASE_URL/openai/deployments/DEPLOYMENT_NAME/chat/completions?api-version=API_VERSION

BASE_URL is an internal URL not accessible outside of the company network and it is not azure openai endpoint.

Since our URL is not azure, the SDK logic doesn't add required path segments and our calls have this url in the end:

https://BASE_URL/chat/completions

To make it working we have to hardcode values when building http client, so it makes us not possible to dynamic change deployment name:

private val client = OpenAIOkHttpClientAsync.Companion.builder()
        .baseUrl(endpoint + "/openai/deployments/gpt-4o-mini")
        .credential(AzureApiKeyCredential.create(apiKey))
        .azureServiceVersion(AzureOpenAIServiceVersion.latestStableVersion())
        .queryParams(QueryParams.builder().put("api-version", "2024-10-21").build())
        .build()

We'd like to have possibility to adjust final url of calls when invoke chat completions

[TomerAberbach]

@jpalvarezl can you help figure out what we should do here?

[TomerAberbach]

@devUndef1ned, to be clear, does your service end up sending the request to Azure? Or is it completely unrelated to Azure? If it's the latter, then using these Azure classes seems wrong in general, since they are not designed for non-Azure usage

[devUndef1ned]

@devUndef1ned, to be clear, does your service end up sending the request to Azure? Or is it completely unrelated to Azure? If it's the latter, then using these Azure classes seems wrong in general, since they are not designed for non-Azure usage

I don't know all details of APIM implementation, but in the end it points indeed to Azure

[devUndef1ned]

@TomerAberbach would it be an issue if for each model that we'd like to use we keep a separate OpenAIAsyncClient instance in memory? It can give us a workaround on the issue. Something like

private val clientRegistry: MutableMap<ChatModel, OpenAIOkHttpClientAsync> = mutableMapOf()

[devUndef1ned]

@jpalvarezl maybe you can help above ⤴

[TomerAberbach]

Sorry, I've been on vacation recently. This is on my list of things to look at today!

[TomerAberbach]

Okay, so question for you, we actually coincidentally have plans to add a withOptions method to the client and every service

So you'd be able to do stuff like:

client..chat().completions()
    .withOptions(optionsBuilder -> {
		optionsBuilder.baseUrl("new base URL just for this call");
	})
   .create(...);

// OR

OpenAIOkHttpClientAsync clientWithOptions = client.withOptions(optionsBuilder -> {
	optionsBuilder.baseUrl("new base URL just for this client");
});

// All these calls use the new options
clientWithOptions.create(...);
clientWithOptions.create(...);
clientWithOptions.create(...);

Note that withOptions returns a view of the original client (it does not mutate it). The nice thing about this is that withOptions returns a client that reuses the original client's thread and connection pools (very important!)

Do you think such a thing would help you? I feel like maybe you could just write your own utility function that calls takes the params, calls withOptions to set the custom base URL based on the model, and then makes the call?

Let me know if you think that'd solve this use-case or not!

[jpalvarezl]

Hi @devUndef1ned , just digging through the code and some samples we prepared here, it seems to me that by setting the model as part of the request object, the deployment name gets set for you as a path parameter for the request here

This could just be a guess, according to what I can understand from your description, it seems like you are using a base URL that might not be picked up as an Azure URL and therefore, the request is not handled as such.

I will discuss internally what can be done in these cases, but do let me know if I am understanding your situation correctly.

Also, sorry for the lack of response from my side as well, I am also playing catch up after some PTO.

[devUndef1ned]

Hi @devUndef1ned , just digging through the code and some samples we prepared here, it seems to me that by setting the model as part of the request object, the deployment name gets set for you as a path parameter for the request here

This could just be a guess, according to what I can understand from your description, it seems like you are using a base URL that might not be picked up as an Azure URL and therefore, the request is not handled as such.

I will discuss internally what can be done in these cases, but do let me know if I am understanding your situation correctly.

Also, sorry for the lack of response from my side as well, I am also playing catch up after some PTO.

Your guess is correct. We use Azure APIM as a gateway to all our GenAI models calls. Since it's not picked up as an "Azure" endpoint, the deployment and other path segments aren't added by end call url so that we need manually hardcode the full url as base url when building OpenAIOkHttpClientAsync

[devUndef1ned]

Okay, so question for you, we actually coincidentally have plans to add a withOptions method to the client and every service

So you'd be able to do stuff like:

client..chat().completions()
.withOptions(optionsBuilder -> {
optionsBuilder.baseUrl("new base URL just for this call");
})
.create(...);

// OR

OpenAIOkHttpClientAsync clientWithOptions = client.withOptions(optionsBuilder -> {
optionsBuilder.baseUrl("new base URL just for this client");
});

// All these calls use the new options
clientWithOptions.create(...);
clientWithOptions.create(...);
clientWithOptions.create(...);
Note that withOptions returns a view of the original client (it does not mutate it). The nice thing about this is that withOptions returns a client that reuses the original client's thread and connection pools (very important!)

Do you think such a thing would help you? I feel like maybe you could just write your own utility function that calls takes the params, calls withOptions to set the custom base URL based on the model, and then makes the call?

Let me know if you think that'd solve this use-case or not!

Looks quite promising! If we can modify the end url (in our case it is the same as the baseUrl) on a fly for each call separately, then we don't need to hardcode a deployment name when building OpenAIOkHttpClientAsync