Secret loading is sequential and missing pre-emptive authentication

[ronfor]

Summary

While using the library to load App Configuration, I’ve observed slow performance due to two key issues:

• Secrets are fetched from Key Vault sequentially
• No pre-emptive authentication is performed

Investigation

This behaviour was identified using Application Insights, with traces collected via OpenTelemetry on a Node.js process.

Below is an example trace showing a single configuration lookup referencing 14 secrets (which are stored in 4 x separate vaults):

The loadConfig trace wraps the point we call the load function:

AppConfiguration-JavaScriptProvider/src/load.ts

Line 28 in 5c1f4a3

export async function load(

Each secret fetch initially results in a 401 (unauthenticated), followed by a successful retry after authentication. This happens for every unique Key Vault endpoint used.

Despite the environment running within Azure, the full configuration load takes approximately 2 seconds, introducing noticeable latency.

Expected behaviour

Performance could be improved significantly if:

• Secrets were fetched in parallel
• Authentication was performed proactively, avoiding repeated 401/retry cycles

[zhiyuanliang-ms]

Hi, @ronfor. Thanks for reaching out!

No pre-emptive authentication is performed

This behavior of Azure Key Vault is by designed. Key Vault backend uses a challenge-based authentication. Please see this issue #31228

Secrets are fetched from Key Vault sequentially

We are looking into make App Config provider fetch Key Vault secret in parallel. I will come back to you after I implement the POC.

[zhiyuanliang-ms]

Hi, @ronfor

I write a POC in this branch zhiyuanliang/secret-performance

Can you try this private build and check to what extent it can improve the performance.

I added 15 key vault reference in my App Config store,

import { DefaultAzureCredential } from "@azure/identity";
const credential = new DefaultAzureCredential();

import { load } from "@azure/app-configuration-provider";
const connectionString = process.env.APPCONFIG_CONNECTION_STRING;

const totalStartTime = performance.now();

const settings = await load(connectionString, {
    selectors: [
        // { keyFilter: "Message" },
        // { keyFilter: "PerformanceTest_*" },
        { keyFilter: "KeyVaultReference_*" },
    ],
    keyVaultOptions: {
        credential: credential
    }
});

const totalEndTime = performance.now();
const totalDuration = totalEndTime - totalStartTime;

console.log(`Total time: ${totalDuration.toFixed(2)}ms`);

This is the code I used for performance test.

It took Total time: 8579.05ms to load 15 key vault secrets on my PC when I used the current stable release of @azure/app-configuration-provider.

With the above private build (which has the parallel secret resolution enhancement), it now takes Total time: 4297.33ms to load all 15 key vault secrets.

I also have some performance baselines of my PC:
It tooks Total time: 3091.76ms to load 500 configuration settings (key value).
It tooks Total time: 1349.15ms to load a single configuration settings.
It tooks Total time: 10956.11ms to load 501 configuration settings and 15 secret references originally (if secret is loaded sequentially).
It tooks Total time: 6858.32ms to load 501 configuration settings and 15 secret references (if secret is loaded in parallel).

[ronfor]

I've tried this out and the approach does now run things in parallel, and has reduced the time taken in my scenario by about 40%. I know this is a POC but here are some additional observations:

• the parallel approach now runs the "challenge-based authentication" approach for each call, forcing each to be rejected by with a 401 before being retried (see screenshot)
• there is a duplication of calls for the same secret, reducing the key names would limit the number of requests which need to be made if there are duplicate secret references in the configuration.
• I would echo the comment being made here: Resolve key vault secret in parallel #192 (comment) - some sort of batching or configuration needs to be available to stop massive parallel calls being issued.
• Error handling could probably also be improved

[zhiyuanliang-ms]

Let me provide more context about Azure JavaScript SDKs here.

All Azure services have client libraries which are under the Azure SDK of certain programming language. Those SDKs are usually just wrappers for the RESTful APIs of those Azure services. This is the standard pattern. Those SDKs are developed by the Azure SDK team.

However, Azure App Configuration is kind of different. In addition to the Azure SDK, we, the App Config team, develop configuration provider libraries which is built on the Azure SDK and provide better user experience and additional functionalities.

Taking JavaScript as example, there are two libraries: @azure/app-configuration (Azure SDK) and @azure/app-configuration-provider (configuration provider)

For our customers, these two packages are just SDKs/client libraries of Azure App Configuration. But for us(App Config team), these are two things. We want our customers to use the configuration provider library for better user experience.

For more information about the difference between App Config configuration provider library and App Config Azure SDK, please go to configuration provider overview

---

The Key Vault secret reference stored in Azure App Configuration is just key-value with special content type and its value is a url to the Key Vault secret. So if you use the @azure/app-configuration (Azure SDK) to get that secret reference, the response you get is just a configuration settings with a url value and a content type indicating this key-value is a secret reference.

The @azure/app-configuration (configuration provider library that you are using) provides the additional functionality to load the secret from the Key Vault according to the url (that secret reference key-value's value).

The configuration provider library has dependency on @azure/keyvault-secrets (which is the Azure SDK of Key Vault). The configuration provider doesn't directly send request to Azure Key Vault, we are calling the SecretClient.getSecret method (API ref and provider's implementation ) from the key vault sdk.
So, the 401 and challenge auth are all handled by the key vault sdk. This behavior is not controlled by Azure App Configuration. I am just guessing, maybe there is other authentication way supported by Key Vault sdk. You can use secret client instance with custom configurations to resolve secret or even use a custom secret resolver (callback function). See this doc.
I also have an unpublished doc for .NET configuration provider which is more detailed:

It explains the three ways for configuration provider to connect key vault better. The API is different from the JS provider, but the general idea is the same.

So if there is a way supported by Key Vault SDK to get rid of the 401, you can use it in the App Config configuration provider.

But, unfortunately, Azure Key Vault doesn't support batch read of secrets. You can only read secret one by one (request by request).

[zhiyuanliang-ms]

Hi, @ronfor

I would echo the comment being made here: #192 (comment) - some sort of batching or configuration needs to be available to stop massive parallel calls being issued.

I replied to the original thread.

I think we can use the built-in retry mechanism supporte by Key Vault sdk to deal with throttling issue. This also follows their best practice.

I think the 401 behavior is not controlled by App Config. As my previous comment mentioned, the configuration provider library depends on the Key Vault SDK, we are calling their API to get secret from Azure Key Vault.

[zhiyuanliang-ms]

Hi, @ronfor

#192 added an option to enable parallel secret resolution. The PR has been merged. It will be released in 2.1.0.

[zhiyuanliang-ms]

Hi, @ronfor

2.1.0 has been released, which includes the new feature to allow parallel resolution for Key Vault secret. release note