[BUG] Azure.CLI timeout when running in AzureHosted Agents

[KrisSodroski]

Library name and version

Azure.identity 1.7.0

Describe the bug

I've been running into this the last couple of months in our Azure DevOps pipelines.

In the pipeline, we use an Azure Powershell task to switch to the Service Principal account when running our C# code. When trying to get credentials to connect to the keyvault, it seems to always timeout now. This works for developers without issues (pulling credentials from Visual Studio or local cache), but when running in the pipeline (using Azure Hosted Build agents), the request times out 99 times out of a hundred now:

Loading secrets from https://mykeyvault.vault.azure.net

There was an issue with keyvault values
Azure PowerShell authentication timed out.

Before, this same procedure (been in use for over 2 years now) worked reliably, but as of late, it basically always fails. Every 1/100 requests works, so I know that it is possible for it to connect successfully still.

I've tried to up the timeout the DefaultAzureCredentials, but it doesn't seem to actually change anything, or even wait longer before failing.

Console.WriteLine($"Loading secrets from ${KeyVaultUrl}");
                
SecretClientOptions secretOptions = new SecretClientOptions();
DefaultAzureCredentialOptions credentialOptions = new DefaultAzureCredentialOptions();
credentialOptions.Retry.NetworkTimeout = TimeSpan.FromSeconds(20);
DefaultAzureCredential credentials = new DefaultAzureCredential(credentialOptions);

this.Client = new SecretClient(new Uri(KeyVaultUrl), credentials, secretOptions);

I've updated to latest Azure.Identity (1.7) and Azure.Security.KeyVaults.Secrets (4.4.0).

Expected behavior

Azure PowerShell token auth shouldn't timeout.

Actual behavior

Loading secrets from https://mykeyvault.vault.azure.net

There was an issue with keyvault values
Azure PowerShell authentication timed out.

Reproduction Steps

1. Create a C# program that makes a call to AzureKeyVault:

Console.WriteLine($"Loading secrets from ${KeyVaultUrl}");
                
SecretClientOptions secretOptions = new SecretClientOptions();
DefaultAzureCredentialOptions credentialOptions = new DefaultAzureCredentialOptions();
credentialOptions.Retry.NetworkTimeout = TimeSpan.FromSeconds(20);
DefaultAzureCredential credentials = new DefaultAzureCredential(credentialOptions);

this.Client = new SecretClient(new Uri(KeyVaultUrl), credentials, secretOptions);
var keys = this.Client.GetPropertiesOfSecrets();

2. Create a Build in AzureDevOps that uses AzureHosted Agents, windows-latest
3. Create an AzurePowershell task that authenticates with your resource group.
4. Call the C# program in the AzurePowershell Task.
5. Notice the AzureCLI timeout when returning the token sometimes
6. Increasing the NetworkTimeout has no effect.

Environment

Azure Powershell Output:

2022-10-18T22:01:22.9631030Z ##[command]"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'D:\a\_temp\418a992a-3b46-4a1c-bd77-650312f89f54.ps1'"
2022-10-18T22:01:24.7635896Z ##[command]Import-Module -Name C:\Modules\az_8.3.0\Az.Accounts\2.10.0\Az.Accounts.psd1 -Global
2022-10-18T22:01:26.7303071Z ##[command]Clear-AzContext -Scope CurrentUser -Force -ErrorAction SilentlyContinue
2022-10-18T22:01:27.4586165Z ##[command]Clear-AzContext -Scope Process
2022-10-18T22:01:27.5988331Z ##[command]Connect-AzAccount -ServicePrincipal -Tenant TenantID -Credential System.Management.Automation.PSCredential -Environment AzureCloud @processScope
2022-10-18T22:01:29.7357522Z ##[command] Set-AzContext -SubscriptionId SubscriptionID -TenantId TenantID

[jsquire]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[christothes]

Hi @KrisSodroski
Could you clarify which credential in the DefaultAzureCredential chain you are expecting to be used in this scenario? Would you mind providing the logging output after reproducing this with logging enabled to determine what decision was made by DefaultAzureCredential?

[KrisSodroski]

Azure Powershell logs in with a managed identity that we specify, using AzModule as you can see from these logs.
I assume it would pull from the ManagedIdentityCredentials.

Clear-AzContext -Scope CurrentUser -Force -ErrorAction SilentlyContinue
Clear-AzContext -Scope Process
Connect-AzAccount -ServicePrincipal -Tenant TenantID -Credential System.Management.Automation.PSCredential -Environment AzureCloud @processScope
Set-AzContext -SubscriptionId SubscriptionID -TenantId TenantID

I've added the diagnostics, but it still only displays the same message:

Azure CLI authentication timed out.

If I don't use the azure powershell task (no identity available), I get more logging, so I know the diagnostics is working.

In Example when I don't use Azure Powershell (AzModule isn't installed, so it should fail as expected):

- EnvironmentCredential authentication unavailable. Environment variables are not fully configured. See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/environmentcredential/troubleshoot
- ManagedIdentityCredential authentication unavailable. Multiple attempts failed to obtain a token from the managed identity endpoint.
- Visual Studio Token provider can't be accessed at C:\Users\US_SVC_TFSBLD\AppData\Local\.IdentityService\AzureServiceAuth\tokenprovider.json
- Stored credentials not found. Need to authenticate user in VSCode Azure Account. See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/vscodecredential/troubleshoot
- Azure CLI not installed
- Az.Account module >= 2.2.0 is not installed.

The task for reference if it helps: