Keda 2.9.1 on AKS with Pod-Identity Looks for AzureCLICredential-

[tshaiman]

Report

when running Keda 2.9.1 with Pod Identity ( No Workload Identity) , the DefaultAzureCredentials() chain look for AzureCLICredentials but fails on "/bin/sh azurecli file not found"

The AzureCLICredential should be remvoed from the Chain list

Expected Behavior

Default Azure Credentials has options to opt-Out several Chain providers such as VisualStudioCredentials /AzureCLI Credentials etc.
so since this is Pod-Identity with distroless image the Azure CLI should not be part of this chain

Actual Behavior

many "AzureCLICredential: fork/exec /bin/sh: no such file or directory\n\terror reading service account token"

Steps to Reproduce the Problem

1. Use AKS 1.24 with Mariner node Pools
2. Use Keda 2.91 with WorkloadIdentity=False
3. Deploy keda with AAD-Pod Identity and add Scaled Object

Logs from KEDA operator

keda-operator-d5464cdd6-zvdw2 keda-operator 2022-12-18T15:03:48Z        ERROR   azure_servicebus_scaler error getting service bus entity length {"type": "ScaledObject", "namespace": "vi-be-map-dev11", "name": "rc-visolo", "error": "ChainedTokenCredential: failed to acquire a token.\nAttempted credentials:\n\tAzureCLICredential: fork/exec /bin/sh: no such file or directory\n\terror reading service account token - open : no such file or directory"}
keda-operator-d5464cdd6-zvdw2 keda-operator github.com/kedacore/keda/v2/pkg/scalers.(*azureServiceBusScaler).GetMetricsAndActivity
keda-operator-d5464cdd6-zvdw2 keda-operator     /workspace/pkg/scalers/azure_servicebus_scaler.go:266
keda-operator-d5464cdd6-zvdw2 keda-operator github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).GetScaledObjectState
keda-operator-d5464cdd6-zvdw2 keda-operator     /workspace/pkg/scaling/cache/scalers_cache.go:136
keda-operator-d5464cdd6-zvdw2 keda-operator github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
^Ckeda-operator-d5464cdd6-zvdw2 keda-operator   /workspace/pkg/scaling/scale_handler.go:360
keda-operator-d5464cdd6-zvdw2 keda-operator github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
keda-operator-d5464cdd6-zvdw2 keda-operator     /workspace/pkg/scaling/scale_handler.go:162
keda-operator-d5464cdd6-zvdw2 keda-operator 2022-12-18T15:03:48Z        ERROR   azure_servicebus_scaler error getting service bus entity length {"type": "ScaledObject", "namespace": "vi-be-map-dev11", "name": "celebs", "error": "ChainedTokenCredential: failed to acquire a token.\nAttempted credentials:\n\tAzureCLICredential: fork/exec /bin/sh: no such file or directory\n\terror reading service account token - open : no such file or directory"}
keda-operator-d5464cdd6-zvdw2 keda-operator github.com/kedacore/keda/v2/pkg/scalers.(*azureServiceBusScaler).GetMetricsAndActivity
keda-operator-d5464cdd6-zvdw2 keda-operator     /workspace/pkg/scalers/azure_servicebus_scaler.go:266
keda-operator-d5464cdd6-zvdw2 keda-operator github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).GetScaledObjectState
keda-operator-d5464cdd6-zvdw2 keda-operator     /workspace/pkg/scaling/cache/scalers_cache.go:136

KEDA Version

2.9.1

Kubernetes Version

1.24

Platform

Microsoft Azure

Scaler Details

Azure Service Bus

Anything else?

No response

[JorTurFer]

Hi,
AzureCLICredentials is useful for local troubleshooting, but it's weird because it shouldn't fail as ChainedTokenCredential doesn't a fallback for every provider in the chain (it's the default behaviour, indeed).
If this is a problem, we have to remove it from the chain, but it's weird because e2e tests cover the scenario where AzureCLICredentials is tried as part of the chain and there isn't any error, and it works

[JorTurFer]

Could the problem be that no provider in the chain can get the token, and you only see the error from the first one?, I have to try it

[tshaiman]

no there are MI pod identity that works fine
when i take the same code and revert it to keda 2.8.1 it works

[barclayadam]

I've experienced the same issue upgrading from (Helm Chart) 2.8.2. Using AAD pod identity works perfectly with that version. The only change I make is to upgrade to 2.9.2

We have a single TriggerAuthentication that does not specify identityId as part of podIdentity spec, which should mean it falls back to aadpodidentity label binding. Adding explicit identityId makes no difference.

Azure SDK is supposed to return the last error. Is it possible that the credentials are not being added to the chain? Is there an error at https://github.com/kedacore/keda/blob/main/pkg/scalers/azure_servicebus_scaler.go#L330 that means it's not added?

[JorTurFer]

Interesting point... Maybe it's added IDK why 🤔
Let me try your scenario in my local cluster to be sure about that

[JorTurFer]

I have added a logger in case of errors that prevent the addition of it. The tags are jorturfer/keda:msi-logger and jorturfer/keda-metrics-apiserver:msi-logger (both are generated from main, so they are 2.9.1 + the logger change). Could you try to deploy that version and see if there is any error? In parallel, I'm adding aad-pod-identity to my local cluster to check it from my side

[JorTurFer]

I have reproduced the issue, but it doesn't seem related with the AzureCLICredential because removing it fails too. I feel that it's related with any wrong configuration after sdk migration (we upgraded the service bus SDK) with the bad luck that AAD-Pod-Identity doesn't have e2e test :(

[JorTurFer]

I have found the issue and sadly it's really complex to solve properly because the SDK it's totally closed for extension... For the moment, I'm going to undo the change that unified both identity providers (which I hope is the future) meanwhile we try to include our requirements.

[tshaiman]

thanks for the update @JorTurFer

[jmos5156]

So what options do we have for those of us experiencing this particular issue, are on :
AKS - version 1.23.5
aad-pod - chart version 4.1.15
keda - chart version: 2.9.1

TIA