CAPZ with ASO doesn't work for AzureUSGovernment

[ionutleca]

/kind bug

What steps did you take and what happened:
The AzureManagedControlPlane supports the following parameter:

spec:
  azureEnvironment: AzureUSGovernmentCloud

The createSecretFromClusterIdentity function doesn't set azureResourceManagerEndpoint and any other cloud specific variables.

The ASO controller fails with:
The subscription '***' could not be found.: PUT https://management.azure.com/subscriptions/***/resourceGroups/***

What did you expect to happen:
CAPZ to also add to the *-aso-secret the values specific to what spec.azureEnvironment on the AzureManagedControlPlane resource points to.

Anything else you would like to add:

Environment:

• cluster-api-provider-azure version: v1.11.1
• Kubernetes version: (use kubectl version): v1.27.3
• OS (e.g. from /etc/os-release): AKSUbuntu-2004gen2fipscontainerd-202309.06.0

[Jont828]

/triage accepted

[Jont828]

/assign @mboersma

[nojnhuh]

It appears ASO can only configure the ARM endpoint globally for all resources it manages. I've opened an issue here to make that capability accessible at the per-resource level to match CAPZ's capabilities from before: Azure/azure-service-operator#3447.

If your management cluster is only managing workload clusters for one cloud, you could possibly modify the ASO deployment manually to configure it as you need as a stopgap.

[ionutleca]

Hi @nojnhuh, I was looking into how we could globally set the ARM endpoint url for the ASO controller but I can't find a way.

CAPZ is always setting the resource scoped secret https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/main/controllers/asosecret_controller.go#L235

ASO is not merging the resource and the global secret, but stops after getCredentialFromAnnotation https://github.com/Azure/azure-service-operator/blob/main/v2/internal/identity/credential_provider.go#L98-L99

Because of this, even if we do manually set the global or namespaced settings for the ASO controller, they will be ignored (unless I missed something) :(

[nojnhuh]

Even if changing it in aso-controller-settings doesn't work, I suppose it would be possible to edit the ASO deployment and hardcode the value there or referring to a different secret instead of deriving it from the global ASO secret.

[nojnhuh]

@ionutleca Did that workaround work for you? I'll reopen this to keep tracking making this more automatic.

[ionutleca]

Yes, I managed to make it work with the right values set in aso-controller-settings. Thank you!

[CecileRobertMichon]

/assign @nojnhuh
/unassign @mboersma
/cc @nawazkh

[nojnhuh]

Just opened this PR to at least allow setting these fields with environment variables when CAPZ is installed. Hopefully that's at least a step in the right direction: #4390

[nojnhuh]

There's still more to iron out here re: CAPZ configuring the environment per-workload cluster and ASO only configuring it globally for all resources it manages, but I don't think I'll be able to follow up with that during this milestone.

/unassign

[dtzar]

Blocked until Azure/azure-service-operator#3447