[Feature] Provide a method for users to identify the "previous key" which is in-use by the API server KMS sidecar

[tspearconquest]

Is your feature request related to a problem? Please describe.
During a troubleshooting call with MS support, an AKS engineer was engaged. The ticket number is 2504080040007299

The AKS engineer explained that KMS makes use of what I will refer to as "key slots" in the sidecar attached to the API server and that if the key in either key slot expires in Keyvault, then the cluster can encounter problems.

We are looking for a way to identify for certain which key is in the "previous key" field. Azure Support is able to see this field, but we are unable to see it from any of the following:

1. az aks show -g <resource group name> -n <cluster name>
2. JSON output for the cluster via the Azure Portal
3. Terraform output and terraform state
4. Export Template feature in the portal for the cluster, to generate an ARM template of the cluster

Because of this, it is impossible for users to know if an expired key or one which has been deleted from Key vault is still in-use within a cluster, and this leads to confusion and downtime despite the documentation around the use of 2 keys at the same time.

Describe the solution you'd like

It would be incredibly helpful for us to have peace-of-mind that our keys in-use on a KMS enabled cluster are not going to cause problems if we were able to view the Previous Key field in any of the above.

Describe alternatives you've considered

There is no alternative. We have no method to view this for ourselves, and must engage Azure Support to check this for us.

Additional context

Our clusters experienced multiple days of downtime due to the lack of clarity. We had extended our existing KMS key to allow us to change the key, and added a new key in its place, but when the old key expired, it still caused us to have an outage. If we had been able to see that the "previous key" was the expired key, we could have avoided contacting support and instead just rotated the key without needing assistance from both support and an AKS engineer.

[lzhecheng]

@tspearconquest , thank you for bringing this up.
AKS team will try to add this support in "diagnose and solve" in azure portal. Will update the progress here.

[tspearconquest]

Good afternoon @lzhecheng thanks that would be great as a method to diagnose when a cluster is already experiencing issues, however we would like to proactively check the value when the cluster is also in a healthy state, and so it would be far more useful if this can also be added to the response body JSON for the AKS API.

According to the below image, this is already supposed to be the case:

Using the az aks show command, the securityProfile.azureKeyVaultKms section shows the KMS config, including Key vault, key, current and previous key versions

However when we do the az aks show command, we can only see the current key.

This screenshot comes from this page: https://learn.microsoft.com/en-us/azure/aks/use-kms-etcd-encryption

[lzhecheng]

@tspearconquest Got your point. I double checked within the team. Now, "previous key" information is not included in managed cluster json. There are 2 actions we are going to take:

1. Update diagnose and solve to include previous key. This doesn't take much effort and should be quick to complete.
2. We are designing a few new features to help customer get rid of the effort to maintain and update keys. It should help avoid such problems resulting from mistakes. We don't have a specific ETA right now, but we are actively working on them.

[tspearconquest]

Sounds great; do you think that the previous key will be added to the JSON also during the update which adds it to the diagnose and solve panel? If not, should I submit a PR to remove the mention of the previous key from the top of that document?