Cannot set System assigned to a VMSS for a Kubernetes cluster node pool or Access to set to KV

[patrickduboucher]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

When creating an azurerm_kubernetes_cluster, the default_node_pool takes a type that defaults to VirtualMachineScaleSets
I would like to enable SystemAssigned for the Identity for the underlying VMSS that is created.

I also want to be able to enable SystemAssigned for any other node pool created in this cluster.

Why I want this:
the pods we are creating take the Identity from the VMSS when they call the metadata service.

time="15-04-2021 15:39:06" level=fatal msg="can't fetch keyvault key <redacted> : azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://<redacted>.vault.azure.net/secrets/<redacted>/?api-version=2016-10-01: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: {\"error\":\"invalid_request\",\"error_description\":\"Identity not found\"} Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fvault.azure.net"

The call doesn't include an identity unless the VMSS has system Identity enabled.
This is not enabled by default, and only can be set outside of terraform, or via some worksaround where we retreive the VMSS via the resource group of the cluster nodes.

The VMSS and Identity should also be exported in a data block as they need to be assignabled to other resources in role assignments

New or Affected Resource(s)

• azurerm_kubernetes_cluster
• azurerm_kubernetes_cluster (Data)

Potential Terraform Configuration

option1)
the cluster identity could be inherited to the VMSS

resource "azurerm_kubernetes_cluster" "cluster" {
  resource_group_name = azurerm_resource_group.main.name
  node_resource_group = "cluster-nodes"

  default_node_pool {
    vnet_subnet_id = azurerm_subnet.main.id
    type       = "VirtualMachineScaleSets"
    }
  
  identity {
    type = "SystemAssigned" #<- this identity could also be passed to the VMSS 
  } 

option2)

resource "azurerm_kubernetes_cluster" "main" {
  resource_group_name = azurerm_resource_group.main.name
  node_resource_group = "cluster-nodes"

  default_node_pool {
    vnet_subnet_id = azurerm_subnet.main.id
    type              = "VirtualMachineScaleSets"
    vmss_identity   = "SystemAssigned"
    
    } 

Data and role assignement / azurerm_key_vault_access_policy

resource "azurerm_key_vault_access_policy" "main" { 
  key_vault_id = var.key_vault_id 
  object_id         = data.azurerm_kubernetes_cluster.main.default_node_pool.vmss_identity.id
  key_permissions = [
    "Get",
  ]

  secret_permissions = [
    "Get",
  ]

} 

[favoretti]

Hi there and thank you for reporting this. You can fetch kubelet identity using the following example:

resource "azurerm_role_assignment" "acr_image_puller" {
  for_each             = var.registries
  scope                = each.value
  role_definition_name = "AcrPull"
  principal_id         = azurerm_kubernetes_cluster.this.kubelet_identity[0].object_id
}

[patrickduboucher]

edited for format: Hi, thanks for the reply. I am already retrieving the kubelet identity and assigning it to the (in my case) keyvault for access. However the applications in the pods are retrieving the VMSS's SystemAssigned Identity when calling the metadata service, and so since the VMSS System Identity is not enabled, we get the Identity Not Found error.

…

On Thu, Apr 15, 2021 at 14:27 Vladimir Lazarenko ***@***.***> wrote: Hi there and thank you for reporting this. You can fetch kubelet identity using the following example: resource "azurerm_role_assignment" "acr_image_puller" { for_each = var.registries scope = each.value role_definition_name = "AcrPull" principal_id = azurerm_kubernetes_cluster.this.kubelet_identity[0].object_id } — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#11345 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIRFV76KINYMTRIL36IZIVLTI4VZRANCNFSM427ZJBDA> .

[raguldevops]

Found any solution for this issue ?