azurerm_kusto_database_principal_assignment: Error: waiting for creation of Database Principal Assignment when using User Assigned Managed Identity

[jamesbwilkinson]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.2.3

AzureRM Provider Version

3.7.0

Affected Resource(s)/Data Source(s)

azurerm_kusto_database_principal_assignment

Terraform Configuration Files

resource "azurerm_user_assigned_identity" "api_identity" {
  resource_group_name = var.resource_group_name
  location            = var.location
  name                = "id-example"
}

resource "azurerm_kusto_database_principal_assignment" "adx_principal_assignment" {
  name                = "adx-example-viewer"
  resource_group_name = var.resource_group_name
  cluster_name        = var.adx_cluster_name
  database_name       = var.example_db

  tenant_id      = data.azurerm_client_config.current.tenant_id
  principal_id   = azurerm_user_assigned_identity.api_identity.principal_id
  principal_type = "App"
  role           = "Viewer"
}

Debug Output/Panic Output

Error: waiting for creation of Database Principal Assignment: (Principal Assignment Name "adx-example-viewer" / Database Name "example_db" / Cluster Name "examplecluster" / Resource Group "example-rg"): Code="BadInput" Message="[BadRequest] Entity ID 'xxxxxx-xxxx' of type 'AAD Application Id' was not found in AAD tenant 'xxxxxx'."

Expected Behaviour

I would expect the assignment to complete.

Actual Behaviour

The ADX assignment is not able to find the managed identity in the tenant. This seems to be a timing issue as the ID is available if manually searching AAD. After creation of the managed identity it must take some time for it to be available to ADX.

Steps to Reproduce

No response

Important Factoids

No response

References

No response

[liuwuliuyun]

Hi @jamesbwilkinson , I have tested locally with your config in version 3.22.0 and it works fine. Here is my config.

provider "azurerm" {
features {
}
}

data "azurerm_client_config" "current" {}

resource "azurerm_resource_group" "example" {
name = "KustoRGyun"
location = "West Europe"
}

resource "azurerm_user_assigned_identity" "user_identity" {
name = "user_assigned_identity"
resource_group_name = azurerm_resource_group.example.name
location = azurerm_resource_group.example.location
}

resource "azurerm_kusto_cluster" "example" {
name = "kustoclusteryun111"
location = azurerm_resource_group.example.location
resource_group_name = azurerm_resource_group.example.name
identity {
type = "UserAssigned"
identity_ids = [ azurerm_user_assigned_identity.user_identity.id ]
}
sku {
name = "Standard_D13_v2"
capacity = 2
}
}

resource "azurerm_kusto_database" "example" {
name = "KustoDatabaseyun111"
resource_group_name = azurerm_resource_group.example.name
location = azurerm_resource_group.example.location
cluster_name = azurerm_kusto_cluster.example.name

hot_cache_period = "P7D"
soft_delete_period = "P31D"
}

resource "azurerm_kusto_database_principal_assignment" "example" {
name = "KustoPrincipalAssignmentyun111"
resource_group_name = azurerm_resource_group.example.name
cluster_name = azurerm_kusto_cluster.example.name
database_name = azurerm_kusto_database.example.name

tenant_id = data.azurerm_client_config.current.tenant_id
principal_id = azurerm_user_assigned_identity.user_identity.principal_id
principal_type = "App"
role = "Viewer"
}

Could you try with a higher version or check whether you assign this user_identity to kusto_cluster? If you still receving this error, welcome to leave a feedback here.

[liuwuliuyun]

I could confirm above config works in 3.7.0 as well :)

[jamesbwilkinson]

Hi @liuwuliuyun
Thank you for the suggestions. Our scenario is different from what you tested.
We create the kusto cluster in a different stage of our deployment, before we create the managed identity so we are not able to assign it to the cluster like you show in your example. We create the MI and assign it at the same stage with azurerm_kusto_database_principal_assignment.

Can you check it in this way?

[liuwuliuyun]

I am not sure how to use terraform in different stages, maybe other community members could give solid suggestions here. But I think you could update the cluster before assign the user_identity using azurerm_kusto_database_principal_assignment, that should work like above in theory. To make sure the assignment happens after the MI associated with cluster, you could try the depend on property of terraform

[Meandron]

To add to this:
We create the cluster that can be used by various services (other stages, like James mentioned) as base for their Terraform based infrastructural runs. So, by the time we create the cluster through TF, we don't know what/how many managed identites there will assigned via azurerm_kusto_database_principal_assignment later nor is it possible, if a certain service's infrastructure gets created and with that, its MI, to update the identities on the cluster as the complete MI list is unknown to that very service. So, with TF capabilities, I don't think it's possible to update the cluster's identities like you proposed.

But anyhow: Could you try the approach without predefining the identities on the cluster, say, skip setting of "identity_ids" to check whether you can reproduce the issue? It worked on our side, but only after several trials. So maybe there is something that could be improved here.

[liuwuliuyun]

I tried following config on myside with version 3.7.0

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "3.7.0"
    }
  }
}

provider "azurerm" {
  features {
  }
}

data "azurerm_client_config" "current" {}

resource "azurerm_resource_group" "example" {
  name     = "KustoRGyun"
  location = "West Europe"
}

resource "azurerm_user_assigned_identity" "user_identity" {
  name = "user_assigned_identity"
  resource_group_name = azurerm_resource_group.example.name
  location = azurerm_resource_group.example.location
}

resource "azurerm_kusto_cluster" "example" {
  name                = "kustoclusteryun111"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
  sku {
    name     = "Standard_D13_v2"
    capacity = 2
  }
}


resource "azurerm_kusto_database" "example" {
  name                = "KustoDatabaseyun111"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  cluster_name        = azurerm_kusto_cluster.example.name

  hot_cache_period   = "P7D"
  soft_delete_period = "P31D"
}

resource "azurerm_kusto_database_principal_assignment" "example" {
  name                = "KustoPrincipalAssignmentyun111"
  resource_group_name = azurerm_resource_group.example.name
  cluster_name        = azurerm_kusto_cluster.example.name
  database_name       = azurerm_kusto_database.example.name

  tenant_id      = data.azurerm_client_config.current.tenant_id
  principal_id   = azurerm_user_assigned_identity.user_identity.principal_id
  principal_type = "App"
  role           = "Viewer"
}

After plan & apply and no error is promoted the first time.

When I reapply after that, it promotes no change which is expected.

[liuwuliuyun]

If someone has the same issue and has a way to reproduce, welcome to leave a comments with steps :)

[jamesbwilkinson]

Hi @liuwuliuyun,

Thanks for your suggestion. I have tried to reproduce the issue using your example, and have done this in 2 ways:

1. If I take your example above and run as it, ie create the ADX cluster, db, MI and db assignment in the same terraform run then everything works as you experienced. As described previously this is not how we are working.

2. If I take your example and create the cluster and db in 1 terraform run, then in a 2nd run add the MI and database principal assignment I get an error which describes the MI is not available, eg Entity ID 'xxx' of type 'AAD Application Id' was not found in AAD tenant.

My assumption here is that in attempt 1, the MI is created long before the ADX, as the ADX takes some minutes to become available. After this assignment is successful as the MI can be found. As in step 2 the time taken to create the MI and do the assignment appears not to be enough for the MI to be properly available in AAD. It appears that some delay/retry is needed between the MI creation and assignment for ADX.

Could you test it from your-side as I did in attempt 2?

[AmyJeanes]

We are also intermittently seeing this issue too, although interestingly only after upgrading from Terraform 1.1.7 to 1.3.4 but not sure if that is the cause. In our scenario we have an existing azurerm_kusto_cluster referenced by a Terraform data source which we then create a azurerm_kusto_database on and then a azuread_service_principal and corresponding azurerm_kusto_database_principal_assignment on that which is failing with this (sanitised) error message:

Error: waiting for creation of Database Principal Assignment: (Principal Assignment Name "AdxAppPrincipalAssignment" / Database Name "xxxx" / Cluster Name "xxxx" / Resource Group "xxxx"): Code="BadInput" Message="[BadRequest] Entity ID 'xxxxxxx-af29-47e4-8ac3-xxxxxxxx' of type 'AAD Application Id' was not found in AAD tenant 'xxxxxxx-44b6-4aa7-ae18-xxxxxxxxxxx'.

We are currently on azurerm 3.26.0 but I see no mention of a fix in newer version release notes for this issue.

[liuwuliuyun]

Hi @jamesbwilkinson I think your assumption is correct. In provider, we parrallel creating the resources except using depend_on to explicitly state the dependency.

[liuwuliuyun]

I tried your attempt and it succeed on myside. But I still think you are correct, could you try to make assignment depend_on the resource user_identity so that it will only be created after user_identity is created? For other resources having the same problem, similar approach could be tried.

terraform apply --auto-approve
data.azurerm_client_config.current: Reading...
azurerm_resource_group.example: Refreshing state... [id=/subscriptions/85b3dbca-5974-4067-9669-67a141095a76/resourceGroups/KustoRGyun]
data.azurerm_client_config.current: Read complete after 0s [id=2022-11-11 04:14:11.3255693 +0000 UTC]
azurerm_kusto_cluster.example: Refreshing state... [id=/subscriptions/85b3dbca-5974-4067-9669-67a141095a76/resourceGroups/KustoRGyun/providers/Microsoft.Kusto/Clusters/kustoclusteryun111]
azurerm_kusto_database.example: Refreshing state... [id=/subscriptions/85b3dbca-5974-4067-9669-67a141095a76/resourceGroups/KustoRGyun/providers/Microsoft.Kusto/Clusters/kustoclusteryun111/Databases/KustoDatabaseyun111]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place

Terraform will perform the following actions:

  # azurerm_kusto_cluster.example will be updated in-place
  ~ resource "azurerm_kusto_cluster" "example" {
        id                            = "/subscriptions/85b3dbca-5974-4067-9669-67a141095a76/resourceGroups/KustoRGyun/providers/Microsoft.Kusto/Clusters/kustoclusteryun111"
        name                          = "kustoclusteryun111"
        tags                          = {}
        # (14 unchanged attributes hidden)

      + identity {
          + identity_ids = (known after apply)
          + type         = "UserAssigned"
        }

        # (1 unchanged block hidden)
    }

  # azurerm_kusto_database_principal_assignment.example will be created
  + resource "azurerm_kusto_database_principal_assignment" "example" {
      + cluster_name        = "kustoclusteryun111"
      + database_name       = "KustoDatabaseyun111"
      + id                  = (known after apply)
      + name                = "KustoPrincipalAssignmentyun111"
      + principal_id        = (known after apply)
      + principal_name      = (known after apply)
      + principal_type      = "App"
      + resource_group_name = "KustoRGyun"
      + role                = "Viewer"
      + tenant_id           = "72f988bf-86f1-41af-91ab-2d7cd011db47"
      + tenant_name         = (known after apply)
    }

  # azurerm_user_assigned_identity.user_identity will be created
  + resource "azurerm_user_assigned_identity" "user_identity" {
      + client_id           = (known after apply)
      + id                  = (known after apply)
      + location            = "westeurope"
      + name                = "user_assigned_identity"
      + principal_id        = (known after apply)
      + resource_group_name = "KustoRGyun"
      + tenant_id           = (known after apply)
    }

Plan: 2 to add, 1 to change, 0 to destroy.
azurerm_user_assigned_identity.user_identity: Creating...
azurerm_user_assigned_identity.user_identity: Creation complete after 8s [id=/subscriptions/85b3dbca-5974-4067-9669-67a141095a76/resourceGroups/KustoRGyun/providers/Microsoft.ManagedIdentity/userAssignedIdentities/user_assigned_identity]
azurerm_kusto_cluster.example: Modifying... [id=/subscriptions/85b3dbca-5974-4067-9669-67a141095a76/resourceGroups/KustoRGyun/providers/Microsoft.Kusto/Clusters/kustoclusteryun111]
azurerm_kusto_cluster.example: Still modifying... [id=/subscriptions/85b3dbca-5974-4067-9669-...soft.Kusto/Clusters/kustoclusteryun111, 10s elapsed]
azurerm_kusto_cluster.example: Still modifying... [id=/subscriptions/85b3dbca-5974-4067-9669-...soft.Kusto/Clusters/kustoclusteryun111, 20s elapsed]
azurerm_kusto_cluster.example: Still modifying... [id=/subscriptions/85b3dbca-5974-4067-9669-...soft.Kusto/Clusters/kustoclusteryun111, 30s elapsed]
azurerm_kusto_cluster.example: Still modifying... [id=/subscriptions/85b3dbca-5974-4067-9669-...soft.Kusto/Clusters/kustoclusteryun111, 40s elapsed]
azurerm_kusto_cluster.example: Still modifying... [id=/subscriptions/85b3dbca-5974-4067-9669-...soft.Kusto/Clusters/kustoclusteryun111, 50s elapsed]
azurerm_kusto_cluster.example: Still modifying... [id=/subscriptions/85b3dbca-5974-4067-9669-...soft.Kusto/Clusters/kustoclusteryun111, 1m0s elapsed]
azurerm_kusto_cluster.example: Modifications complete after 1m7s [id=/subscriptions/85b3dbca-5974-4067-9669-67a141095a76/resourceGroups/KustoRGyun/providers/Microsoft.Kusto/Clusters/kustoclusteryun111]
azurerm_kusto_database_principal_assignment.example: Creating...
azurerm_kusto_database_principal_assignment.example: Still creating... [10s elapsed]
azurerm_kusto_database_principal_assignment.example: Still creating... [20s elapsed]
azurerm_kusto_database_principal_assignment.example: Still creating... [30s elapsed]
azurerm_kusto_database_principal_assignment.example: Creation complete after 36s [id=/subscriptions/85b3dbca-5974-4067-9669-67a141095a76/resourceGroups/KustoRGyun/providers/Microsoft.Kusto/Clusters/kustoclusteryun111/Databases/KustoDatabaseyun111/PrincipalAssignments/KustoPrincipalAssignmentyun111]

Apply complete! Resources: 2 added, 1 changed, 0 destroyed.