azurerm_kusto_database_principal_assignment: Error: waiting for creation of Database Principal Assignment when using User Assigned Managed Identity

[jamesbwilkinson]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.2.3

AzureRM Provider Version

3.7.0

Affected Resource(s)/Data Source(s)

azurerm_kusto_database_principal_assignment

Terraform Configuration Files

resource "azurerm_user_assigned_identity" "api_identity" {
  resource_group_name = var.resource_group_name
  location            = var.location
  name                = "id-example"
}

resource "azurerm_kusto_database_principal_assignment" "adx_principal_assignment" {
  name                = "adx-example-viewer"
  resource_group_name = var.resource_group_name
  cluster_name        = var.adx_cluster_name
  database_name       = var.example_db

  tenant_id      = data.azurerm_client_config.current.tenant_id
  principal_id   = azurerm_user_assigned_identity.api_identity.principal_id
  principal_type = "App"
  role           = "Viewer"
}

Debug Output/Panic Output

Error: waiting for creation of Database Principal Assignment: (Principal Assignment Name "adx-example-viewer" / Database Name "example_db" / Cluster Name "examplecluster" / Resource Group "example-rg"): Code="BadInput" Message="[BadRequest] Entity ID 'xxxxxx-xxxx' of type 'AAD Application Id' was not found in AAD tenant 'xxxxxx'."

Expected Behaviour

I would expect the assignment to complete.

Actual Behaviour

The ADX assignment is not able to find the managed identity in the tenant. This seems to be a timing issue as the ID is available if manually searching AAD. After creation of the managed identity it must take some time for it to be available to ADX.

Steps to Reproduce

No response

Important Factoids

No response

References

No response

[liuwuliuyun]

Hi @jamesbwilkinson , I have tested locally with your config in version 3.22.0 and it works fine. Here is my config.

provider "azurerm" {
features {
}
}

data "azurerm_client_config" "current" {}

resource "azurerm_resource_group" "example" {
name = "KustoRGyun"
location = "West Europe"
}

resource "azurerm_user_assigned_identity" "user_identity" {
name = "user_assigned_identity"
resource_group_name = azurerm_resource_group.example.name
location = azurerm_resource_group.example.location
}

resource "azurerm_kusto_cluster" "example" {
name = "kustoclusteryun111"
location = azurerm_resource_group.example.location
resource_group_name = azurerm_resource_group.example.name
identity {
type = "UserAssigned"
identity_ids = [ azurerm_user_assigned_identity.user_identity.id ]
}
sku {
name = "Standard_D13_v2"
capacity = 2
}
}

resource "azurerm_kusto_database" "example" {
name = "KustoDatabaseyun111"
resource_group_name = azurerm_resource_group.example.name
location = azurerm_resource_group.example.location
cluster_name = azurerm_kusto_cluster.example.name

hot_cache_period = "P7D"
soft_delete_period = "P31D"
}

resource "azurerm_kusto_database_principal_assignment" "example" {
name = "KustoPrincipalAssignmentyun111"
resource_group_name = azurerm_resource_group.example.name
cluster_name = azurerm_kusto_cluster.example.name
database_name = azurerm_kusto_database.example.name

tenant_id = data.azurerm_client_config.current.tenant_id
principal_id = azurerm_user_assigned_identity.user_identity.principal_id
principal_type = "App"
role = "Viewer"
}

Could you try with a higher version or check whether you assign this user_identity to kusto_cluster? If you still receving this error, welcome to leave a feedback here.

[liuwuliuyun]

I could confirm above config works in 3.7.0 as well :)

[jamesbwilkinson]

Hi @liuwuliuyun
Thank you for the suggestions. Our scenario is different from what you tested.
We create the kusto cluster in a different stage of our deployment, before we create the managed identity so we are not able to assign it to the cluster like you show in your example. We create the MI and assign it at the same stage with azurerm_kusto_database_principal_assignment.

Can you check it in this way?

[liuwuliuyun]

I am not sure how to use terraform in different stages, maybe other community members could give solid suggestions here. But I think you could update the cluster before assign the user_identity using azurerm_kusto_database_principal_assignment, that should work like above in theory. To make sure the assignment happens after the MI associated with cluster, you could try the depend on property of terraform

[Meandron]

To add to this:
We create the cluster that can be used by various services (other stages, like James mentioned) as base for their Terraform based infrastructural runs. So, by the time we create the cluster through TF, we don't know what/how many managed identites there will assigned via azurerm_kusto_database_principal_assignment later nor is it possible, if a certain service's infrastructure gets created and with that, its MI, to update the identities on the cluster as the complete MI list is unknown to that very service. So, with TF capabilities, I don't think it's possible to update the cluster's identities like you proposed.

But anyhow: Could you try the approach without predefining the identities on the cluster, say, skip setting of "identity_ids" to check whether you can reproduce the issue? It worked on our side, but only after several trials. So maybe there is something that could be improved here.

[liuwuliuyun]

I tried following config on myside with version 3.7.0

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "3.7.0"
    }
  }
}

provider "azurerm" {
  features {
  }
}

data "azurerm_client_config" "current" {}

resource "azurerm_resource_group" "example" {
  name     = "KustoRGyun"
  location = "West Europe"
}

resource "azurerm_user_assigned_identity" "user_identity" {
  name = "user_assigned_identity"
  resource_group_name = azurerm_resource_group.example.name
  location = azurerm_resource_group.example.location
}

resource "azurerm_kusto_cluster" "example" {
  name                = "kustoclusteryun111"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
  sku {
    name     = "Standard_D13_v2"
    capacity = 2
  }
}


resource "azurerm_kusto_database" "example" {
  name                = "KustoDatabaseyun111"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  cluster_name        = azurerm_kusto_cluster.example.name

  hot_cache_period   = "P7D"
  soft_delete_period = "P31D"
}

resource "azurerm_kusto_database_principal_assignment" "example" {
  name                = "KustoPrincipalAssignmentyun111"
  resource_group_name = azurerm_resource_group.example.name
  cluster_name        = azurerm_kusto_cluster.example.name
  database_name       = azurerm_kusto_database.example.name

  tenant_id      = data.azurerm_client_config.current.tenant_id
  principal_id   = azurerm_user_assigned_identity.user_identity.principal_id
  principal_type = "App"
  role           = "Viewer"
}

After plan & apply and no error is promoted the first time.

When I reapply after that, it promotes no change which is expected.

[liuwuliuyun]

If someone has the same issue and has a way to reproduce, welcome to leave a comments with steps :)

[jamesbwilkinson]

Hi @liuwuliuyun,

Thanks for your suggestion. I have tried to reproduce the issue using your example, and have done this in 2 ways:

1. If I take your example above and run as it, ie create the ADX cluster, db, MI and db assignment in the same terraform run then everything works as you experienced. As described previously this is not how we are working.

2. If I take your example and create the cluster and db in 1 terraform run, then in a 2nd run add the MI and database principal assignment I get an error which describes the MI is not available, eg Entity ID 'xxx' of type 'AAD Application Id' was not found in AAD tenant.

My assumption here is that in attempt 1, the MI is created long before the ADX, as the ADX takes some minutes to become available. After this assignment is successful as the MI can be found. As in step 2 the time taken to create the MI and do the assignment appears not to be enough for the MI to be properly available in AAD. It appears that some delay/retry is needed between the MI creation and assignment for ADX.

Could you test it from your-side as I did in attempt 2?

[MattJeanes]

We are also intermittently seeing this issue too, although interestingly only after upgrading from Terraform 1.1.7 to 1.3.4 but not sure if that is the cause. In our scenario we have an existing azurerm_kusto_cluster referenced by a Terraform data source which we then create a azurerm_kusto_database on and then a azuread_service_principal and corresponding azurerm_kusto_database_principal_assignment on that which is failing with this (sanitised) error message:

Error: waiting for creation of Database Principal Assignment: (Principal Assignment Name "AdxAppPrincipalAssignment" / Database Name "xxxx" / Cluster Name "xxxx" / Resource Group "xxxx"): Code="BadInput" Message="[BadRequest] Entity ID 'xxxxxxx-af29-47e4-8ac3-xxxxxxxx' of type 'AAD Application Id' was not found in AAD tenant 'xxxxxxx-44b6-4aa7-ae18-xxxxxxxxxxx'.

We are currently on azurerm 3.26.0 but I see no mention of a fix in newer version release notes for this issue.

[liuwuliuyun]

Hi @jamesbwilkinson I think your assumption is correct. In provider, we parrallel creating the resources except using depend_on to explicitly state the dependency.

[liuwuliuyun]

I tried your attempt and it succeed on myside. But I still think you are correct, could you try to make assignment depend_on the resource user_identity so that it will only be created after user_identity is created? For other resources having the same problem, similar approach could be tried.

terraform apply --auto-approve
data.azurerm_client_config.current: Reading...
azurerm_resource_group.example: Refreshing state... [id=/subscriptions/85b3dbca-5974-4067-9669-67a141095a76/resourceGroups/KustoRGyun]
data.azurerm_client_config.current: Read complete after 0s [id=2022-11-11 04:14:11.3255693 +0000 UTC]
azurerm_kusto_cluster.example: Refreshing state... [id=/subscriptions/85b3dbca-5974-4067-9669-67a141095a76/resourceGroups/KustoRGyun/providers/Microsoft.Kusto/Clusters/kustoclusteryun111]
azurerm_kusto_database.example: Refreshing state... [id=/subscriptions/85b3dbca-5974-4067-9669-67a141095a76/resourceGroups/KustoRGyun/providers/Microsoft.Kusto/Clusters/kustoclusteryun111/Databases/KustoDatabaseyun111]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place

Terraform will perform the following actions:

  # azurerm_kusto_cluster.example will be updated in-place
  ~ resource "azurerm_kusto_cluster" "example" {
        id                            = "/subscriptions/85b3dbca-5974-4067-9669-67a141095a76/resourceGroups/KustoRGyun/providers/Microsoft.Kusto/Clusters/kustoclusteryun111"
        name                          = "kustoclusteryun111"
        tags                          = {}
        # (14 unchanged attributes hidden)

      + identity {
          + identity_ids = (known after apply)
          + type         = "UserAssigned"
        }

        # (1 unchanged block hidden)
    }

  # azurerm_kusto_database_principal_assignment.example will be created
  + resource "azurerm_kusto_database_principal_assignment" "example" {
      + cluster_name        = "kustoclusteryun111"
      + database_name       = "KustoDatabaseyun111"
      + id                  = (known after apply)
      + name                = "KustoPrincipalAssignmentyun111"
      + principal_id        = (known after apply)
      + principal_name      = (known after apply)
      + principal_type      = "App"
      + resource_group_name = "KustoRGyun"
      + role                = "Viewer"
      + tenant_id           = "72f988bf-86f1-41af-91ab-2d7cd011db47"
      + tenant_name         = (known after apply)
    }

  # azurerm_user_assigned_identity.user_identity will be created
  + resource "azurerm_user_assigned_identity" "user_identity" {
      + client_id           = (known after apply)
      + id                  = (known after apply)
      + location            = "westeurope"
      + name                = "user_assigned_identity"
      + principal_id        = (known after apply)
      + resource_group_name = "KustoRGyun"
      + tenant_id           = (known after apply)
    }

Plan: 2 to add, 1 to change, 0 to destroy.
azurerm_user_assigned_identity.user_identity: Creating...
azurerm_user_assigned_identity.user_identity: Creation complete after 8s [id=/subscriptions/85b3dbca-5974-4067-9669-67a141095a76/resourceGroups/KustoRGyun/providers/Microsoft.ManagedIdentity/userAssignedIdentities/user_assigned_identity]
azurerm_kusto_cluster.example: Modifying... [id=/subscriptions/85b3dbca-5974-4067-9669-67a141095a76/resourceGroups/KustoRGyun/providers/Microsoft.Kusto/Clusters/kustoclusteryun111]
azurerm_kusto_cluster.example: Still modifying... [id=/subscriptions/85b3dbca-5974-4067-9669-...soft.Kusto/Clusters/kustoclusteryun111, 10s elapsed]
azurerm_kusto_cluster.example: Still modifying... [id=/subscriptions/85b3dbca-5974-4067-9669-...soft.Kusto/Clusters/kustoclusteryun111, 20s elapsed]
azurerm_kusto_cluster.example: Still modifying... [id=/subscriptions/85b3dbca-5974-4067-9669-...soft.Kusto/Clusters/kustoclusteryun111, 30s elapsed]
azurerm_kusto_cluster.example: Still modifying... [id=/subscriptions/85b3dbca-5974-4067-9669-...soft.Kusto/Clusters/kustoclusteryun111, 40s elapsed]
azurerm_kusto_cluster.example: Still modifying... [id=/subscriptions/85b3dbca-5974-4067-9669-...soft.Kusto/Clusters/kustoclusteryun111, 50s elapsed]
azurerm_kusto_cluster.example: Still modifying... [id=/subscriptions/85b3dbca-5974-4067-9669-...soft.Kusto/Clusters/kustoclusteryun111, 1m0s elapsed]
azurerm_kusto_cluster.example: Modifications complete after 1m7s [id=/subscriptions/85b3dbca-5974-4067-9669-67a141095a76/resourceGroups/KustoRGyun/providers/Microsoft.Kusto/Clusters/kustoclusteryun111]
azurerm_kusto_database_principal_assignment.example: Creating...
azurerm_kusto_database_principal_assignment.example: Still creating... [10s elapsed]
azurerm_kusto_database_principal_assignment.example: Still creating... [20s elapsed]
azurerm_kusto_database_principal_assignment.example: Still creating... [30s elapsed]
azurerm_kusto_database_principal_assignment.example: Creation complete after 36s [id=/subscriptions/85b3dbca-5974-4067-9669-67a141095a76/resourceGroups/KustoRGyun/providers/Microsoft.Kusto/Clusters/kustoclusteryun111/Databases/KustoDatabaseyun111/PrincipalAssignments/KustoPrincipalAssignmentyun111]

Apply complete! Resources: 2 added, 1 changed, 0 destroyed.

[jamesbwilkinson]

Hi @liuwuliuyun ,
As suggested I tried with "depends on" but I still see the same error.

[liuwuliuyun]

That is weird, I did not get any error like that. Here is the suggested config I menthoned.

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "3.7.0"
    }
  }
}

provider "azurerm" {
  features {
  }
}

data "azurerm_client_config" "current" {}

resource "azurerm_resource_group" "example" {
  name     = "KustoRGyun"
  location = "West Europe"
}

resource "azurerm_kusto_cluster" "example" {
  name                = "kustoclusteryun111"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
  sku {
    name     = "Standard_D13_v2"
    capacity = 2
  }
}


resource "azurerm_kusto_database" "example" {
  name                = "KustoDatabaseyun111"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  cluster_name        = azurerm_kusto_cluster.example.name

  hot_cache_period   = "P7D"
  soft_delete_period = "P31D"
}

resource "azurerm_kusto_database_principal_assignment" "example" {
  name                = "KustoPrincipalAssignmentyun111"
  resource_group_name = azurerm_resource_group.example.name
  cluster_name        = azurerm_kusto_cluster.example.name
  database_name       = azurerm_kusto_database.example.name

  tenant_id      = data.azurerm_client_config.current.tenant_id
  principal_id   = azurerm_user_assigned_identity.user_identity.principal_id
  principal_type = "App"
  role           = "Viewer"

  depends_on = [
    azurerm_user_assigned_identity.user_identity
  ]
}

resource "azurerm_user_assigned_identity" "user_identity" {
  name = "user_assigned_identity"
  resource_group_name = azurerm_resource_group.example.name
  location = azurerm_resource_group.example.location
}

After that I run terraform graph to see the dependency order with following results:

As it can show that database assignments will be created only user_identity exists. Plus, you can remove the identity_ids block in kusto_cluster and it also works. My local terraform version is 1.2.3 but I dont think that is the problem.

[MattJeanes]

The dependency graph is correct but it seems that it kusto is not always able to immediately see an app registration after it is created and needs some amount of time before it works. It must be correct as in order to know the id of the app registration to use in the principal assignment it must have created it first.

For now I am working around this by adding the database principal assignment outside of terraform but it's not ideal. Interestingly I found that doing it through Az PowerShell takes quite some time before it will error due to an incorrect id so I wonder if there is some kind of internal delay / wait to ensure new app registrations are detected.

For us we are seeing that this issue is very much intermittent and we are not sure if the error message comes from the azure backplane or from something inside terraform.

[jamesbwilkinson]

It appears to be an error from Azure API, a review of the Terraform logs shows the resource cannot be found:
{"error":{"code":"ResourceNotFound","message":"The resource with identifier '/subscriptions/XXX is not found."}}

[msterin]

I am getting this error consistently when azurerm_kusto_database_principal_assignment is created right after the azurerm_user_assigned_identity

• We generate our "environment" once, and it includes the ADX cluster, so it has no idea about future identities
• as we go, we add/remove components which need a dedicated DB , new identity and it's assigment. When we add a component, we ALWAYS hit this error. Rerunning 'terraform apply' in 10-15 minutes always helps.

We 'd like to avoid assignment creation outside of terraform, or some hacky wait with "external" data source using a CLI ..

Any suggestions?

versions installed:

• azurerm v3.92.0
• Terraform v1.7.4 on darwin_arm64

Error msg (100% reproducible)

Principal Assignment Name: "some-new-assignment-name"): polling after 
CreateOrUpdate: polling failed: the Azure API returned the following error:
│ 
│ Status: "Failed"
│ Code: "ResourceNotFound"
│ Message: "[NotFound] Failed to run add principal assignment action for service 
Engine-<our adx cluster name here>. AAD principal was not found."