Terraform plan fails when retrieving azurerm_kusto_database_principal_assignment when there are more than 300 permissions defined

[alex-slynko]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.4.4

AzureRM Provider Version

3.34+

Affected Resource(s)/Data Source(s)

azurerm_kusto_database_principal_assignment

Terraform Configuration Files

We have 36 Kusto databases defined, each has 10 different permissions.

Debug Output/Panic Output

The Terraform plan that fails with the following error:

Error: retrieving Database Principal Assignment: (Principal Assignment Name "dw-admin-01234567-0123-0123-01234568" / Database Name "db" / Cluster Name "kustoprod" / Resource Group "kustoprod"): kusto.DatabasePrincipalAssignmentsClient#Get: Failure sending request: StatusCode=429 -- Original Error: context deadline exceeded

The HTTP response from the provider that fails

{"error":{"code":"ResourceRequestsThrottled","message":"Number of requests for action 'Microsoft.Kusto/clusters/databases/principalAssignments/read' exceeded the limit of '300' for time interval '00:05:00'. Please try again after '312' seconds."}}: timestamp=2023-05-30T13:16:51.660Z
2023-05-30T13:16:52.007Z [DEBUG] provider.terraform-provider-azurerm_v3.34.0_x5: AzureRM Response for https://management.azure.com/subscriptions/xxxxxxxx-xxxx-xxxx-xxxxxxxx/resourceGroups/kustoprod/providers/Microsoft.Kusto/clusters/kustoprod/databases/db/principalAssignments/dw-admin-01234567-0123-0123-01234568?api-version=2022-02-01: 
HTTP/2.0 200 OK
Cache-Control: no-cache
Content-Type: application/json; charset=utf-8
Date: Tue, 30 May 2023 13:16:51 GMT
Expires: -1
Pragma: no-cache
Server: Microsoft-HTTPAPI/2.0
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: 055d4739-e390-64bb-c357-9abfd16a60bf
X-Ms-Ratelimit-Remaining-Subscription-Resource-Requests: -1
X-Ms-Request-Id: c5326f13-5b5c-48e9-82f1-5b275d65e6e0
X-Ms-Routing-Request-Id: EASTUS:20230530T131651Z:701aa983-7f84-482a-9210-5c2afab39371

Expected Behaviour

Terraform plan succeeds

Actual Behaviour

Terraform plan fails.

Steps to Reproduce

1. terraform plan on the workspace with defined permissions

Important Factoids

No response

References

No response

[liuwuliuyun]

Hi @alex-slynko , thanks for raising this issue. From the error message, it looks like the root cause of this issue is API rate limitation of getting principal assignements. A potential solution is to retry after thottling, I will try to test it locally. In the meantime, I will try to contact Kusto team about whether they have a workaround. Thanks.

[liuwuliuyun]

Hi @alex-slynko sorry for the late response. Could you try the custom timeouts? It should solve you issue here. Thanks.

[alex-slynko]

The custom timeouts do not work for our case, since the timeout happens during terraform plan phase when we are receiving data from Azure.

[tombuildsstuff]

@alex-slynko since this issue is using an older version of the Provider and these resources have since been switched over to using our new SDK and base layer, we should be handling these retries better (although I suspect the issue is the 5m read timeout, given the API throttling/limitations) - but it'd be good to confirm if that's the case with the latest base layer too - so would you mind upgrading to the latest version of the provider and confirming if that's still an issue for you? Thanks!

[alex-slynko]

@tombuildsstuff
I still see the same error when running terraform plan using 3.68

databaseprincipalassignments.DatabasePrincipalAssignmentsClient#Get: Failure sending request: StatusCode=429 -- Original Error: context deadline exceeded

[alex-slynko]

One more annoying thing with this bug.
We use Terraform Cloud and when the run fails in the middle of apply, we lose some of the permissions from the state.