New-AzResourceGroupDeployment should have a thread limit to call Deployment API.

[mmmbai]

Description

We are using New-AzResourceGroupDeployment to deploy API Management services. We have found one issue that is if the API have more than 100 operations then the deployment will fail.
For example, we have create one API using raw.githubusercontent.com/microsoft/PowerBI-CSharp/master/sdk/swaggers/swagger.json which have 278 operations. When we deploy it to API Management services using New-AzResourceGroupDeployment, the deployment will take around 2 hours and finally failed.
When we goes to Resource group->Deployment, we found New-AzResourceGroupDeployment created more than 278 calls in parallel. Then it cause some of them can not have enough resource to complete and then failed with timeout. The failed deployment will have retry triggered somewhere and cause more and more resource competition. Then later some deployment failed after more than 2 hours.
The root cause is New-AzResourceGroupDeployment(or some API it called) trigger all the operations together in parallel. Can you improve this by limit the calls for deployment? I suggest in parallel call for deployment should less than 50 then to make sure the deployment have enough resource to complete the deployment.

Issue script & Debug output

New-AzResourceGroupDeployment `
    -mode incremental `
    -name ((Get-ChildItem $templateFile).BaseName + '-' + ((Get-Date).ToUniversalTime()).ToString('MMdd-HHmmss')) `
    -ResourceGRoupName $apimInstance.ResourceGroupName `
    -templatefile $templateFile.Name `
    -TemplateParameterObject $paramSet -Force

Environment data

7.x

Module versions

6.2.1

Error output

2023-06-21T17:03:30.9702421Z ##[error]5:03:30 PM - The deployment 'apimgmt01-apim-master.template-0621-144327' failed with error(s). Showing 3 out of 155 error(s).
Status Message: The resource provision operation did not complete within the allowed timeout period. (Code:ResourceDeploymentFailure)

Status Message: The resource provision operation did not complete within the allowed timeout period. (Code:ResourceDeploymentFailure)

Status Message: The resource provision operation did not complete within the allowed timeout period. (Code:ResourceDeploymentFailure)

Tasks

Preview Give feedback

No tasks being tracked yet.

[mmmbai]

[mmmbai]

Example of one failed deployment
{

"status": "Failed",

"error": {

    "code": "ResourceDeploymentFailure",

    "message": "The resource provision operation did not complete within the allowed timeout period."

}

}

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @Azure/deployments-owners.

Issue Details

---

Description

We are using New-AzResourceGroupDeployment to deploy API Management services. We have found one issue that is if the API have more than 100 operations then the deployment will fail.
For example, we have create one API using raw.githubusercontent.com/microsoft/PowerBI-CSharp/master/sdk/swaggers/swagger.json which have 278 operations. When we deploy it to API Management services using New-AzResourceGroupDeployment, the deployment will take around 2 hours and finally failed.
When we goes to Resource group->Deployment, we found New-AzResourceGroupDeployment created more than 278 calls in parallel. Then it cause some of them can not have enough resource to complete and then failed with timeout. The failed deployment will have retry triggered somewhere and cause more and more resource competition. Then later some deployment failed after more than 2 hours.
The root cause is New-AzResourceGroupDeployment(or some API it called) trigger all the operations together in parallel. Can you improve this by limit the calls for deployment? I suggest in parallel call for deployment should less than 50 then to make sure the deployment have enough resource to complete the deployment.

Issue script & Debug output

New-AzResourceGroupDeployment `
    -mode incremental `
    -name ((Get-ChildItem $templateFile).BaseName + '-' + ((Get-Date).ToUniversalTime()).ToString('MMdd-HHmmss')) `
    -ResourceGRoupName $apimInstance.ResourceGroupName `
    -templatefile $templateFile.Name `
    -TemplateParameterObject $paramSet -Force

Environment data

7.x

Module versions

6.2.1

Error output

2023-06-21T17:03:30.9702421Z ##[error]5:03:30 PM - The deployment 'apimgmt01-apim-master.template-0621-144327' failed with error(s). Showing 3 out of 155 error(s).
Status Message: The resource provision operation did not complete within the allowed timeout period. (Code:ResourceDeploymentFailure)

Status Message: The resource provision operation did not complete within the allowed timeout period. (Code:ResourceDeploymentFailure)

Status Message: The resource provision operation did not complete within the allowed timeout period. (Code:ResourceDeploymentFailure)

### Tasks

Author:	mmmbai
Assignees:	-
Labels:	Service Attention, bug, ARM - Templates, customer-reported
Milestone:	-