Need better async support for New-AzureRmResourceGroupDeployment

[KirkMunro]

I have an issue with New-AzureRmResourceGroupDeployment that I'm hoping someone can help me with. I use PowerShell in Azure Functions, where the maximum amount of time that a function can run is 10 minutes (5 by default, but extensible). In some of the scripts I run through Azure Functions, I invoke New-AzureRmResourceGroupDeployment, which may take a really long time depending on what is being deployed. By default this cmdlet blocks, waiting for the results to come back, but you can invoke it with -AsJob to run the deployment in the background and continue working. That sort of helps, and I realize that was added to provide better async support, however it's still not sufficient in this case because the Azure Function can still only run for 10 minutes before timing out, and the job will still block until the deployment completes.

What I would much rather see is a way to invoke New-AzureRmResourceGroupDeployment such that it supports either one or both of the following:

• returns details (a web endpoint) immediately about where I can go check the deployment, without waiting for deployment results and without a background job
• accepts a callback web endpoint that will be invoked by Azure once the deployment is complete (successful or not)

In cases where you want to kick off a (deployment) task that may be very long running, where you don't need to or want to (or you simply cannot due to timeouts) consume any compute resources until the task has completed, this would be a very welcome feature for New-AzureRmResourceGroupDeployment and any other cmdlets that launch tasks which may take a long time to complete inside of Azure without requiring the use of local compute resources other than to report when the task completes.

[KirkMunro]

Just to add to this, since I am using Azure Functions, ideally I'd love to give New-AzureRmResourceGroupDeployment an Azure Function endpoint in my Azure Function app (a webhook) that would be invoked when the deployment completes. Then I'd have one Azure Function which launches the deployment and returns success/fail based on whether or not the deployment launched successfully, and another Azure Function that would be invoked via HTTP POST when the deployment is complete, sending the deployment details (and perhaps some additional details provided with the deployment to uniquely distinguish it from other deployments and know exactly which deployment it represents) in the message body to some webhook endpoint that would have to return results in a very short period of time (like GitHub does with webhooks, where you have to respond within 10 seconds in order for the webhook invocation to be considered successful).

[KirkMunro]

Aside from a webhook (which I think would be the best approach, where the command returns once the task has been successfully launched without waiting for it to complete), another useful possibility would be to receive the URL for the deployment, so that I can invoke this cmdlet, and once the deployment has sucessfully started (not completed, I just need to know it launched properly, and if not receive an error), I can get back a deployment object that includes the URL where I can go monitor the deployment details. I would like that piece of information to be passed back whether I get webhook support or not, but I would want it in both cases, to allow me to get a webhook for updates on deployment completion (success or fail), and to allow me to manually check for updates by going directly to the deployment in Azure using the URL I get back in the deployment object.

[markcowl]

@KirkMunro There is some service-side functionality coming to support this for deployments.

In general, I think we should have a unified mechanism for remote jobs in Azure that allows Waiting for the job to complete (via asjob or synchronous), Returning on Job startup and allowing querying on the job later, or simply returning on job startup. The latter two options could enable a push mechanism for notifications

[KirkMunro]

Related to this issue, I was just watching someone try to kick off a bunch of jobs asynchronously, each of which would deploy multiple ARM templates in sequence (one is dependent on another, which is dependent on another) from local files into different Azure subscriptions, and eventually they just threw up their arms in frustration and decided to do the entire task sequentially. The issues here were: AzureRM's default handling of session caching, combined with a conflict it ran into with the token cache on disk while multiple jobs were trying to access it (improper handling of shared resources, plus an inappropriate default caching behavior), and no easy way to sequence ARM template deployments from local files without getting into linked ARM deployments (unnecessarily complicated) or Start-Job (which adds its own complexity to the mix). Being able to sequence the deployment of multiple templates using PowerShell with AzureRM cmdlets and -AsJob would have been much easier.

[bgelens]

We are now creating Invoke-RestMethod wrapper functions for long running tasks and return the operationId and correlationId from the response headers. EventGrid subscriptions kick off the next task as we can correlate based on correlationId. I'd like to have a switch on all Az cmdlets to just give back the CorrelationId and OperationId if applicable and let me decided how to handle progress monitoring (if required).