azurerm_resource_group_template_deployment Destroy : Error 404 at the end of App Service Environments successfull deletion

[jhauray]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

• Terraform v0.13.5
• azurerm v2.48.0

Affected Resource(s)

• azurerm_resource_group_template_deployment

Terraform Configuration Files

data "azurerm_resource_group" "rg" {
  name = "<your_ase_rg>"
}

data "azurerm_subnet" "subnet_ase" {
  name                 = "<your_ase_subnet>"
  resource_group_name  = data.azurerm_resource_group.rg.name
  virtual_network_name = "<your_vnet>"
}

locals {
  # ASE DNS Suffix
  dnsSuffix = "${var.asename}.${var.dnsZone}"

  # Timestamp for unique deployment name
  timestamp = formatdate("YYYYMMDD-hh'.'mm", timestamp())
}

resource "azurerm_resource_group_template_deployment" "ase" {
  name                = "${var.asename}-deployment-${local.timestamp}"
  resource_group_name = data.azurerm_resource_group.rg.name
  deployment_mode     = "Incremental"
  template_content    = file("${path.module}/ase_template.json")

  # these key-value pairs are passed into the ARM Template's `parameters` block
  parameters_content = jsonencode({
    "location" = {
      value = var.location
    }
    "aseName" = {
      value = var.asename
    }
    "kind" = {
      value = var.kind
    }
    "ilbMode" = {
      value = var.ilbMode
    }
    "subnetId" = {
      value = data.azurerm_subnet.subnet_ase.id
    }
    "dnsSuffix" = {
      value = local.dnsSuffix
    }
    "userWhitelistedIpRanges" = {
      value = var.userWhitelistedIpRanges
    }
  })

  timeouts {
    create = "4h"
    delete = "3h"
  }
}

My ARM template : https://gist.github.com/jhauray/acd806c3802bc6096ba1ffaae29e76c2

You could almost use a template published by Azure : Create App Service Environment v2 with an ILB Address

Debug Output

https://gist.github.com/jhauray/bee86a3bb03e316bb26772537a89f021

Error message :
Error: removing items provisioned by this Template Deployment: waiting for deletion of Nested Resource "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/Terr-testnb/providers/Microsoft.Web/hostingEnvironments/ase-zoned-legacy-ic-terraform": Future#WaitForCompletion: the number of retries has been exceeded: StatusCode=404 -- Original Error: Code="ResourceNotFound" Message="The Resource 'Microsoft.Web/hostingEnvironments/ase-zoned-legacy-ic-terraform' under resource group 'Terr-testnb' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"

Panic Output

Expected Behaviour

During Terraform Destroy, ARM template nested resources must be deleted properly, without error. Especcially if resource deletion is done successfully.

Actual Behaviour

• During Terraform Destroy, nested App Service Environnement deletion is requested.
• Frequently, terraform check resource status by querying App Service Environnement GET /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/<resource-group-name>/providers/Microsoft.Web/hostingEnvironments/<ase-name>/operations/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx?api-version=2020-10-01 HTTP/1.1 REST API. The API send Response Code: 202
• When deletion is finished successfully, REST API return Response Code: 404.
• Terraform raise an Error, stop running and return error code.

Steps to Reproduce

1. Get an ARM template creating an App Service Environment (My ARM template : https://gist.github.com/jhauray/acd806c3802bc6096ba1ffaae29e76c2 or a template published by Azure : Create App Service Environment v2 with an ILB Address).
2. terraform init, terraform plan and terraform apply.
3. terraform destroy
4. Notice that error is raised.

Important Factoids

References

Official doc about Operation tracking : https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/async-operations

[jhauray]

Hello,

I opened an request to Azure support, about unexpected HTTP 404 error, on async operation url. I'm waiting for an answer. Error seems to be specific to App Service Environment.

I found a workaround that seems to work, via a provisioner :

resource "azurerm_resource_group_template_deployment" "ase" {
  name                = "${var.asename}-deployment-${local.timestamp}"
  resource_group_name = data.azurerm_resource_group.rg.name
  deployment_mode     = "Incremental"
  template_content    = file("${path.module}/ase_template.json")

  # these key-value pairs are passed into the ARM Template's `parameters` block
  parameters_content = jsonencode({
    "location" = {
      value = var.location
    }
    "aseName" = {
      value = var.asename
    }
    "kind" = {
      value = var.kind
    }
    "ilbMode" = {
      value = var.ilbMode
    }
    "subnetId" = {
      value = data.azurerm_subnet.subnet_ase.id
    }
    "dnsSuffix" = {
      value = local.dnsSuffix
    }
    "userWhitelistedIpRanges" = {
      value = var.userWhitelistedIpRanges
    }
  })

  timeouts {
    create = "4h"
    delete = "3h"
  }

  provisioner "local-exec" {
  when    = destroy
  command = <<EOT
    az login  --service-principal -u $ARM_CLIENT_ID -p $ARM_CLIENT_SECRET -t $ARM_TENANT_ID > /dev/null
    echo "az login result : $?"
    az account set -s "${jsondecode(self.output_content).subscription_id.value}" > /dev/null
    echo "az account set : $?"
    az appservice ase delete --name "${jsondecode(self.parameters_content).aseName.value}" --resource-group "${self.resource_group_name}" --subscription "${jsondecode(self.output_content).subscription_id.value}" --yes
EOT
  on_failure = continue
 }
}

A unexpected error is raised by provisioner at end of deletion :

module.ase.azurerm_resource_group_template_deployment.ase[0] (local-exec): Deployment failed. Correlation ID: 53478351-fdc6-401b-a3e7-47dc2b3df61f. The Resource 'Microsoft.Web/hostingEnvironments/ase-legacy-ic-terraform' under resource group 'Terr-testnb' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix

I don't understand what exactly is going on under the hood. But terraform destroy command end without error, and the AppService Environment is effectively deleted.

[tombuildsstuff]

👋

App Service Environments are natively supported in Terraform - so whilst it's possible to use an ARM Template deployment for this, you should be able to use the native resource instead, is there a specific piece of functionality you're using which isn't supported in the native resource?

As you've mentioned above, unfortunately this is a bug in the App Service Environment API, whilst the native resource accounts for this quirk the template deployment resource doesn't - so I believe you'd be better of using the native resource here if possible?

Thanks!

[jhauray]

Hi @tombuildsstuff,

I already use native ressource azurerm_app_service_environment for new resources. It's great!

But for old platform, not yet migrated to new ASE and created before may 2019, I still need to create ASE with custom DNS Prefix (see https://docs.microsoft.com/en-us/azure/app-service/environment/using-an-ase#dns-configuration).

I need to be able to create App Service Environment with an old API : 2018-02-01. That's why I use an ARM template.

[tombuildsstuff]

Got it, just double checking 👍

[jhauray]

For information,

Azure Support answer :

Issue : Azure REST API ASE Delete - Error 404 Async
Resolution/Conclusion : Platform limitation – Operation logs are destroyed with resource deletion. Workaround is to monitor for 404 to confirm deletion.