Managed Private Endpoint for Synapses is crashing terraform plan after it's approval -

[erthalmvp]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Hello everyone, I already looked for some errors like this before but any solutions helped me. Basically, I have the following scenario:
1) Create a stg, assign rule "Storage Account Contributor" for my pipeline, and transform this stg into a data lake;
2) Create Synapses, private endpoint and create a spark pool;
3) Use null_resource for az-cli to assign Synapse to stg (Networking > Resource Instance);
4) Create manage private endpoint;
5) Use null_resource for az-cli command and approve private endpoint.

My error occurs after step 5 and is represented in session 'Debug Output/Panic Output'.

I already tested some tips that address this type of error in the community like role and network issues. I research about the possibility to use some Terraform resources to approve private endpoints or created this link stg/synapses automatically approved but I'm not able to find anything.

My Synapses work properly after those steps. I'm not able to run any plan or work with Terraform anymore after this process.

Terraform Version

1.4.6

AzureRM Provider Version

3.54.0

Affected Resource(s)/Data Source(s)

azurerm_storage_data_lake_gen2_filesystem; azurerm_synapse_managed_private_endpoint;

Terraform Configuration Files

resource "azurerm_storage_account" "storage_account_syn" {
  name                     = "xxxsyn"
  resource_group_name      = "rg_test_name"
  location                 = "location"
  account_kind             = "StorageV2"
  account_tier             = "Standard"
  account_replication_type = var.storage_type
  is_hns_enabled           = true
  tags                     = var.tags

  routing {
    choice                      = "MicrosoftRouting"
    publish_microsoft_endpoints = true
    publish_internet_endpoints  = false
  }

  network_rules {
    bypass         = ["AzureServices"]
    default_action = "Deny"

    virtual_network_subnet_ids = [
      vnet1,
      vnet2,
      vnet3
    ]

     ip_rules = [
      "xxx.xxx.xx.xx"
     ]
  }

  lifecycle {
    ignore_changes = [
      network_rules[0].private_link_access
    ]
  }

  blob_properties {
    delete_retention_policy {
      days = var.delete_retention_policy_days_stg_syn
    }

    container_delete_retention_policy {
      days = var.delete_retention_policy_days_stg_syn
    }
  }
}

data "azurerm_client_config" "current" {}

resource "azurerm_role_assignment" "role_assignmen_syn_sac" {
  scope                = var.stg_syn_id
  role_definition_name = "Storage Account Contributor"
  principal_id         = data.azurerm_client_config.current.object_id
  depends_on = [
     azurerm_storage_account.storage_account_syn
  ]
}

resource "time_sleep" "role_assignment_sleep" {
  create_duration = "60s"

  triggers = {
    role_assignment = azurerm_role_assignment.role_assignmen_syn_sac.id
  }
}

resource "azurerm_storage_data_lake_gen2_filesystem" "filesystem" {
  name               = "xxxdlsgsyn"
  storage_account_id = azurerm_storage_account.storage_account_syn.id
  depends_on = [
    azurerm_storage_account.storage_account_syn,
    time_sleep.role_assignment_sleep
  ]
}

resource "azurerm_synapse_workspace" "synapse_workspace" {
  name                                 = "testsyn"
  resource_group_name                  = "test-rg-name"
  location                             = "location"
  storage_data_lake_gen2_filesystem_id = azurerm_storage_account.storage_account_syn.id
  sql_administrator_login              = var.mssqlsrv_admin_user
  sql_administrator_login_password     = var.mssqlsrv_admin_password
  managed_virtual_network_enabled      = true
  managed_resource_group_name          = "test-rg-syn-managed"
  public_network_access_enabled        = true
  tags                                 = var.tags

  lifecycle {
    ignore_changes = [
      azure_devops_repo
    ]
  }

  identity {
    type = "SystemAssigned"
  }
}

resource "azurerm_private_endpoint" "synapse_server_priv" {
  name                = "TEST-PEP"
  resource_group_name = "test-rg-name"
  location            = "location"
  subnet_id           = azurerm_subnet.sn_priv.id
  tags                = var.tags
  depends_on = [
    azurerm_resource_group.rg_sec,
    azurerm_subnet.sn_priv,
    azurerm_synapse_workspace.synapse_workspace
  ]

  ip_configuration {
    name               = "TEST-IP"
    private_ip_address = var.private_ip_address_syn
    subresource_name   = "Sql"
  }

  private_service_connection {
    name                           = "TEST-CONN"
    private_connection_resource_id = azurerm_synapse_workspace.synapse_workspace.id
    is_manual_connection           = false
    subresource_names              = ["Sql"]
  }

  private_dns_zone_group {
    name                 = "default"
    private_dns_zone_ids = [var.synapse_dns_zone_id]
  }
} # https:

resource "azurerm_synapse_spark_pool" "synapse_spark_pool" {
  name                 = "SmallSpark"
  synapse_workspace_id = azurerm_synapse_workspace.synapse_workspace.id
  node_size_family     = "MemoryOptimized"
  node_size            = "Small"
  cache_size           = 50
  spark_version        = 3.3

  auto_scale {
    max_node_count = 9
    min_node_count = 3
  }

  auto_pause {
    delay_in_minutes = 15
  }
}

resource "null_resource" "add_resouce_instance_syn_stg_syn" {
  triggers = {
    run_when_synapse_created = azurerm_synapse_workspace.synapse_workspace.id
  }
  provisioner "local-exec" {
    command = <<-EOT
      az login --service-principal \
          --username ${var.service_connection_client_id} \
          --password ${var.service_connection_secret} \
          --tenant ${var.tenant_id} \
          --output "none"
      az storage account network-rule add \
          --account-name "${var.stg_name}" \
          --resource-id "${azurerm_synapse_workspace.synapse_workspace.id}" \
          --tenant-id "${var.tenant_id}" \
          --resource-group "${var.rg_name}"
    EOT
  }
  depends_on = [
    azurerm_synapse_workspace.synapse_workspace
  ]
}

resource "azurerm_synapse_managed_private_endpoint" "synapse_managed_pep_syn" {
  name                 = "TEST-MPEP"
  synapse_workspace_id = azurerm_synapse_workspace.synapse_workspace.id
  target_resource_id   = var.stg_syn_id
  subresource_name     = "dfs"
  depends_on = [
    azurerm_synapse_workspace.synapse_workspace
  ]
}

resource "time_sleep" "wait_60_seconds" {
  depends_on = [
    azurerm_synapse_managed_private_endpoint.synapse_managed_pep_syn,
  ]
  create_duration = "60s"
}

resource "null_resource" "approve_synapse_managed_pep_syn" {
  depends_on = [
    time_sleep.wait_60_seconds
  ]
  provisioner "local-exec" {
    command = <<-EOT
      az login --service-principal \
          --username ${var.service_connection_client_id} \
          --password ${var.service_connection_secret} \
          --tenant ${var.tenant_id} \
          --output "none"
      id=$(az network private-endpoint-connection list -g "${var.rg_name}" \
      -n "${var.stg_name}" \
      --type Microsoft.Storage/storageAccounts \
      --query "[?properties.privateLinkServiceConnectionState.status=='Pending'].{id:id}" --output tsv)
      az network private-endpoint-connection approve --id $id \
      --type Microsoft.Storage/storageAccounts \
      --description "Auto-Approved by Terraform"
    EOT
  }
}

Debug Output/Panic Output

Error: retrieving File System "xxxsyn" in Storage Account "yyysyn": datalakestore.Client#GetProperties: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
│ 
│   with azurerm_storage_data_lake_gen2_filesystem.filesystem,
│   on zzzz.tf line --, in resource "azurerm_storage_data_lake_gen2_filesystem" "filesystem":
│  279: resource "azurerm_storage_data_lake_gen2_filesystem" "filesystem" {

Expected Behaviour

Since I just apply an null-resource to run az-cli command for approving a private endpoint I'm expecting in the next plan the message 'Not to update'.

Actual Behaviour

The error mentioned above.

Steps to Reproduce

No response

Important Factoids

No response

References

No response

[erthalmvp]

Hi all, any update on this?

[andigwandi]

I am having a similar issue

My storage account is accessible publically and has a managed endpoint from Synapse.

[erthalmvp]

Just to share a workaround, I removed the storage account used by Synapse from my '.tfstate' file. After it is possible to run 'plan' and 'apply' without any problem.

[andigwandi]

In my case, I have a 'managed private endpoint' from Synapse to the storage account. Deleting the managed endpoint is working for me. But the endpoint is created again with deployment, so it is a loop for me.