Support for EncryptionAtHost Feature Registration

[paul-hugill]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

To use the encryption_at_host_enabled on virtual machine resources you need to enable the EncryptionAtHost feature in the Microsoft.Compute Provider Namespace.

compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidParameter" Message="The property 'securityProfile.encryptionAtHost' is not valid because the 'Microsoft.Compute/EncryptionAtHost' feature is not enabled for this subscription." Target="securityProfile.encryptionAtHost"

The automatic provider registration does not appear to register this feature, though it looks like the support on the resources has been there for a while. I have tested with both provider version 3.6.0 and 3.9.0 but also not found any references to people using it and setting that feature up, so I am not sure if I am just doing something wrong.

Only current workaround I could find (other than doing manually) would be to turn off the automatic provider registration and having to register every provider individually, which is not going to be ideal.
Without doing that I get this error, which is expected behaviour whilst automatic registration is enabled:

Error: The Resource Provider "Microsoft.Compute" is automatically registered by Terraform. To manage this Resource Provider Registration with Terraform you need to opt-out of Automatic Resource Provider Registration (by setting 'skip_provider_registration' to 'true' in the Provider block) to avoid conflicting with Terraform.

If this doesn't want to be added as a default on the automatic registration, then maybe as a provider feature flag?
Apologies if I have just missed something obvious.

New or Affected Resource(s)/Data Source(s)

provider

Potential Terraform Configuration

provider "azurerm" {
  features {
    virtual_machine {
      encryption_at_host_enabled = true
    }
}

References

https://docs.microsoft.com/en-us/azure/virtual-machines/linux/disks-enable-host-based-encryption-cli#prerequisites

[tombuildsstuff]

hey @paul-hugill

Thanks for opening this issue.

Taking a look through here, whilst the Azure Provider does register Resource Providers by default, we intentionally don't register any features as these are purely for Preview functionality.

In this case it's rather odd that this is still a Feature when it's not documented as in Preview, however since this page references that you must use a specific Portal link to access the Encryption functionality I believe this may still be in some kind of Preview phase here?

With regards to having a feature-flag for Terraform to register specific features, since this is intended to be used for Preview features we intentionally don't do this and instead call these out in the documentation (to use the Azure CLI to do so, although the Resource Provider Registration resource should also work) - however I think the main question here is why this Feature is required to be turned on when this isn't in Preview, which (given the Portal reference above) leads me to think that this must still be in (some kind of) Preview? We'd need the Service Team to confirm the status of the resource here, @mybayern1974 would you be able to reach out to the Service Team to confirm why this is a Feature here?

Thanks!

[paul-hugill]

Thanks @tombuildsstuff for the quick response, I had thought the features were mainly just for Preview stuff too but I couldn't find anything about it still being preview and it is in the Public Portal now without using the link, so it seems a bit weird.
I know MS has pretty long previews at times but the parameter on the resources was added in provider version 2.27 in September 2020, so it is at least that long.

I can open an support case with MS as well if that would help try and get some clarity.

[myc2h6o]

@paul-hugill I've checked with service team. The feature is in GA, and they will be removing this prerequisite of registering the feature flag at some point of time.

[paul-hugill]

they will be removing this prerequisite of registering the feature flag at some point of time.

Thanks @myc2h6o, did they give a rough timeframe? Are we talking days, weeks, months?
Just trying to understand if we put in some effort to work around this or wait.
I think it's fine to close this issue otherwise, but would be good to know that first, thank you.

[myc2h6o]

@paul-hugill I haven't got a rough timeframe yet from service team, will update the issue once I have that

[myc2h6o]

Hi @paul-hugill just got an update from service team, the current plan is to remove the prerequisite at Q3 2022. We could expect it to be removed before the end of Sept. 2022.

[paul-hugill]

Thanks @myc2h6o that sounds good, I'll keep an eye out for it and try later in Q3.
If there are no provider changes that will be required to support this, feel free to close the issue, it sounds like there probably aren't any but I'm not certain.

[ghost]

I am facing the same issue here with this requirement. Relating to the feature enablement, is there a bug somewhere where we try to register a feature in an existing namespace which is already enabled (whether that is preview or not) but because the existing namespace is already registered, its asking us to import that feature registration in the state? My belief is the following code should just register the EncryptionAtHost feature, it shouldn't be trying to do anything with the parent Microsoft.Compute namespace:

resource "azurerm_resource_provider_registration" "encryption_at_host" {
  name = "Microsoft.Compute"

  feature {
    name       = "EncryptionAtHost"
    registered = true
  }
}

I realise there may be a limitation with the call that is made to Azure here, I guess the TLDR is that we should be able to enable features in an existing namespace that is already enabled without having to import the registration status of that existing namespace into the state.

[ghost]

As of 2022-09-05 the feature flag requirement is still present. Will try again in October.

[dhishanbbg]

I am stuck on the same issue as well. Unable to register the feature when a parent namespace provider is already registered. This is not an issue when you use the az cli or the REST API.

After snooping through the logs, I noticed that the code flow is currently as follows

1. A call is made to verify if the resource_provider is registered (parent namepace)
2. If the provider is already registered the terraform will throw and error as stated in the other comments.

However there is never a call made to ensure that the feature in question is ever registered. It simply errors out upon provider namespace registration check. The correct implementation should be a call to the feature registration API upon checking if the feature is registered and not on the parent namespace check.

On a side note: The API used to register the feature for the one's that succeeds is not the one documented in the official documentation. Currently the API used is https://management.azure.com/subscriptions/<subid>/providers/Microsoft.Features/providers/<provider amespace>/features/<feature>/register?api-version=2015-12-0 However according to the MSFT docs, the API should be
https://management.azure.com/subscriptions/<subid>/providers/Microsoft.Features/featureProviders/<provider namespace>/subscriptionFeatureRegistrations/<feature>/?api-version=2021-07-01

Steps to reproduce:
Register the namespace Microsoft.Network and attempt to run the following terraform code:

provider "azurerm" {
  subscription_id = <subscription id here>
  features {}
  skip_provider_registration = true
}
resource "azurerm_resource_provider_registration" "example" {
  name = "Microsoft.Network"

  feature {
    name       = "AllowGlobalTagsForStorage"
    registered = true
  }
}

[cschipper1]

Same issue. Also the new "featureProviders" api says registered and the "features" api stays registering for 10-15 minutes. PS1 Get-AzProviderFeature command also use "features" api.

Used the azapi provider to workaround:

resource "azapi_update_resource" "encryptionathost" {
  type = "Microsoft.Features/featureProviders/subscriptionFeatureRegistrations@2021-07-01"
  resource_id = "/subscriptions/${var.subscription_id}/providers/Microsoft.Features/featureProviders/Microsoft.Compute/subscriptionFeatureRegistrations/encryptionathost"
  body = jsonencode({
    properties = {}
  })
}

[RoFz]

Hi @myc2h6o , just wanted to check/confirm that this is still an issue (Sep/2023) since the encryption_at_host_enabled feature still requires manual activation. Would that be correct?

│ Error: The Resource Provider "Microsoft.Compute" is automatically registered by Terraform.
│ 
│ To manage this Resource Provider Registration with Terraform you need to opt-out
│ of Automatic Resource Provider Registration (by setting 'skip_provider_registration'
│ to 'true' in the Provider block) to avoid conflicting with Terraform.
│ 
│   with azurerm_resource_provider_registration.encryption_at_host,
│   on main.tf line 120, in resource "azurerm_resource_provider_registration" "encryption_at_host":
│  120: resource "azurerm_resource_provider_registration" "encryption_at_host" {
│ 
│ The Resource Provider "Microsoft.Compute" is automatically registered by Terraform.
│ 
│ To manage this Resource Provider Registration with Terraform you need to opt-out
│ of Automatic Resource Provider Registration (by setting 'skip_provider_registration'
│ to 'true' in the Provider block) to avoid conflicting with Terraform.

[evmimagina]

Hi tombuildsstuff,

Is there any solution for this kind of scenario? I'm affected too, when trying to enable a Preview Feature for enabling EncryptionAtHost for AKS service with the following config:

provider "azurerm" {
features { }
skip_provider_registration = true
}

resource "azurerm_resource_provider_registration" "EnableEncryptionAtHostPreview" {
name = "Microsoft.ContainerService"
feature {
name = "EnableEncryptionAtHostPreview"
registered = true
}
}

If we let terraform to handle the providers registration, as recommended, (not setting the skip_provider_registration variable)... we get the error that says:

│ The Resource Provider "Microsoft.ContainerService" is automatically
│ registered by Terraform.
│
│ To manage this Resource Provider Registration with Terraform you need to
│ opt-out
│ of Automatic Resource Provider Registration (by setting
│ 'skip_provider_registration'
│ to 'true' in the Provider block) to avoid conflicting with Terraform.

If we enable the skip_provider_registration (as per documentation), then we get the error:

A resource with the ID
│ "/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxxxx/providers/Microsoft.ContainerService"
│ already exists - to be managed via Terraform this resource needs to be
│ imported into the State. Please see the resource documentation for
│ "azurerm_resource_provider_registration" for more information.

All right, we might solve that error importing that provider before executing plan phase... BUT when setting that variable to true, then we would require to enable any other provider we might require during coding, even those that would be automatically registered by Terraform, right? (and who knows the providers we might need...), AND, what's even worst, we would also need to import any other provider already registered previous plan execution (not sure but I think some providers are registered by default when provisioning a subscription)... just to enable any desired preview feature that is under it's scope, right?... Hopefully you will agree that all of this makes things even more difficult rather than easier.

I think you, AzureRM provider Team, should look into this carefully and offer a proper solution and not what is documented here: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/resource_provider_registration

In the meantime, I will try to find a WA other than the null_resource and the local-exec... that solution is not for me..., having to do the az login on the command, I don't like it ... storing the credential secret somewhere as plain text...

I might try the WA of sending requests directly to the azapi ...

I will be looking forward to hearing from you :).

Many thanks for all your work and efforts.

All the best,

[evmimagina]

Hi,

In case it can help someone that reaches this issue :)

Well, this is my WA, finally there's no need to pass to the script any secret, it will catch it from the environment variables

resource "null_resource" "EnableEncryptionAtHostPreview" {
  provisioner "local-exec" {
    command = "bash path_to_script/enable_resource_provider_feature.sh --provider_name \"Microsoft.ContainerService\" --feature_name \"EnableEncryptionAtHostPreview\" "
  }
}

The script content:

#!/bin/bash
set -e

POSITIONAL_ARGS=()

while [[ $# -gt 0 ]]; do
  case $1 in
    -fetname|--feature_name)
      FEAT_NAME="$2"
      shift # past argument
      shift # past value
      ;;
    -provname|--provider_name)
      PROV_NAME="$2"
      shift # past argument
      shift # past value
      ;;
    -*|--*)
      echo "Unknown option $1"
      exit 1
      ;;
    *)
      POSITIONAL_ARGS+=("$1") # save positional arg
      shift # past argument
      ;;
  esac
done

FEATURE_NAME=${FEAT_NAME}
PROVIDER_NAME=${PROV_NAME}

az login --service-principal --username $ARM_CLIENT_ID --password $ARM_CLIENT_SECRET --tenant $ARM_TENANT_ID
az account set --subscription $ARM_SUBSCRIPTION_ID
state=$(az feature show --namespace $PROVIDER_NAME --name $FEATURE_NAME -o tsv --query "properties.state")
timeout=1200 ; start_time=$(date +%s)
if [ -z "$state" ]; then
    echo "Unable to resolve the state variable. Feature doesn't exists anymore?"
    exit 0
fi

if [ "$state" != 'Registered' ]; then
	echo "Current state is: $state"
	if ([ "$state" = 'Unregistered' ] || [ "$state" = 'NotRegistered' ] || [ "$state" = 'Unregistering' ] ); then
		az feature register --namespace $PROVIDER_NAME --name $FEATURE_NAME
	fi
    while [ "$state" = 'Registering' ]
    do
        current_time=$(date +%s) ; elapsed_time=$((current_time - start_time))
        if [ "$elapsed_time" -ge "$timeout" ]; then
            echo "Timeout reached while waiting for a registration state change. Exiting the script." ; exit 1
        fi
		echo "Current state is: $state. Sleep for 30 sec"
        sleep 30 ; state=$(az feature show --namespace $PROVIDER_NAME --name $FEATURE_NAME -o tsv --query "properties.state")
    done
    az provider register --namespace $PROVIDER_NAME
else
    echo "Unexpected Feature State or Feature already registered?. current value is: $state" ; exit 0
fi

state=$(az feature show --namespace $PROVIDER_NAME --name $FEATURE_NAME -o tsv --query "properties.state")
echo "EOS - Feature State registration value is: $state"
exit 0

Regards,

[Speeddymon]

I think this is resolved now? I can use https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/resource_provider_registration to register EncryptionAtHost in my subscriptions.

[craigpay]

I've just arrived at this thread because I still have the issue trying to enable encryption_at_host_enabled on a VM.

I've tried to register the provider in Terraform but I get stuck in the same loop as @evmimagina

[craigpay]

Hi all. Here's another workaround, this time using AZ CLI...

terraform {
  required_version = ">=0.12"

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~>3.3"
    }
  }
}

provider "azurerm" {
  features {}
}

resource "null_resource" "example" {
  provisioner "local-exec" {
    command = "az feature register --namespace Microsoft.Compute --name EncryptionAtHost"
  }
}

resource "azurerm_resource_group" "example" {
  name     = "rg-uks-vm-host-encrypt"
  location = "UK South"
}

resource "azurerm_virtual_network" "example" {
  name                = "example-network"
  address_space       = ["10.0.0.0/16"]
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
}

resource "azurerm_subnet" "example" {
  name                 = "internal"
  resource_group_name  = azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.example.name
  address_prefixes     = ["10.0.2.0/24"]
}

resource "azurerm_network_interface" "example" {
  name                = "example-nic"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name

  ip_configuration {
    name                          = "internal"
    subnet_id                     = azurerm_subnet.example.id
    private_ip_address_allocation = "Dynamic"
  }
}

resource "azurerm_windows_virtual_machine" "example" {
  name                = "example-machine"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  size                = "Standard_B2ms"
  admin_username      = "adminuser"
  admin_password      = "P@$$w0rd1234!"
  network_interface_ids = [
    azurerm_network_interface.example.id,
  ]
  encryption_at_host_enabled = true

  os_disk {
    caching              = "ReadWrite"
    storage_account_type = "Standard_LRS"
  }

  source_image_reference {
    publisher = "MicrosoftWindowsServer"
    offer     = "WindowsServer"
    sku       = "2022-Datacenter"
    version   = "latest"
  }
}

[davidkarlsen]

Is this still required? At least I do not find the option to register this any longer in the features section in the portal.

[Speeddymon]

Still required. It's a preview feature so it's not so easy to find in the portal. In fact I don't know how to look for it in the portal myself but I can confirm that as of 2 weeks ago it was still required in order to create clusters with node encryption. We registered it in Azure CLI to get Terraform working.

[825i]

I think this is resolved now? I can use https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/resource_provider_registration to register EncryptionAtHost in my subscriptions.

How did you do this exactly?

I have:

# Enable Host Level Encryption on the subscription
resource "azurerm_resource_provider_registration" "encryption_at_host" {
  name = "Microsoft.Compute"

    feature {
    name       = "EncryptionAtHost"
    registered = true
  }
}

Which throws the error:

│ Error: importing Subscription Provider (Subscription: "[[REDACTED]]"
│ Provider Name: "Microsoft.Compute"): The Resource Provider "Microsoft.Compute" is automatically registered by Terraform.
│
│ To manage this Resource Provider Registration with Terraform you need to opt-out
│ of Automatic Resource Provider Registration (by setting 'skip_provider_registration'
│ to 'true' in the Provider block) to avoid conflicting with Terraform.

How did you get around this?

[825i]

Ok to be completely clear:

# Enable Host Level Encryption on the subscription
resource "azurerm_resource_provider_registration" "encryption_at_host" {
  name = "Microsoft.Compute"

    feature {
    name       = "EncryptionAtHost"
    registered = true
  }
}

Does not work.

I get in Terraform Cloud:

Error: A resource with the ID "/subscriptions/[[REDACTED]]/providers/Microsoft.Compute" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for "azurerm_resource_provider_registration" for more information.
with azurerm_resource_provider_registration.encryption_at_host
on main.tf line 12, in resource "azurerm_resource_provider_registration" "encryption_at_host":
resource "azurerm_resource_provider_registration" "encryption_at_host" {
A resource with the ID "/subscriptions/[[REDACTED]]/providers/Microsoft.Compute" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for "azurerm_resource_provider_registration" for more information.

After which when trying to import it, I get another error:

│ Error: importing Subscription Provider (Subscription: "[[REDACTED]]"
│ Provider Name: "Microsoft.Compute"): The Resource Provider "Microsoft.Compute" is automatically registered by Terraform.
│
│ To manage this Resource Provider Registration with Terraform you need to opt-out
│ of Automatic Resource Provider Registration (by setting 'skip_provider_registration'
│ to 'true' in the Provider block) to avoid conflicting with Terraform.

So this is CLEARLY not set up properly in azurerm at all. It merely checks if the namespace is registered and if it is already, it fails. It doesn't actually check if EncryptionAtHost is ever enabled or not. It's definitely not working as intended at all.

I suppose I could add to my providers.tf:
skip_provider_registration = true

But not only should this not be even required, I don't want to for the reasons this person @evmimagina: #17185 (comment) mentioned. As it will break other things in my setup.

I also tried "EncryptionAtHostPreview" which did not work either. So this is simply not supported by azurerm at all, unless someone else has an actual working example to prove that it does.

In the meantime I will simply use this:

# Alternative way of enabling Host Level Encryption using azapi
resource "azapi_update_resource" "encryptionathost" {
  type = "Microsoft.Features/featureProviders/subscriptionFeatureRegistrations@2021-07-01"
  resource_id = "/subscriptions/${var.subscription_id}/providers/Microsoft.Features/featureProviders/Microsoft.Compute/subscriptionFeatureRegistrations/encryptionathost"
  body = jsonencode({
    properties = {}
  })
}

[istairbn]

So does that mean we should raise a new issue about this code not working:

# Enable Host Level Encryption on the subscription
resource "azurerm_resource_provider_registration" "encryption_at_host" {
  name = "Microsoft.Compute"

    feature {
    name       = "EncryptionAtHost"
    registered = true
  }
}

i.e. that we can't register missing sub-dependencies
OR should we be raising an upstream ticket about why this is still a "preview feature" 2 years down the line

[jkroepke]

I found a workaround to mitigate the The Resource Provider "Microsoft.Compute" is automatically registered by Terraform. issue.

However, I don't know, if this is working on fresh subscriptions where Microsoft.Compute is not registered.

import {
  id = "/subscriptions/${data.azurerm_client_config.this.subscription_id}/providers/Microsoft.Compute"
  to = azurerm_resource_provider_registration.encryption_at_host
}

resource "azurerm_resource_provider_registration" "encryption_at_host" {
  name = "Microsoft.Compute"

  feature {
    name       = "EncryptionAtHost"
    registered = true
  }
}