azurerm_machine_learning_datastore_blobstorage throws error when using service_data_auth_identity meta argument

[VickyWinner]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v1.0.4

AzureRM Provider Version

provider registry.terraform.io/hashicorp/azurerm v3.52.0

Affected Resource(s)/Data Source(s)

azurerm_machine_learning_datastore_blobstorage

Terraform Configuration Files

resource "azurerm_machine_learning_datastore_blobstorage" "example" {
  name                 = "test"
  workspace_id         = module.azureml.id
  storage_container_id = data.azurerm_storage_container.example.id
  service_data_auth_identity = "WorkspaceSystemAssignedIdentity"
}

Debug Output/Panic Output

Error: Missing required argument

  with azurerm_machine_learning_datastore_blobstorage.example,
  on register_storage_blob.tf line 1, in resource "azurerm_machine_learning_datastore_blobstorage" "bookitnow":
   1: resource "azurerm_machine_learning_datastore_blobstorage" "example" {

"shared_access_signature": one of `account_key,shared_access_signature` must
be specified

Error: Invalid combination of arguments

  with azurerm_machine_learning_datastore_blobstorage.example,
  on register_storage_blob.tf line 1, in resource "azurerm_machine_learning_datastore_blobstorage" "example":
   1: resource "azurerm_machine_learning_datastore_blobstorage" "example" {

"account_key": one of `account_key,shared_access_signature` must be specified

Expected Behaviour

blobstore registered.

Actual Behaviour

terraform plan erroring out

Steps to Reproduce

use the block fromabove

Important Factoids

No response

References

https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/machine_learning_datastore_blobstorage

[sinbai]

@VickyWinner thanks for opening this issue. In fact, one of account_key,shared_access_signature must
be specified, this is by design. I have submitted PR to update the terraform doc to clarify it. Could you please provide one of account_key, shared_access_signature in configuration to resolve above error?

[VickyWinner]

@sinbai, we don't want to rely on account_key or sas key at all. I am assuming that if I use service_data_auth_identity = "WorkspaceSystemAssignedIdentity" I don't need to pass account_key or sas key which gets rotated frequently. I want to use MSI similar to what is available in portal. Where is this available in TF?

[tombuildsstuff]

@sinbai I've reopened this issue, would you mind taking another look at this one, as @VickyWinner has mentioned, it should be possible to use a System Assigned Identity for this purpose?

[mlomat]

@tombuildsstuff any update here? It's very annoying issue, especially that resource for DFS is working normally :(

[chiman-wend]

@VickyWinner have you found a solution for your bug besides using account key or sas? I have stumbled upon the same bug.

SystemAssignedIdentity should be possible for Datastore registration or creation.
Same goes for Datastore_Fileshares https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/machine_learning_datastore_fileshare

Can you take another look on this please @tombuildsstuff and @sinbai?

[pavanmuni321]

@tombuildsstuff , @VickyWinner , @sinbai

There is more to this ---
Unlike user-created datastores, the default datastores created by the Azure ML workspace system use storage account key authentication. Disabling shared key access for the storage account will result in notebooks becoming inaccessible to users irrespective if manage identities are given access to the storage account, as these notebooks are stored in the workspace's default datastores. I dont see this handled by terraform or is there other ways to enforce the use of managed identity for these default datastores instead of relying on account key/SAS for authentication?

[sdehm]

Shared key access is discouraged for stronger security as documented here. Entra ID is the preferred method of access secure storage accounts. AzureML allows this with the option of not storing credentials with a datastore. This document shows scenarios including those with "No Credential" on the datastore where uses the user's identity or the workspace's managed service identity. This applies to all datastores including the workspace's default datastore.

You can also see this behavior in the AzureML UI tooltip on the credential saving toggle in the datastore's authentication settings:

On the Azure portal when creating a new AzureML workspace it provides an option to use identity access by for the initially created default datastore and also has an extra checkbox to disable shared key access from the start. As far as I can tell when the option to use "identity-based access" sets up the default workspaceblobstore with saved credentials turned off (no access key or sas token).

This leads me to believe that it is valid to not include an access key or sas token when creating an AzureML blob storage datastore and the Terraform provider should allow this. When that configuration is applied it by default uses identity based access. The user must still configure their identities to get data preview, notebooks, and other features that require access to the data from the portal but I don't believe that needs to be enforced in the provider.

It would be greatly appreciated if we could have this restriction removed from this provider so that it can be used for AzureML workspaces that must adhere to certain security standards.