[Feature Request] Azure-repository snapshot support Client-side encryption for blobs

[cthtrifork]

Is your feature request related to a problem? Please describe

We would like the plugin to support client side encryption:
https://learn.microsoft.com/en-us/azure/storage/blobs/client-side-encryption?tabs=java

Is there support for adopting this feature? We might make the pull request but I wanted to hear the opinion of the maintainers.

Describe the solution you'd like

Support the optional setup for client side encryption and allow configuration of access to Azure Key Vault.

Related component

Plugins

Describe alternatives you've considered

No response

Additional context

The Azure Blob Storage client libraries use envelope encryption to encrypt and decrypt your data on the client side. Envelope encryption encrypts a key with one or more additional keys.

The Blob Storage client libraries rely on Azure Key Vault to protect the keys that are used for client-side encryption. For more information about Azure Key Vault, see What is Azure Key Vault?.

[cthtrifork]

I think we should also look into support managed identities for Azure Key Vault as well. As it became supported in #12423

[andrross]

[Triage - attendees 1 2 3 4 5]

Thanks for filing. We do have work in flight for general client-side encryption for all repository stores: #5800, #7229

We could also explore the option of adding the option to use the feature within repository-azure as well as an alternative. I'm not totally opposed to that, but would be happy to hear the opinion from others as well.

/cc @vikasvb90

[vikasvb90]

@cthtrifork We rolled out client side encryption support outside of the repository plugins on top of repository layer so that encryption and repository plugins remain decoupled.
The whole framework is divided in two parts:

1. Crypto Key provider plugin: A plugin of this type is responsible for providing data key pairs to be used in encryption/decryption. There is a AWS KMS based crypto-kms plugin available which retrieves keys from AWS KMS service. Similar plugins can be added for other master key providers.
2. Encryption/Decryption layer: A crypto abstraction plugin is available in OpenSearch here to encrypt or decrypt provided streams using the configured key provider against a repository. Unfortunately, there is no default plugin available for this yet and hence, you may need to build and keep a custom implementation. Once this plugin or module is available, this gets loaded by registry against the configured encrypted repository.

Once, both of these components are available, you can create an encrypted repository. Following is how a sample request would look like.

PUT _snapshot/vikasvb-repository-test-1
{
  "type": "s3",
  "settings": { // Repository settings
    "bucket": "<bucket-name>",
    "region": "<region>"
  },    
  "crypto_settings": {
      "key_provider_name": "user-db-repository-key",
      "key_provider_type": "aws-kms", // Key Provider Type
      "settings": { // Key provider settings 
          "key-1": "sample value"
      }
  }
}

key_provider_type here is the identifier you will configure for your key provider impl. aws-kms is the type for the crypto-kms plugin as mentioned here.
key_provider_name is an identifier used in registration of a master key for a given key provider type. You can use the same combination of key provider name and key provider type against any repository or you can configure different keys depending on your use case.

[cthtrifork]

@vikasvb90 That is great. I don't think it make sense to do a proprietary/vendor solution if there is upcoming support for a generic approach. Any thoughts on when this could be available? Summer? Fall? Winter?

[linuxpi]

[Storage Triage - attendees 1 2 3 4 5 6 7 8]

@vikasvb90 Please share some details if you have any on which version can we expect to see this!

[cthtrifork]

@andrross Do you have any more insights to share? any news? To my understanding, we are waiting for #5800

[cthtrifork]

To be clear, I am hoping clientside encryption becomes supported in repository-azure. It would make a lot of sense to use core capabilities instead of relying on a azure specific solution, if there has been a lot of progress on core support already.

[vikasvb90]

@cthtrifork As I mentioned earlier, if an implementation of encryption plugin is available, core already has capability to register encrypted snapshots. Entire thing is decoupled and you don't need to put encryption within a repository plugin. Regarding impl of encryption plugin, there's no plan yet but feel free to refer this old PR #9823.

Please refer to my previous comment on how an implementation of encryption plugin can allow registration of encrypted snapshots automatically.

[cthtrifork]

@vikasvb90 Thanks a lot for getting back to me. Perhaps I am just missing some understanding of the concept.
I am uncertain of what work needs to be finished to support clientside encyption today, if any.

So for my understanding. If we have this configured:

PUT _snapshot/vikasvb-repository-test-1
{
  "type": "s3",
  "settings": { // Repository settings
    "bucket": "<bucket-name>",
    "region": "<region>"
  },    
  "crypto_settings": {
      "key_provider_name": "user-db-repository-key",
      "key_provider_type": "aws-kms", // Key Provider Type
      "settings": { // Key provider settings 
          "key-1": "sample value"
      }
  }
}

It would work "today"? given that we have a aws-kms setup in AWS and have loaded the crypto-kms plugin.

In our case, we would need to make a new plugin for the KMS part, as we can not rely on AWS-KMS, but have to make a plugin for Azure vault or hashicorp vault then.

[vikasvb90]

Correct! you will need to build key provider for azure. But in addition to this you also need to implement the actual encryption interface in the form of a module. You can refer to the old PR which is a bit complex #9823 but you can choose to provide a simpler or different encryption implementation.

[cthtrifork]

Thanks so much for clarifying. It seems like there is a lot of work left and no official work being done to progress this at the moment.

I think we will try and test if https://github.com/Aiven-Open/encrypted-repository-opensearch covers our needs instead, as I am hesitant to implement an encryption implementation in a reliable, performant and secure manner.