Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create design for client side encryption #6480

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

jelemux
Copy link

@jelemux jelemux commented Jul 10, 2023

Thank you for contributing to Velero!

Summary

Currently, backups are not encrypted at rest.
This PR introduces a design proposal that aims to fix this while still retaining backwards compatibility.
A proof-of-concept implementation can be found at https://github.com/cloudogu/velero/tree/feature/client-side-encryption.

Does your change fix a particular issue?

Fixes #434
Fixes #3218

Please indicate you've done the following:

  • Accepted the DCO. Commits without the DCO will delay acceptance.
  • Created a changelog file or added /kind changelog-not-required as a comment on this pull request.
  • Updated the corresponding documentation in site/content/docs/main. not necessary, since this is a design doc

Signed-off-by: Jeremias Weber <jeremias.weber@cloudogu.com>
@github-actions github-actions bot added the Area/Design Design Documents label Jul 10, 2023
@jelemux
Copy link
Author

jelemux commented Jul 10, 2023

/kind changelog-not-required

@github-actions github-actions bot added the kind/changelog-not-required PR does not require a user changelog. Often for docs, website, or build changes label Jul 10, 2023
@reasonerjt reasonerjt self-requested a review July 17, 2023 06:13
Then however, it has to be implemented in every single object store plugin.
This is not viable.

We could just make use of server-side encryption of an object store.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

most cloud based Object stores do support encryption at rest using user managed keys.

Can you give production examples where users are using object stores which don't support such functionality.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our customers primary concern is that encryption of object stores happens server-side so the data is unencrypted during transit.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Data should also be encrypted in transit using TLS, though having client side encryption is useful as well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A case where this may help is if bsl is set to use insecure for whatever reason.. ie. User decides to use aws plugin and override s3url to use a minio server which is uses self signed cert and have not bothered to setup cacert properly.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the main use case this solves is allowing the consumption of third party object storage for backups without having to entrust the service provider with data encryption keys.

Server-side encryption and transit encryption isn't enough to protect against potential malicious activity at the service provider level. This is just another very useful layer in the security onion.


Encryption will happen during the backup process before the objects are persisted (e.g. written to an object store).
Decryption will happen during a restore after the persisted objects have been read.
The encryption key will be stored in a Kubernetes secret by the user.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what are your thoughts on using cloud provider based secret stores for example KeyVault in Azure / AWS Secret manager for maintaing the secret.

Because we are all aware that kubernetes secrets are just plaintext and do no come with any security.

Additionally the lifecycle of the secret is tied to the lifecycle of the cluster.

Storing clusters in secret stores outside might be more secure and durable. Think cross cluster restore

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand that this might not be specific use case for you, but I want to bring this up in context of design extensibility. How can current design extend to support such scenarios.
or if it is not planned to supported at all, please put it in non goals

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I designed this feature explicitly in such a way that fetching secrets from a KMS can be implemented as well. See the KeyRetriever interface. Although, maybe this might have to be extended by a plugin API for it to be truly swappable.


Encryption will happen during the backup process before the objects are persisted (e.g. written to an object store).
Decryption will happen during a restore after the persisted objects have been read.
The encryption key will be stored in a Kubernetes secret by the user.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand that this might not be specific use case for you, but I want to bring this up in context of design extensibility. How can current design extend to support such scenarios.
or if it is not planned to supported at all, please put it in non goals


## Background

As of today, Velero stores only PersistentVolume data in a Restic/Kopia repository.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you perhaps explain how the Kopia repository is encrypted today. I am personally not aware of the mechanism, will be good to callout in this doc somewhere.

Copy link
Author

@jelemux jelemux Jul 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a secret velero-repo-credentials containing the repository password. By default, the password there has the value static-passw0rd. As you see, this is not a strong password and my guess is that the only reason this was used at all is because Kopia and Restic require encryption. While it is possible to change the password in the secret, it is as of yet not a documented feature.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documenting the secret is being tracked in #6443.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kopia encryption is described here: https://kopia.io/docs/advanced/encryption/.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for explaining @jelemux / and the info @draghuram

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area/Design Design Documents kind/changelog-not-required PR does not require a user changelog. Often for docs, website, or build changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

S3 client-side encryption support Encrypt backups at rest
5 participants