Multiple encryption keys/key methods

[sjau]

I'd like to propose a feature request to support multiple key slots/method slots like LUKS does it.

This can be very useful so that for example 2 different people have different passwords and they can independantly open up the encrypted dataset without needing to know the other's (highly secure and complex) passphrase.

Also, if you have multiple zpools it could be useful to provide path to the keyfile to automatically unlock it if needed...

[Lady-Galadriel]

I too desire multiple key slots. My thoughts are these:

1. Having a second key may allow recovery. Meaning you can have a master encryption key on a USB flash drive, kept locked up. But the normal operations staff have access to a another one with a password. If the password get's lost or changed without knowing what it is, the master can be used.

2. Additionally, an option to allow either R/W or R/O keys would be useful. This would affect mounting the dataset and ability to change the keys, (like operations for backups would only need R/O). But, there would be a safeguard to prevent deleting the last R/W key.

3. Also, perhaps a Master / Secondary flag for each key. This would allow 2 slots to both be R/W, but a Secondary could not delete / change the key or password from the Master slot. Again useful when dealing with operations staff who may perform scheduled changes involving reboots or pool imports. Plus, there should be a safeguard in place that would prevent the last Master key from being deleted.

For the last, it could very well be numerical order, and not Master / Secondary. That would be useful if we had more than 2 keys.

Having watched the chaos when users have used FreeNAS's Geli disk encryption, and not understood it, I would prefer a scheme that had more than 1 key slot. (Still won't protect users from themselves. But, this may protect Unix SysAdmins from having to perform restores when someone else looses or changes the key, and can't supply any key...)

[Lady-Galadriel]

@sjau, in regards to;

Also, if you have multiple zpools it could be useful to provide path to the keyfile to automatically unlock it if needed...

This sort of works already. It's in the manual page, (which I don't have handy). I did submit a feature request to allow RAW keys to be stored on disk partitions, (like an USB flash drive), #6556

[sjau]

well the think with auto-unlock was how I used luks/dm-crypt... I can either provide the key file or enter manually the key... which is convenvient

[Lady-Galadriel]

All variations are supported if I remember correctly. Prompt or path to passphrase or key;

Prompt for passphrase
Prompt for location of key file

Path to file with passphrase
Path to file with key

As I said, I don't have the manual page handy to show examples. And if the path does not exist, it defaults back to prompt, (if I remember correctly...).

[sjau]

the benefit is if you have multiple key slots... if you can provide a password or a path to file.. just whatever seems more convenient... e.g. have a keyfile in /root on your server, take out the disk, use it somewhere else and since the keyfile won't exist on the other system, provide password

[Lady-Galadriel]

Yes, that would be useful at times.

Plus. it would allow for split mirror pools. Basically have a mirrored pool, (at least 2 way, but more is supported), and use the zpool split command to create a second pool. One you could export and import elsewhere using a passphrase.

[sjau]

that's why I suggested it...

[sjau]

@tcaputi

Is this possible to add as feature?

[tcaputi]

@sjau

Is this possible to add as feature?

Yes. The current implementation was actually designed with the ability to support multiple keys at some point. It has just not been a priority for me at the moment and I currently have a few other projects which I've committed to working on first. With regards to some of the specific features and use-cases that have been mentioned:

This can be very useful so that for example 2 different people have different passwords and they can independantly open up the encrypted dataset without needing to know the other's (highly secure and complex) passphrase.

This is probably the biggest use case that I can think of.

Having a second key may allow recovery. Meaning you can have a master encryption key on a USB flash drive, kept locked up. But the normal operations staff have access to a another one with a password. If the password get's lost or changed without knowing what it is, the master can be used.

This really just boils down to the above use case of having 2 people with different passphrases / keys.

Additionally, an option to allow either R/W or R/O keys would be useful. This would affect mounting the dataset and ability to change the keys, (like operations for backups would only need R/O). But, there would be a safeguard to prevent deleting the last R/W key.

Also, perhaps a Master / Secondary flag for each key. This would allow 2 slots to both be R/W, but a Secondary could not delete / change the key or password from the Master slot. Again useful when dealing with operations staff who may perform scheduled changes involving reboots or pool imports. Plus, there should be a safeguard in place that would prevent the last Master key from being deleted.

Unfortunately, the cryptography here doesn't allow for this. All data is protected with symmetric key encryption, so there is no R/O key. Using public key cryptography would render the implementation uselessly slow. Without the ability to do this, it becomes impossible to have primary / secondary keys since then the secondary key could only be stopped from making changes via permissions which are trivial to patch over if you have root access to the machine. As a general rule, we have decided not to implement encryption functionality that cannot be enforced with cryptography so that we don't have this odd distinction between things that can / can't be done maliciously and things that can / can't be done inadvertently. My personal belief on the matter is that encryption and permissions need to be thought of differently and kept separate.

Plus. it would allow for split mirror pools. Basically have a mirrored pool, (at least 2 way, but more is supported), and use the zpool split command to create a second pool. One you could export and import elsewhere using a passphrase.

Not quite sure if I understand here. Couldn't you just copy the key and use it for the new pool?

[sjau]

@tcaputi

No worries... just take your time.. I know you're quite busy... but at least #8684 is merged now :)

I wasn't sure if you've seen this before, that's why I sent a message to you. It's a convenience feature and I'm glad to hear that the current encryption is designed with multi-slots use later on.

[tcaputi]

Additionally, an option to allow either R/W or R/O keys would be useful. This would affect mounting the dataset and ability to change the keys, (like operations for backups would only need R/O). But, there would be a safeguard to prevent deleting the last R/W key.

As a quick side note, scrubs / resilvers can be accomplished without any loaded keys. Backups can be taken without keys as well if you use the zfs send --raw feature.

[sjau]

I think it's really great that scrubs / resilver / raw send don't required to load the keys. So even third parties can do that maintenance easily :)

[jknockaert]

Any chance this feature will be included in a 0.8 point upgrade?

[tcaputi]

In general, point releases are meant for bug fixes and so I would not expect this feature to show up in 0.8.x release. I am also not currently working on this feature at the moment.

[viggy96]

This feature would be awesome to have. I'm really paranoid about my keyfile being somehow corrupted or something, and my data being inaccessible without it. I will of course backup the keyfile to several locations, but still, one can't help but worry.

[tcaputi]

I agree that this feature would be nice to have, but I don't think it will help you with protecting your key file from corruption. At the end of the day keeping access to your data is just a matter of keeping access to a key that will work. Therefore, having a backup of the key is exactly the same as having a different key for this purpose. This feature is more about allowing multiple people to have their own passwords or keys to access the data.

[kithrup]

The general way to do this is to have an encrypted key, and then multiple keys which can be used to decrypt it. (Tom knows this, I'm just not sure how much other people do, so sorry if it sounds condescending.) So the change here would be to add user/key pairs somewhere, which would then be used to unlock the main key.

[viggy96]

@kithrup Isn't that the idea behind the current implementation? Aren't the user provided passwords/keys just used as a wrapper for the real encryption key?

[kithrup]

Yes, the user/key mapping just hasn't been done :).

[mddeff]

Consider another vote for interest in the feature. I want to implement a clevis/tang unlocker for ZFS. Use case is needing a breakglass key in case the tang server goes up in flames. I like clevis/tang, but I'm not willing to run it (in production) without a backup way to decrypt the data.

[bill-mcgonigle]

Could we use properties for this securely, as an interim/0.8.x measure?
e.g.

# openssl enc -aes-256-cbc -pbkdf2 -a -A < tank-test.key > tank-test.key.aes
enter aes-256-cbc encryption password: ********
Verifying - enter aes-256-cbc encryption password: ********
# cat tank-test.key.aes
U2FsdGVkX19Ifd8hLvjm1RGFfA23zkNxE/GilHh9Z/32b6DDqeO44mkskgPcZy4ec1PQH/FowzPQLDZRKoP9OQ==#
# zfs set cryptkey:001='U2FsdGVkX19Ifd8hLvjm1RGFfA23zkNxE/GilHh9Z/32b6DDqeO44mkskgPcZy4ec1PQH/FowzPQLDZRKoP9OQ==' tank/test
# zfs get cryptkey:001 tank/test
NAME          PROPERTY      VALUE                                                                                     SOURCE
tank/test  cryptkey:001  U2FsdGVkX19Ifd8hLvjm1RGFfA23zkNxE/GilHh9Z/32b6DDqeO44mkskgPcZy4ec1PQH/FowzPQLDZRKoP9OQ==  local

And then running the inverse ( -d ) and setting keyformat=raw keylocation=outfile. My interest is disaster recovery here; this would probably be inconvenient for frequent key swapping. Some simple scripts could help.

I think this wouldn't work for pools with encrypted root datasets, but if somebody could correct me about the status of the root-level properties being encrypted or not that would be great.

[tcaputi]

@bill-mcgonigle yes that should work as long as you are happy with the sercurity of aes-cbc. I'm not actually all that familiar with the openssl command line tool, but this looks right. It should also work for pools with encrypted root filesystems. You can still import such pools to zfs get the properties you need. It just won't mount the root filesystem without the key.

[jknockaert]

@bill-mcgonigle I am not sure if I correctly understand what you're doing here. I assume that tank-test.key is the (unique) encryption key, and you're storing an encrypted version of it (encrypted using password ********) as an arbitrary zfs property (which you call cryptkey:001 but it could effectively have any name not yet in use by an existing zfs property). So in fact you're using the zfs properties table to store an (externally) encrypted version of the zfs key. Am I correct?

[mdPlusPlus]

Is this currently being worked on? I was hoping I could replace LUKS with native ZFS encryption soon.

[robszy]

We need some form of recover password as it was said earlier but it is very important topic

[crocket]

Until /etc/init.d/zfs-load-key is merged and this is implemented, I use this workaround.

/usr/lib/dracut/modules.d/99local/module-setup.sh

#!/bin/sh

check() {
  return 0
}

depends() {
  echo zfs
  return 0
}

install() {
  inst_hook pre-mount 80 "${moddir}/zfs-load-key-all.sh"
}

/usr/lib/dracut/modules.d/99local/zfs-load-key-all.sh

modprobe zfs
zpool import -N -a
# Prevent linux kernel from printing messages over passphrase prompt
echo 3 > /proc/sys/kernel/printk

while true; do
  stty -echo
  read -p "Passphrase for datasets: " PASS
  stty echo
  echo "$PASS" | zfs load-key encryption-root1 || continue
  echo "$PASS" | zfs load-key encryption-root2 || continue
  break
done

[mdPlusPlus]

@crocket Isn't this something else? The issue is about having multiple keys for a single zpool/zfs dataset, not a single key for multiple zpools/zfs datasets.

[crocket]

Yes, it is something else. I just wanted one password to unlock everything.

[Djarid]

Is there any movement on this issue? It is the only thing stopping us from using zfs :(

[f1d094]

Step 2: Malicious actor uses zfs change-key -i to re-encrypt zfs volumes

Just require the key to change it.

This would be a signficant improvement.

However, it does not entirely remediate the issue. If I have compromised your system it would be fairly trivial to keylog capture a passphrase or copy a locally stored key. Binary keys could also be captured when loaded, similar to keylogging.

[sjau]

Wouldn't have thought that after 6 years, this suddenly gets so much attention again.

[almereyda]

There is a discussion on Practical ZFS about

• The future of the ZFS native encryption - OpenZFS - Practical ZFS

in which some developer has announced to be actively working on some of these issues. I have pointed them to the discussion here. If any organisation around is able to funnel funds towards their direction, it would be possible to use an existing momentum and build upon that.

Esp. now with the review offer.

[1da1a172]

I agree with tcaputi on the re-key attack issue. It is not mitigated by what is proposed in this issue, but is mitigated with #12649. Also, if I'm understanding the underlying mechanisms correctly, old snapshots would still be using the old user key. Since a rekey only changes the user key (what the user enters), and not the master key (what the system actually encrypts blocks with), one could likely use zdb to grab the master key from an old backup and use that to restore data. This would require some in depth knowledge of zfs to pull off, but should be possible (realistically, with the help of a paid consultant).

I attempted to tackle this issue last summer, but the scope of the project defeated me. I'll be graduating in the spring, and I may be able to take another stab at it after that. I did manage to write a shell script that accomplishes much of the same thing using user properties. Obviously, this has its own small stack of caveats. I'll see what state that code is in and post a link to it here, hopefully in the next few days.

[tcaputi]

@1da1a172
Just to be sure we're all on the same page with the scope / current implementation (since there has been a lot of discussion). You are correct that the data is encrypted with the master key and the the master key is encrypted with the user key which is in turn derived from the user's password / key material along with some parameters stored on disk. All of these are not saved in a snapshot, and instead live in the top-level meta object store (MOS). When the user does a raw zfs send, the send file includes some extra instructions that REPLACE these on-disk values on the target system. So after a raw zfs recv has occurred, the old values are no longer accessible. Because of this, there isn't currently a good way to backup these values in a way where they will be permanently safe.

If we can save these parameters in a separate file which can be stored securely, these parameters should always be usable to decrypt the master key as long as the user remembers the corresponding password. This could be used in break-glass kind of situations, but it also comes with a (manageable) security risk that anyone who can access this file AND the password has a permanent way to access the dataset.

Theoretically, though, the implementation shouldn't be that hard. We just need to save the parameters needed to derive the user key to a file and then allow the proper hooks into the kernel to allow those parameters to be used instead of the ones that are saved in zfs. Then that would need to be wired out into the command line interface.

[f1d094]

Encrypted ZFS Volumes Vulnerable to Encryption Re-Key Attack

@tcaputi Hey Tom: It has been a couple months and nobody on the team has even responded to my CVE submission. Do you have an inside track on who to poke or best forum to raise this issue? Does anyone even own encryption since you left?

[tcaputi]

It has been a couple months and nobody on the team has even responded to my CVE submission. Do you have an inside track on who to poke or best forum to raise this issue?

I'm not 100% sure who the best person to ask about this is at the moment. In general, zfs is a pretty large project with a ton of users and so the maintainers (including me, formerly) regularly had to pick and choose issues which we felt had the highest value and were the easiest / clearest to implement. In several cases, I prioritized work that was important to my company, which I think was fair since they were the ones paying me to work on an open source project. Other maintainers (on all sorts of open source projects, not just zfs) are in a similar boat. If you feel this issue has been overlooked I would consider sending an email to one of the core maintainers (such as @behlendorf or @ahrens), but just keep these challenges in mind when you do.

Does anyone even own encryption since you left?

In general, I would say this is a bit of a misunderstanding of how the project works. Nobody 'owns' encryption because all of the maintainers work on the whole project. I think all of us have bits of the code that we are more familiar with than others, but many new features (especially larger ones) tend to impact many areas. ZFS is able to do a lot of neat things because the borders between layers are a little blurrier than in a traditional filesystem / block device / caching stack.

Hope that helps.

[f1d094]

I certainly understand priorities and have an idea of the breadth/depth of the project. However I'm not looking for someone to drop everything and work the issue immediately, as much as that would make me happy and a lot less nervous. I am looking for someone to acknowledge that there is a vulnerability and approve the CVE. As evidenced by your need to explain to @1da1a172 (above), users need to be made aware that using encryption with the otherwise fantastic filesystem can potentially be a ransom-trap. This way each may weigh the risk according to their own requirements.

@ahrens: I see that you gave a thumbs up...could you have a peek at my CVE and forward to whoever needs to review?

If it is not valid or needs amending, I would be happy to be better informed and will make changes or withdraw as appropriate. Otherwise, please see that it is approved so it doesn't just sit in limbo...

[amano-kenji]

@f1d094 It seems that you need to create a separate issue for that. That issue doesn't seem relevant to multiple key slots.

[f1d094]

Sorry for hijacking this thread earlier. I finally made a separate issue #15952 please comment there. Thanks!

[shymega]

Agreed, it got a bit off-topic.

If multiple keyslots were implemented, I would state that we should put them behind a feature flag, and work with both Linux and (Free)BSD to bring their ZFS version/implementation up to date.

Has anyone from OpenZFS discussed multiple keyslots with the BSD and Linux folks?

[amano-kenji]

The problem is that native zfs encryption is unmaintained. Once somebody becomes dedicated to this, we can worry about BSD.

[amano-kenji]

It turns out that I needed only one key slot.

With keyformat=passphrase and keylocation=file:///path/to/keyfile, even if I lose the keyfile, I can reconstruct it if I remember the passphrase. I can just write the passphrase in keyfile.

I use the same passphrase for root filesystem and other filesystems. Before I knew this, I thought I needed one key slot for a keyfile and another key slot for a passphrase.

If your goal is just to unlock all zfs pools with one passphrase. You don't need multiple key slots.

I unlock the root filesystem with one passphrase, and the keyfile will unlock other filesystems with the passphrase.

[anna328p]

+1 for this feature request. I want to be able to auto-unlock using a key stored in a TPM with a backup passphrase stored in hard-copy if the motherboard fails.

[amano-kenji]

Are you sure you want TPM? It will just unlock your computer without password. If an adversary has physical access to your computer, an unlocked computer can be read easily.

[anna328p]

I can use the TPM with Secure Boot attestation.

[amano-kenji]

If your threat model is an advanced adversary with physical access to your computer, TPM and secure boot will only enable them to read everything from the hardware. Requiring a password for unlocking storage is more secure.

You have to assume they can just take your computer and read what's on it. Technical adversaries who have unlimited physical access will just read what's on a machine that doesn't require a password for unlocking storage. NSA. FBI. CIA. Etc...

And, TPM and secure boot have been pushed by microsoft because they want control over your computers. Those mechanisms will be used to take control of your computer away from you. I just disable secure boot and TPM when they are available. If I can't disable them, then they probably exist to benefit microsoft and their allies. TPM and secure boot will be used to implement hardware-level DRM and prevent softwares that haven't been approved by microsoft from running.

Once attackers gain access to your computer remotely via software vulnerabilities, TPM and secure boot are useless.

TPM and secure boot cannot protect you from physical attacks and software vulnerabilities.

[Majiir]

@anna328p In case you missed the earlier comment, there is a slightly convoluted workaround you can use to combine LUKS key management with ZFS native encryption:

Another way to get LUKS key management with ZFS is to create a tiny (unencrypted) zvol, use LUKS on that zvol, format the LUKS volume with something simple like ext4, and store the ZFS key there. I run this configuration on one of my machines to unlock a ZFS root in initrd using any of the LUKS keyslots.

As you described, I have a recovery key in one keyslot, and for the other I use a TPM and Secure Boot. I also use a PIN with the TPM. The effect is that I enter the PIN at boot, and my ZFS filesystems are unlocked and mounted. If I lose the PIN or the TPM PCRs don't match, I have to enter the recovery key.

Since the LUKS volume is a zvol on the same pool, it gets the same benefits from ZFS as other data (replication, bitrot protection, etc).

[amano-kenji]

Also, if your computer boots fully without any password, any attacker who stole your computer will not even need to circumvent encryption because the operating system is already fully functional without a password.

[amano-kenji]

@Majiir If you are worried about losing keyfile, you can use a passphrase keyfile which contains a passphrase. If you lose the keyfile, you can reconstruct it from your memory. You can remember the passphrase if you use it everyday.

I unlock the root filesystem with one passphrase, and the other zfs file systems are unlocked with the same passphrase keyfile on the root filesystem.

[1da1a172]

Well, I meant to post this a long time ago. It is another work-around using zfs user properties.
https://github.com/1da1a172/zfs-multisecret

[Majiir]

@Majiir If you are worried about losing keyfile, you can use a passphrase keyfile which contains a passphrase. If you lose the keyfile, you can reconstruct it from your memory. You can remember the passphrase if you use it everyday.

I'm not worried about anything. I was just explaining the mechanics of the workaround.

I unlock the root filesystem with one passphrase, and the other zfs file systems are unlocked with the same passphrase keyfile on the root filesystem.

I understand the scheme that you are using. However, it does not meet my needs, nor those of some others who have posted here. Multiple keyslots are necessary in order to have a recovery key in addition to a normal passphrase that's tied to a security key or TPM, for example.

[1da1a172]

The solution I posted gives you multiple key slots. It just does it by layering on top of what already exists instead of adding it as a first party feature (which would obviously be better).

[anna328p]

@amano-kenji I'm not worried about state-level adversaries backdooring my TPM. I'm not worried about being forced to sign stuff with Microsoft's keys (my laptop, like all x86_64 machines with secure boot tend to, allows custom key management) or MS somehow spying on me via some data I load into a chip on my motherboard. And I require a login password on my machine.

My use-case is convenience and allowing unattended boot while making it difficult to read personal information from my laptop if it is stolen or lost.

@Majiir Thank you for that workaround, it's really clever and I will probably do something like that on my next install.

[amano-kenji]

@anna328p Unattended boot means any hacker who has physical access to your computer can read your computer.

At the very least, you should protect TPM with a password.

[anna328p]

@anna328p Unattended boot means any hacker who has physical access to your computer can read your computer.

...if they have arbitrary code execution inside getty or gdm.

No, they can't just freeze and pull the RAM, my CPU has transparent memory encryption and the RAM on my laptop is soldered.

Either way, my threat model isn't a hacker who would have detailed knowledge of exploiting obscure unix setups, it's someone who steals my computer or picks it up if I lose it.

[amano-kenji]

If NSA steals your laptop, they will boot it up and somehow read it. TPM has already been backdoored for state-level adversaries. But, your threat model isn't NSA or mossad. Frankly, it is they who have valuable information. Not you.

[norpol]

@amano-kenji can you please stop this? This has nothing to do with the ticket and also is completely out of place to unsolicitely educate someone on your personal opinions on TPM and state level attackers.