Chacha20-Poly1305 encryption

[robn]

I’ve added a new encryption option to OpenZFS, chacha20-poly1305, which implements the Chacha20-Poly1305 AEAD described in RFC 8439. I’m interested in feedback, code review, testing and benchmarking from anyone interested in such things!

Important note: I have no particular skill in math, cryptography or secure software. I’m as confident as a reasonably good programmer can be that this is right, but I wouldn’t go trusting your secrets to this just yet!

Closes #8679.

Rationale

AES (and particularly the default AES-GCM mode used in OpenZFS) is known to be very slow on systems without hardware assistance. There are many such machines out there could make good use of OpenZFS, especially low-cost machines and small boards that would otherwise make very nice storage machines. The Raspberry Pi series of machines are a good example.

The best option for these systems is an encryption option that performs well in software. Chacha20-Poly1305 is the current “standard” option for this in many contexts, and is a good choice for OpenZFS.

More information about the rationale can be found in Google’s work on Adiantum (though their solution is necessarily different because unlike OpenZFS it operates on whole disk blocks).

Implementation

Core algorithms

I’ve taken the “core” Chacha20 and Poly1305 implementations from Loup Vaillant’s Monocypher. They are small and easy to read and understand, and the author has written extensively about its development and in particular the mistakes made along the way, which (at least to me) inspires confidence in the implementation.

I considered a number of other implementations, including those in Sodium and OpenSSL as well as a number of other smaller implementations. Despite the relative simplicity of the algorithms, there is lot of variability in the overall quality and usability of different implementations, so I opted for the most straightforward ones I could find.

ICP/KCF module

I’ve written a new KCF-style module for the ICP, that implements the AEAD (the “construction” that binds the cipher (ChaCha20) and authenticator (Poly1305) together) and the glue to make it available to OpenZFS proper.

Illumos does not currently support these algorithms, so it wasn’t possible to just take one from there. I’ve implemented a module, however it only implements what OpenZFS needs and is not sufficiently general-purpose to contribute back to Illumos. I have no plans to do this.

The rest of the module is as simple as I could figure out how to make, especially in some of the buffer management which is quite complex in the AES modes.

Tests

I’ve added a test program that runs the test vectors in RFC 8439. For Chacha20 and Poly1305 this is mostly a basic sanity check to assist in the future with possible alternate implementations (eg hardware-accelerated versions), as there are many implementations out there that have surprising quirks even when used correctly (most commonly endianness problems).

There’s also a full module test, checking that the AEAD is constructed correctly (again using the test vectors) and also makng sure the ICP module interface works.

The tests are only there to confirm basic operation; they have nothing to say about security properties or correctness.

Performance

I have only done very light performance tests to gain a rough idea of how much “better” or “worse” this is than aes-256-gcm on a handful of machines I have here. I do not know how to drive fio very well and I did not make much effort to control variables like other programs running on the machine, so I will not defend these numbers very hard. I’ve run them enough times to be reasonably confident they’re not wildly off the mark.

Method and fio output: https://gist.github.com/robn/4b40251ba84728cf8e669649ce77ad56

• Broadcom BCM2711 (Raspberry Pi 4B): ~2.4x faster (no AES hardware accel)
• Celeron J3455 (Fitlet2): ~1.3x faster (partial AES hardware accel)
• Core i7-8565U (Thinkpad X1 Gen7): ~1.2x slower (full AES hardware accel)

(ChaPoly in software coming so close to AES in hardware makes me think there’s probably still optimisation opportunities in the AES code, but I have not really considered this even a little bit).

Other notes

• This is only for Linux right now. It should be straightforward to implement for FreeBSD too, which has the necessary crypto primitives available in the kernel. I will try to add that soon. Now with FreeBSD support.
• Related to that, I have not fully updated the tests yet because I don’t want to them to break any automatic PR test stuff that might run on FreeBSD. Done.
• XChacha20 (the “better” Chacha variant) isn’t an option because OpenZFS cannot provide a large enough nonce to initialise it. In practice it wouldn’t matter because OpenZFS already takes steps to mitigate nonce reuse.

TODO

After discussion in monthly call:

• figure out if anyone else wants this feature
• document when to choose chapoly vs AES
• test chapoly dataset on older ZFS (failure modes)
  • get encryption property
  • attempt to mount
  • (other dataset ops?)
• test send/recv raw send
  • both sides support
  • source supports, receiver doesn't
  • source doesn't support (?), receiver does
  • neither support (?)
• add feature flag

Future work

Just stuff I thought about along the way that I’m not committing to but might like to look at in the future.

• I want to at least try accelerated versions (read: assembly implementations of Chacha20 and/or Poly1305) of this in the future, to see if there are significant gains to be had. That will likely make a runtime selection mechanism necessary (like icp_aes_impl and icp_gcm_impl).
• The entire ICP/KCF system has a lot of unused stuff and overhead that I’d like to see radically slimmed down. I think its probably possible lift zio_crypt.c up to be common to all ports, with a much smaller layer that calls down to OS-provided cryptographic functions or uses the bundled ones where the system doesn’t have something available. That’s not just for Linux, but would help in (for example) Illumos, which has AES in the kernel but not ChaPoly. I’d probably wait to see what the Windows and Mac ports look like when they’re finally merged before considering this seriously.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[robn]

Last push has support for FreeBSD, which is very straightforward because all the crypto code is already available in the kernel.

Importantly, I've done a interop test: a pool with a chacha20-poly1305 dataset created on Linux can be imported and read and written to on FreeBSD, and back again. I expected that, but its nice to see it in action.

[lundman]

I appreciate very much you made it part of ICP, it means nearly no work needs to be done for me.

[robn]

So, what's the next step for this? If its just waiting until volunteers have time to review, no problem, I get it! If there's something I could do or need to do though, I'd like to get on to that.

(Should I be dropping in on the monthly call to talk about it?)

[ryao]

So, what's the next step for this? If its just waiting until volunteers have time to review, no problem, I get it! If there's something I could do or need to do though, I'd like to get on to that.

Yes. There are a number of PRs waiting review and reviewers have limited bandwidth to look at them. Larger PRs like this take substantially longer to see review.

(Should I be dropping in on the monthly call to talk about it?)

That might help attract reviewers.

[evrim]

Hey there,

I must admit I'm a little bit confused here. Linux has already support for Chacha20Poly1305 under lib/ and relevant arch/, why do we need another implementation?

[robn]

@evrim there's two ways to look at it. One is that Linux's crypto code is only available to GPL kernel modules, which OpenZFS is not. The other is that by doing it in the ICP, other ICP-using ports of OpenZFS (Windows and macOS) can use it.

[jittygitty]

@robn Some (#11357) of us are building ZFS and setting it as "GPL" module when we build, how can we then use the existing Linux built-in GPL implementations of chacha20 etc? thx

[robn]

@jittygitty there's no code in ZFS right now that can call into the kernel crypto API, regardless of how you build it - you will always be using the ICP for crypto even for AES modes. You'd need a variant of zio_crypt for that, which is well outside the scope of this PR.

[robn]

We talked about this at the 31 Jan call. Feedback was generally favourable, thanks all!

I was waiting to check my notes against the recording but that hasn't surfaced yet and I want to get moving on this a bit this week. So here's what I wrote down. If it turns out I got something wrong, I will of course adjust it.

• We agreed that it was fine to use Monocypher, under a more general principle that we don't really have the resources or skills to review crypto implementations ourselves, and trusting external implementations that appear to know what they're doing is a reasonable approach for us. The existence of a proof was welcomed (Add machine-checked proofs for limb overflows LoupVaillant/Monocypher#246).
• There should be wording in the manpage to give guidance about using AES vs chapoly that mostly boils down to "AES if you have hardware acceleration, chapoly otherwise".
• More testing is required. I didn't write down specifics, but I've been thinking about it, and this is at least around upgrade/downgrade from/to ZFS versions with/without chapoly support, and how send/receive copes with same. Possibly this will require a new feature (like we do with checksum & compression algorithms)

Finally, and most importantly, it was agreed that this is obviously a nice feature, but if no one else wants it then perhaps it we shouldn't be taking on the maintenance burden. I will soon go and try to get a rough show of hands to see if anyone would else would use this. Probably, a carefully-worded query to the mailing list, discussion forum and reddit, and go from there.

I'll return this to draft status for the moment, and add some additional todo items to the top post.

[mcmilk]

We talked about this at the 31 Jan call. Feedback was generally favourable, thanks all!

I was waiting to check my notes against the recording but that hasn't surfaced yet and I want to get moving on this a bit this week. So here's what I wrote down. If it turns out I got something wrong, I will of course adjust it.

• We agreed that it was fine to use Monocypher, under a more general principle that we don't really have the resources or skills to review crypto implementations ourselves, and trusting external implementations that appear to know what they're doing is a reasonable approach for us. The existence of a proof was welcomed (Add machine-checked proofs for limb overflows LoupVaillant/Monocypher#246).
• There should be wording in the manpage to give guidance about using AES vs chapoly that mostly boils down to "AES if you have hardware acceleration, chapoly otherwise".
• More testing is required. I didn't write down specifics, but I've been thinking about it, and this is at least around upgrade/downgrade from/to ZFS versions with/without chapoly support, and how send/receive copes with same. Possibly this will require a new feature (like we do with checksum & compression algorithms)

Finally, and most importantly, it was agreed that this is obviously a nice feature, but if no one else wants it then perhaps it we shouldn't be taking on the maintenance burden. I will soon go and try to get a rough show of hands to see if anyone would else would use this. Probably, a carefully-worded query to the mailing list, discussion forum and reddit, and go from there.

I'll return this to draft status for the moment, and add some additional todo items to the top post.

I find this feature usefull and would help with this PR if you want.

There is also a rfc7905 which update TLS / DTLS with ChaCha20-Poly1305 - so why should OpenZFS not go with the time and have this option.

[robn]

I did all the testing I wanted to do with a view to adding a feature flag.

I created a new pool on Linux with master 620a977 + this PR, exported it, then rebuilt on master alone and imported the pool.

All the encryption properties are returned except for encryption itself:

root@fitlet:~# zfs get encryption,encryptionroot,keyformat,keylocation,keystatus sandisk
NAME     PROPERTY        VALUE               SOURCE
sandisk  encryption      -                   -
sandisk  encryptionroot  sandisk             -
sandisk  keyformat       passphrase          -
sandisk  keylocation     file:///tmp/zfskey  local
sandisk  keystatus       unavailable         -

This appears to be because the underlying value is out bounds for this enum, so the default (ZIO_CRYPT_OFF) is returned.

Attempting to load the key fails, in a slightly weird way:

root@fitlet:~# zfs load-key sandisk
Key load error: Incorrect key provided for 'sandisk'.

and of course it can't be mounted:

root@fitlet:~# zfs mount sandisk
cannot mount 'sandisk': encryption key not loaded

I say it fails in a slightly weird way, because I think its just lucky this time.

If we're compiled with --enable-debug, it fails a lot earlier:

root@fitlet:~# zfs load-key sandisk

Message from syslogd@fitlet at Mar  4 20:25:08 ...
 kernel:[1281777.454629] VERIFY3(crypt < ZIO_CRYPT_FUNCTIONS) failed (9 < 9)

Message from syslogd@fitlet at Mar  4 20:25:08 ...
 kernel:[1281777.454642] PANIC at zio_crypt.c:568:zio_crypt_key_unwrap()

When asserts are disabled, it seems like its only doing ok because the crypto table entry does actually exist for crypt == 9, its the NULL entry at the end. Linux zio_crypt_key_unwrap sets up to decrypt the key and even tries, fails and gets an ECKSUM, which is then swallowed and EACCES is returned. On FreeBSD I think it will fall out a little earlier, at the top of zio_do_crypt_uio_opencrypto, and returns ENOTSUP, which again gets swallowed and replaced for EACCES.

I do think that behaviour is lucky. For the next new cipher suite crypt == 10, it'd be running off the end of the crypto table. Even now, on Linux, its trying to decrypt using only a partial state, so it could easily crash in other circumstances.

So probably this alone is enough to suggest a feature flag is warranted.

[robn]

I did further send/receive testing as well.

When sender and receiver both support chapoly, it works exactly as it does for AES - raw send doesn't require the dataset loaded on the sender, not-raw does. All fine.

When there's no chapoly support, raw send still works. A stream is produced, and if that's later received to a chapoly-supporting pool, it goes in just fine, key can be loaded and it can't be mounted.

Importing a chapoly stream into a pool with no support fails:

root@fitlet:~# zfs recv -v sandisk/recv@snap < snap-from-chapoly.zfs                                                           
receiving full stream of sandisk@snap into sandisk/recv@snap                                                                   
cannot receive new filesystem stream: invalid backup stream     

Its not totally clear to me without looking inside that that's indicating. Presumably, invalid property of some sort. I'll check more closely tomorrow.

But again, likely the a feature flag is going to benefit.

[robn]

I find this feature usefull and would help with this PR if you want.

@mcmilk sorry I missed this! Thanks! At this point I don't think there's much else to do other than as much code review as we can. The main thing I want to feel good about is that I'm not misusing the crypto stuff in some important way. General bugs would be bad but survivable, but the crypto being no good is worse than not having it at all. I think its probably fine, but if you've got good ideas that'd be great.

[robn]

Ehh, I've got my brain in backwards this evening. A feature in the dataset isn't going to make an old implementation magically able to cope with it.

That said, I'll see if I can do something now to help for the next new crypto algorithm.

[robn]

Last push updates Monocypher to 4.0.2, adds a test to ensure replication streams with encrypted datasets are accepted or not depending on whether or not the feature flag is set, and collapses it down into a single commit.

The CodeQL failures at the same as the ones we saw right at the beginning, and are safe.

@behlendorf how do you feel about this nearly a year on? :) I don't think there's any more to do.

[robszy]

Did anyone thought about wireguard zinc implementation like in comment: #8679 (comment)
It would be out of the box a high performance ( vector extensions ) formally verified implementation from wireguard.

Today we will only have implementation which don't use vector extensions targeted only at raspbery pi like devices :(

[ErrorNoInternet]

Hi, thanks for working on this!

I decided to try this out on a Raspberry Pi 4B (4 core, 4 GB RAM) and a 1 TB HDD connected via a USB enclosure.

One issue though, I noticed I'm only getting about 30 MB/s read speeds and massive amounts of CPU usage. I can get 100 MB/s write speeds just fine (and this matches the speed of the chacha20 Rust crate without neon).

Without any encryption, I get 150 MB/s write and 170 MB/s read.

Upon further inspection, it seems to be because of the massive amounts of messages in dmesg. I don't get them with AES-256-GCM though.

[77650.336483] CPU: 3 PID: 1235586 Comm: z_rd_int Tainted: P         C O       6.6.26 #1-NixOS
[77650.336491] Hardware name: Raspberry Pi 4 Model B Rev 1.4 (DT)
[77650.336494] Call trace:
[77650.336498]  dump_backtrace+0x9c/0x128
[77650.336507]  show_stack+0x20/0x38
[77650.336511]  dump_stack_lvl+0x48/0x60
[77650.336520]  dump_stack+0x18/0x28
[77650.336526]  spl_kmem_alloc+0x140/0x158 [spl]
[77650.336557]  chapoly_decrypt_atomic+0x74/0x2a0 [zfs]
[77650.337177]  crypto_decrypt+0x80/0x1c0 [zfs]
[77650.337596]  zio_do_crypt_uio+0x1b8/0x228 [zfs]
[77650.337971]  zio_do_crypt_data+0x1e8/0xf98 [zfs]
[77650.338522] Large kmem_alloc(131328, 0x0), please file an issue at:
               https://github.com/openzfs/zfs/issues/new
[77650.338349]  spa_do_crypt_abd+0x1fc/0x308 [zfs]
[77650.338802]  zio_decrypt+0xf4/0x498 [zfs]
[77650.339167]  zio_pop_transforms+0x68/0x98 [zfs]
[77650.339572]  zio_done+0x12c/0x10e0 [zfs]
[77650.339991]  zio_execute+0xb4/0x160 [zfs]
[77650.340353]  taskq_thread+0x2cc/0x518 [spl]
[77650.340388]  kthread+0xe8/0xf8
[77650.340395]  ret_from_fork+0x10/0x20
[77650.340403] CPU: 2 PID: 1235585 Comm: z_rd_int Tainted: P         C O       6.6.26 #1-NixOS
[77650.340409] Hardware name: Raspberry Pi 4 Model B Rev 1.4 (DT)
[77650.340412] Call trace:
[77650.340415]  dump_backtrace+0x9c/0x128
[77650.340424]  show_stack+0x20/0x38
[77650.340428]  dump_stack_lvl+0x48/0x60
[77650.340437]  dump_stack+0x18/0x28
[77650.340443]  spl_kmem_alloc+0x140/0x158 [spl]
[77650.340474]  chapoly_decrypt_atomic+0x74/0x2a0 [zfs]
[77650.341008]  crypto_decrypt+0x80/0x1c0 [zfs]
[77650.341407]  zio_do_crypt_uio+0x1b8/0x228 [zfs]
[77650.341814]  zio_do_crypt_data+0x1e8/0xf98 [zfs]
[77650.342376] Large kmem_alloc(131328, 0x0), please file an issue at:
               https://github.com/openzfs/zfs/issues/new
[77650.342204]  spa_do_crypt_abd+0x1fc/0x308 [zfs]
[77650.342694]  zio_decrypt+0xf4/0x498 [zfs]
[77650.343064]  zio_pop_transforms+0x68/0x98 [zfs]
[77650.343436]  zio_done+0x12c/0x10e0 [zfs]
[77650.343805]  zio_execute+0xb4/0x160 [zfs]
[77650.344177]  taskq_thread+0x2cc/0x518 [spl]
[77650.344211]  kthread+0xe8/0xf8
[77650.344218]  ret_from_fork+0x10/0x20
[77650.344227] CPU: 3 PID: 1235586 Comm: z_rd_int Tainted: P         C O       6.6.26 #1-NixOS
[77650.344234] Hardware name: Raspberry Pi 4 Model B Rev 1.4 (DT)
[77650.344238] Call trace:
[77650.344241]  dump_backtrace+0x9c/0x128
[77650.344251]  show_stack+0x20/0x38
[77650.344255]  dump_stack_lvl+0x48/0x60
[77650.344263]  dump_stack+0x18/0x28
[77650.344269]  spl_kmem_alloc+0x140/0x158 [spl]
[77650.344300]  chapoly_decrypt_atomic+0x74/0x2a0 [zfs]
[77650.344873]  crypto_decrypt+0x80/0x1c0 [zfs]
[77650.345284]  zio_do_crypt_uio+0x1b8/0x228 [zfs]
[77650.345685]  zio_do_crypt_data+0x1e8/0xf98 [zfs]
[77650.346261] Large kmem_alloc(131328, 0x0), please file an issue at:
               https://github.com/openzfs/zfs/issues/new
[77650.346059]  spa_do_crypt_abd+0x1fc/0x308 [zfs]
[77650.346512]  zio_decrypt+0xf4/0x498 [zfs]
[77650.346873]  zio_pop_transforms+0x68/0x98 [zfs]
[77650.347266]  zio_done+0x12c/0x10e0 [zfs]
[77650.347755]  zio_execute+0xb4/0x160 [zfs]
[77650.348141]  taskq_thread+0x2cc/0x518 [spl]
[77650.348179]  kthread+0xe8/0xf8
[77650.348187]  ret_from_fork+0x10/0x20
[77650.348195] CPU: 2 PID: 1235585 Comm: z_rd_int Tainted: P         C O       6.6.26 #1-NixOS
[77650.348202] Hardware name: Raspberry Pi 4 Model B Rev 1.4 (DT)
[77650.348205] Call trace:
[77650.348209]  dump_backtrace+0x9c/0x128
[77650.348218]  show_stack+0x20/0x38
[77650.348222]  dump_stack_lvl+0x48/0x60
[77650.348231]  dump_stack+0x18/0x28
[77650.348236]  spl_kmem_alloc+0x140/0x158 [spl]
[77650.348267]  chapoly_decrypt_atomic+0x74/0x2a0 [zfs]
[77650.348746]  crypto_decrypt+0x80/0x1c0 [zfs]
[77650.349117]  zio_do_crypt_uio+0x1b8/0x228 [zfs]
[77650.349540]  zio_do_crypt_data+0x1e8/0xf98 [zfs]
[77650.350165] Large kmem_alloc(131328, 0x0), please file an issue at:
               https://github.com/openzfs/zfs/issues/new
[77650.349927]  spa_do_crypt_abd+0x1fc/0x308 [zfs]
[77650.350404]  zio_decrypt+0xf4/0x498 [zfs]
[77650.350766]  zio_pop_transforms+0x68/0x98 [zfs]
[77650.351133]  zio_done+0x12c/0x10e0 [zfs]
[77650.351500]  zio_execute+0xb4/0x160 [zfs]
[77650.351916]  taskq_thread+0x2cc/0x518 [spl]
[77650.351959]  kthread+0xe8/0xf8
[77650.351968]  ret_from_fork+0x10/0x20
[77650.351977] CPU: 3 PID: 1235586 Comm: z_rd_int Tainted: P         C O       6.6.26 #1-NixOS
[77650.351984] Hardware name: Raspberry Pi 4 Model B Rev 1.4 (DT)
[77650.351987] Call trace:
[77650.351990]  dump_backtrace+0x9c/0x128
[77650.352001]  show_stack+0x20/0x38
[77650.352005]  dump_stack_lvl+0x48/0x60
[77650.352013]  dump_stack+0x18/0x28
[77650.352019]  spl_kmem_alloc+0x140/0x158 [spl]
[77650.352049]  chapoly_decrypt_atomic+0x74/0x2a0 [zfs]
[77650.352645]  crypto_decrypt+0x80/0x1c0 [zfs]
[77650.353010]  zio_do_crypt_uio+0x1b8/0x228 [zfs]
[77650.353396]  zio_do_crypt_data+0x1e8/0xf98 [zfs]
[77650.353765]  spa_do_crypt_abd+0x1fc/0x308 [zfs]
[77650.354077] Large kmem_alloc(131328, 0x0), please file an issue at:
               https://github.com/openzfs/zfs/issues/new
[77650.354217]  zio_decrypt+0xf4/0x498 [zfs]
[77650.354597]  zio_pop_transforms+0x68/0x98 [zfs]
[77650.354982]  zio_done+0x12c/0x10e0 [zfs]
[77650.355356]  zio_execute+0xb4/0x160 [zfs]
[77650.355727]  taskq_thread+0x2cc/0x518 [spl]
[77650.355761]  kthread+0xe8/0xf8
[77650.355769]  ret_from_fork+0x10/0x20
[77650.355778] CPU: 2 PID: 1235585 Comm: z_rd_int Tainted: P         C O       6.6.26 #1-NixOS
[77650.355786] Hardware name: Raspberry Pi 4 Model B Rev 1.4 (DT)
[77650.355789] Call trace:
[77650.355792]  dump_backtrace+0x9c/0x128
[77650.355802]  show_stack+0x20/0x38
[77650.355806]  dump_stack_lvl+0x48/0x60
[77650.355814]  dump_stack+0x18/0x28
[77650.355820]  spl_kmem_alloc+0x140/0x158 [spl]
[77650.355850]  chapoly_decrypt_atomic+0x74/0x2a0 [zfs]
[77650.356370]  crypto_decrypt+0x80/0x1c0 [zfs]
[77650.356742]  zio_do_crypt_uio+0x1b8/0x228 [zfs]
[77650.357155]  zio_do_crypt_data+0x1e8/0xf98 [zfs]
[77650.357576]  spa_do_crypt_abd+0x1fc/0x308 [zfs]
[77650.358118]  zio_decrypt+0xf4/0x498 [zfs]
[77650.358631]  zio_pop_transforms+0x68/0x98 [zfs]
[77650.359133]  zio_done+0x12c/0x10e0 [zfs]
[77650.359632]  zio_execute+0xb4/0x160 [zfs]
[77650.360101]  taskq_thread+0x2cc/0x518 [spl]
[77650.360141]  kthread+0xe8/0xf8
[77650.360148]  ret_from_fork+0x10/0x20
[77650.360160] CPU: 0 PID: 1857 Comm: z_rd_int Tainted: P         C O       6.6.26 #1-NixOS
[77650.360181] Hardware name: Raspberry Pi 4 Model B Rev 1.4 (DT)
[77650.360185] Call trace:
[77650.360188]  dump_backtrace+0x9c/0x128
[77650.360198]  show_stack+0x20/0x38
[77650.360203]  dump_stack_lvl+0x48/0x60
[77650.360212]  dump_stack+0x18/0x28
[77650.360218]  spl_kmem_alloc+0x140/0x158 [spl]
[77650.360252]  chapoly_decrypt_atomic+0x74/0x2a0 [zfs]
[77650.360861]  crypto_decrypt+0x80/0x1c0 [zfs]
[77650.361271]  zio_do_crypt_uio+0x1b8/0x228 [zfs]
[77650.361756]  zio_do_crypt_data+0x1e8/0xf98 [zfs]
[77650.362434]  spa_do_crypt_abd+0x1fc/0x308 [zfs]
[77650.363077]  zio_decrypt+0xf4/0x498 [zfs]
[77650.363705]  zio_pop_transforms+0x68/0x98 [zfs]
[77650.364241]  zio_done+0x12c/0x10e0 [zfs]
[77650.364844]  zio_execute+0xb4/0x160 [zfs]
[77650.365425]  taskq_thread+0x2cc/0x518 [spl]
[77650.365469]  kthread+0xe8/0xf8
[77650.365478]  ret_from_fork+0x10/0x20

zpool iostat -vw during read:

ata-WDC_WD10EZEX-08M2NA0_WD-WCC3F6RAKJ32     total_wait     disk_wait    syncq_wait    asyncq_wait
latency                                      read  write   read  write   read  write   read  write  scrub   trim  rebuild
------------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
1ns                                             0      0      0      0      0      0      0      0      0      0      0
3ns                                             0      0      0      0      0      0      0      0      0      0      0
7ns                                             0      0      0      0      0      0      0      0      0      0      0
15ns                                            0      0      0      0      0      0      0      0      0      0      0
31ns                                            0      0      0      0      0      0      0      0      0      0      0
63ns                                            0      0      0      0      0      0      0      0      0      0      0
127ns                                           0      0      0      0      0      0      0      0      0      0      0
255ns                                           0      0      0      0      0      0      0      0      0      0      0
511ns                                           0      0      0      0      0      0      0      0      0      0      0
1us                                             0      0      0      0      0      0      0      0      0      0      0
2us                                             0      0      0      0      0      0      0      0      0      0      0
4us                                             0      0      0      0      0      0      0      0      0      0      0
8us                                             0      0      0      0      0      0      0      0      0      0      0
16us                                            0      0      0      0      0      0      0      0      0      0      0
32us                                            0      0      0      0      0      0      0      0      0      0      0
65us                                            0      0      0      0      0      0      0      0      0      0      0
131us                                           0      0      0      0      0      0      0      0      0      0      0
262us                                           0      0      0      0      0      0      0      0      0      0      0
524us                                           0      0      0      0      0      0      0      0      0      0      0
1ms                                             0      0      0      0      0      0      0      0      0      0      0
2ms                                             0      0      0      0      0      0      0      0      0      0      0
4ms                                             0      0      0      0      0      0      0      0      0      0      0
8ms                                             0      0      0      0      0      0      0      0      0      0      0
16ms                                            0      0      0      0      0      0      0      0      0      0      0
33ms                                            0      0      0      0      0      0      0      0      0      0      0
67ms                                            0      0      0      0      0      0      0      0      0      0      0
134ms                                           0      0     29      0      0      0      0      0      0      0      0
268ms                                           0      0      0      0      0      0      0      0      0      0      0
536ms                                           0      0      0      0      0      0      0      0      0      0      0
1s                                              0      0      0      0      0      0      0      0      0      0      0
2s                                             29      0      0      0      0      0     29      0      0      0      0
4s                                              0      0      0      0      0      0      0      0      0      0      0
8s                                              0      0      0      0      0      0      0      0      0      0      0
17s                                             0      0      0      0      0      0      0      0      0      0      0
34s                                             0      0      0      0      0      0      0      0      0      0      0
68s                                             0      0      0      0      0      0      0      0      0      0      0
137s                                            0      0      0      0      0      0      0      0      0      0      0
-----------------------------------------------------------------------------------------------------------------------

[robn]

@ErrorNoInternet thanks for the report! I've pushed a small change that should at least deal with the log spam.

[ErrorNoInternet]

Nice! Getting stable 110+ MB/s read speeds now :) Thank you!

[robn]

Great to know, thanks :)

[rsalvaterra]

What's keeping this from being merged? I'm looking forward to it, since it's extremely useful for devices that meet the RAM requirements for ZFS but don't have CPUs with AES instructions/acceleration.

[robn]

@rsalvaterra still intended, just hasn't been anywhere on my radar for a good long while.

I've just rebased it to master, though barely any testing done. I wouldn't want it merged as-is though, at minimum I want to do a full read through and make sure I still like everything in it given everything I've learned about OpenZFS internals since I first wrote it.

[robn]

Last push removes the test vectors and runner in favour of a proper test set from the Project Wycheproof, matching what we have for AES-CCM and AES-GCM.

[robn]

I'm setting this to draft for the moment. It works, but for the moment I'm very uncomfortable about making the on-disk format change until I have chance to do some more cleanup work on the crypto code and infrastructure. I'll continue using this PR as something of a testbed for that.