HEAD issues mounting encrypted datasets/corrupting metadata

[65a]

System information

Type	HEAD/Linux 5.9
Distribution Name	Arch-ish
Distribution Version	n/a
Linux Kernel	5.9.1
Architecture	x86_64
ZFS Version	2.0.0-248_gd1d47691c
SPL Version	2.0.0-248_gd1d47691c

Describe the problem you're observing

Encrypted datasets failed to mount. Permanent errors were logged on the pool:
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.

The erroneous files were the "<0x0>" metadata files for each encrypted mountpoint

zfs mount -a returned "Input/Output error" for each mountpoint.

Describe how to reproduce the problem

Create a pool (possibly using encrypted send/recv) a few years ago with encrypted mountpoints, use HEAD variants of spl and zfs. Attempt to mount encrypted mountpoints.

Include any warning/errors/backtraces from the system logs

I was able to resolve the data access issue (but not the permanent pool error log) by reverting to 2.0.0-150_g3928ec533, which successfully mounts the encrypted datasets. Scrub under both versions returns no errors or checksum problems.

[65a]

I have a build with d1d4769 reverted, I will try that tomorrow after a full pool backup.

[gamanakis]

I am the author of d1d4769.
Which version of ZFS had you created the encrypted dataset with?

[gamanakis]

CC: @tcaputi @behlendorf

[65a]

I haven't actually bisected it to that commit yet, but that commit seemed relevant given the nature of the problem. I think I saw a segfault and some bizarre output during zfs load-key when I first brought up the affected version, but there was nothing in dmesg. Once I've backed up on the working version, I'll bring it up on the other two versions and see if I can get any more detail out of it.

Re: pool/dataset age: I'm not actually sure, the pool's disks are from approximately October 2019 (was running HEAD then as well), but the datasets may predate that. Definitely not that recent.

[gamanakis]

I am interested to know if d1d4769 is the one to blame. I did test with encrypted datasets created with 0.8.5 and have them raw-sent to d1d4769 as well as locally mounted by d1d4769 but did not see any problems.

Thank you for reporting it and helping troubleshoot!

[65a]

Backup is running now. I thought I had built a machine image with that commit reverted, but evidently I failed because the revision matched the failing version, I'll try again after backup completes. I can confirm that rolling forward to 2.0.0-248_gd1d47691c reproduces for me, and rolling back to 2.0.0-150_g3928ec533 fixes. If it's useful, the failing mounts aren't the encrypted dataset proper, but the encrypted (same key, inherited) sub-datasets under that. I don't see errors for the metadata for the top level dataset, only its children.

The dataset was created 2019-10-11, so it would have been created with HEAD zfs around that time, I usually update at least monthly.

[gamanakis]

I see what you mean regarding encrypted children. Just tested with 0.8.5 and d1d4769, raw sends and local mounting work as expected for encrypted (inherited key) children too. Could you provide a script that reproduces the faulty behaviour?

[65a]

I can confirm reverting d1d4769 does allow me to mount the pool. I'm not sure how to make this a test case, or why mounting the pool is failing unfortunately :(

[65a]

Is it possible my metadata is in a state that invalidates the assumptions in the commit? I've been running bleeding edge commits for about a year on the pool.

[gamanakis]

It seems you are running some intermediate version of encryption that existed in master between releases which is causing the problem with that commit, and is not covered by Errata3 in ZTS. I will take another look to see how we can resolve it.

[gamanakis]

I believe I found a way around this, in spa_do_crypt_objset_mac_abd() we could skip comparing the local_mac if OBJSET_FLAG_USERACCOUNTING_COMPLETE and OBJSET_FLAG_USEROBJACCOUNTING_COMPLETE are not set. This also resolves Errata3 in ZTS without having to check the key version in d1d4769.

[65a]

Any update here? I am still having to revert the patchset, I have various pools created at different times that have been affected. I'm also interested in understanding how I can clear the error log on my pools without recreating them.