file-reflink, a storage driver optimized for CoW filesystems

[rustybird]

This adds the file-reflink storage driver. It is never selected automatically for pool creation, especially not the creation of varlibqubes (though it can be used if set up manually).

The code is quite small:

	reflink.py	lvm.py	file.py + block-snapshot
sloccount	334 lines	447 (134%)	570 (171%)

Background: btrfs and XFS (but not yet ZFS) support instant copies of individual files through the FICLONE ioctl behind cp --reflink. Which file-reflink uses to snapshot VM image files without an extra device-mapper layer. All the snapshots are essentially freestanding; there's no functional origin vs. snapshot distinction.

In contrast to 'file'-on-btrfs, file-reflink inherently avoids CoW-on-CoW. Which is a bigger issue now on R4.0, where even AppVMs' private volumes are CoW. (And turning off the lower, filesystem-level CoW for 'file'-on-btrfs images would turn off data checksums too, i.e. protection against bit rot.)

Also in contrast to 'file', all storage features are supported, including

• any number of revisions_to_keep
• volume.revert()
• volume.is_outdated
• online fstrim/discard

Example tree of a file-reflink pool - *-dirty.img are connected to Xen:

• /var/lib/testpool/appvms/foo/volatile-dirty.img
• /var/lib/testpool/appvms/foo/root-dirty.img
• /var/lib/testpool/appvms/foo/root.img
• /var/lib/testpool/appvms/foo/private-dirty.img
• /var/lib/testpool/appvms/foo/private.img
• /var/lib/testpool/appvms/foo/private.img@2018-01-02T03:04:05Z
• /var/lib/testpool/appvms/foo/private.img@2018-01-02T04:05:06Z
• /var/lib/testpool/appvms/foo/private.img@2018-01-02T05:06:07Z
• /var/lib/testpool/appvms/bar/...
• /var/lib/testpool/appvms/...
• /var/lib/testpool/template-vms/fedora-26/...
• /var/lib/testpool/template-vms/...

It looks similar to a 'file' pool tree, and in fact file-reflink is drop-in compatible:

$ qvm-shutdown --all --wait
$ systemctl stop qubesd
$ sed 's/ driver="file"/ driver="file-reflink"/g' -i.bak /var/lib/qubes/qubes.xml
$ systemctl start qubesd
$ sudo rm -f /path/to/pool/*/*/*-cow.img*

If the user tries to create a fresh file-reflink pool on a filesystem that doesn't support reflinks, qvm-pool will abort and mention the setup_check=no option. Which can be passed to force a fallback on regular sparse copies, with of course lots of time/space overhead. The same fallback code is also used when initially cloning a VM from a foreign pool, or from another file-reflink pool on a different mountpoint.

journalctl -fu qubesd will show all file-reflink copy/rename/remove operations on VM creation/startup/shutdown/etc.

---

TODO: port unit tests

[rustybird]

Does CI policy require no new pylint warnings? There's one protected-access warning that I had planned on leaving in, because the accessed attribute isn't mine.

(I've also fixed one warning and silenced one false positive error. They weren't printed by my local pylint version.)

[rustybird]

There's one protected-access warning that I had planned on leaving in, because the accessed attribute isn't mine.

On second thought, let me handle that more cleanly...

[marmarek]

Does CI policy require no new pylint warnings?

Generally yes. But in some specific cases, muting with # pylint: disable=... is ok too. It's here to at least force developer to look at those cases.

[codecov]

Codecov Report

Merging #188 into master will decrease coverage by 1.66%.
The diff coverage is 0.37%.

@@            Coverage Diff             @@
##           master     #188      +/-   ##
==========================================
- Coverage   56.32%   54.65%   -1.67%     
==========================================
  Files          55       56       +1     
  Lines        8734     9000     +266     
==========================================
  Hits         4919     4919              
- Misses       3815     4081     +266

Flag	Coverage Δ
#unittests	54.65% <0.37%> (-1.67%)	⬇️

Impacted Files	Coverage Δ
qubes/vm/dispvm.py	67.12% <ø> (ø)	⬆️
qubes/vm/appvm.py	90% <ø> (ø)	⬆️
qubes/storage/reflink.py	0% <0%> (ø)
qubes/app.py	74.47% <100%> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9b5256f...1695a73. Read the comment docs.

[rustybird]

I've fixed the underlying cause of the warning (and added a previously missing fsync after flush).

[rustybird]

In contrast to 'file'-on-btrfs, file-reflink inherently avoids CoW-on-CoW.

Huh, that's not actually true for the root volume of TemplateBasedVMs and DispVMs - even in R4.0, they still implement their own device-mapper based CoW thing, /dev/mapper/dmroot, inside the VM, using the second partition of volatile. Is my understanding correct that this approach is simply a remnant of the pre-R4.0 days? @marmarek @kalkin

If so, I propose to make these root volumes work exactly like the private volume of DispVMs, using a two step transition:

1. Flip their root volume from read-only to read-write in dom0.
2. At some later point (maybe R4.1), rip out the dmroot stuff inside the VMs.

Old e.g. backed up VMs still using dmroot would continue to work after that.

[marmarek]

Huh, that's not actually true for the root volume of TemplateBasedVMs and DispVMs - even in R4.0, they still implement their own device-mapper based CoW thing, /dev/mapper/dmroot, inside the VM, using the second partition of volatile. Is my understanding correct that this approach is simply a remnant of the pre-R4.0 days? @marmarek @kalkin

This is intentionally left in place, to allow VM to verify original root image. This would allow semi-untrusted storage domain. See original architecture paper. This part is yet to be implemented...

Since this is unused right now, I think its ok to disable it, but not completely remove. And leave clear comment why it is this way. So:

1. Flip their root volume from read-only to read-write in dom0.

Yes.

2. At some later point (maybe R4.1), rip out the dmroot stuff inside the VMs.

No. And not really needed, as currently if /dev/xvda is read-write, then /dev/mapper/dmroot is just a symlink (not dm-linear as it was in R3.2).

BTW Are you waiting for me on this PR? I've skimmed through it and looks fine, but haven't done careful review yet. Are you going to port tests in this PR, or separate at some future time?

[rustybird]

This would allow semi-untrusted storage domain.

Ah, okay.

Since this is unused right now, I think its ok to disable it, but not completely remove. And leave clear comment why it is this way. So:

Flip their root volume from read-only to read-write in dom0.

Yes.

Done.

At some later point (maybe R4.1), rip out the dmroot stuff inside the VMs.

No. And not really needed, as currently if /dev/xvda is read-write, then /dev/mapper/dmroot is just a symlink (not dm-linear as it was in R3.2).

That's fantastic!

BTW Are you waiting for me on this PR? I've skimmed through it and looks fine, but haven't done careful review yet. Are you going to port tests in this PR, or separate at some future time?

If possible, I'd implement tests in a separate PR, before submitting a third one to automatically use file-reflink (instead of 'file') for varlibqubes when a btrfs-without-LVM layout has been set up during R4.1 installation. As for this first PR, getting it merged would conveniently stop breaking my qubesd during qubes-core-admin package upgrades. 8)

Hey while I have your ear, should is_outdated() return True if the source volume has been started but not yet been stopped since the snapshot? file-reflink currently returns True only after source volume commit (because restarting the consumer volume would still give the same data), but I can change that.

[marmarek]

As for this first PR, getting it merged would conveniently stop breaking my qubesd during qubes-core-admin package upgrades. 8)

Not sure if you know, but you can install additional storage pool drivers from outside of qubes-core-admin. Just provide appropriate entry point (as you've done here in setup.py). That wouldn't work for default_pool, but you can set it using qubes-prefs.

file-reflink currently returns True only after source volume commit (because restarting the consumer volume would still give the same data),

That's ok.

[marmarek]

Better patch it directly where volume_config is defined (and similar for other VM types).

[marmarek]

And please separate commit for such changes.

[rustybird]

Done.

[rustybird]

Not sure if you know, but you can install additional storage pool drivers from outside of qubes-core-admin. Just provide appropriate entry point (as you've done here in setup.py).

Thanks, I'll give that a try.