COW cp (--reflink) support

[torn5]

This is a feature request for an implementation of the BTRFS_IOC_CLONE in zfsonlinux, or something similar, so that it's possible to COW-copy a single file in zero space, zero RAM and zero time, without having to enable super-expensive things like filesystem-wide deduplication (which btw is not zero RAM or zero time).

If it can be done at the directory level, so to clone entire directory trees with one call, even better.

On the mailing list, doubts were raised regarding semantics on:

1. Quotas, especially when the "copy" start to be modified.
2. Source and destination dataset don't use the same compression.
3. Source and destination dataset don't use the same encryption key.
4. Source and destination dataset haven't the same "copies" attribute.
5. Source and destination dataset haven't the same recordsize.

Firstly, I don't expect this to work across datasets. Secondly, I'd suggest that the same semantics of deduplication are used. It should just be a shortcut of 1) enabling deduplication + 2) copying the file by reading it byte-by-byte and writing it byte-by-byte elsewhere.

If you can implement exactly the BTRFS_IOC_CLONE, the same cp with --reflink that works for btrfs could be used here. If the IOC is different, we will also need patches to the cp program or another another cp program.

[behlendorf]

Link to zfs-discuss thread for this feature request:
http://groups.google.com/a/zfsonlinux.org/group/zfs-discuss/browse_thread/thread/9af181ed0129b77c#

[darthur]

I have my doubts regarding whether this is needed -- because the functionality is already available at the dataset(filesystem) level, and ZFS is intended to be implemented with fine-grained datasets(filesystems).

1. I see no reason why quotas should be special for this feature. Already if you turn on zfs copies, any new writes will write 2 or 3 times the number of blocks an application would expect. ZFS based systems are intended to have enough free space, and quotas are both difficult to calculate/understand and more forgiving than quotas on systems that do not have COW.
2. Same compression does not matter. If you have a cloned dataset and change the compression on one of them -- existing data is unchanged, the new compression setting only matters for new writes.
3. Clearly if encryption keys are different, then the data may not be stored in the same blocks and COW does not apply.
4. The copies attribute, like the compression settings, only applies to new writes. This is no different semantically than cloning a dataset (or promoting a snapshot).
5. This issue is similar to 3.

COW is already implemented across specific datasets -- e.g. clones or datasets with snapshots (promoted or otherwise). Therefore, I propose a more generally useful version of this request: implement or allow COW for all copy operations in the same pool base on a setting implemented at both the dataset and pool level.

[akorn]

Just to reiterate here what has been discussed in https://groups.google.com/a/zfsonlinux.org/forum/?fromgroups=#!msg/zfs-discuss/mvGB7QEpt3w/r4xZ3eD7nn0J -- snapshots, dedup and so on already provide most of the benefits of COW hardlink breaking, but not all.

Specifically, binaries and libs with the same inode number get mapped to the same memory locations for execution, which results in memory savings. This matters a lot with container based virtualisation where you may have dozens or hundreds of identical copies of libc6 or sshd or apache in memory. KSM helps there, but it needs madvise() and is not free (it costs CPU at runtime).

COW hardlinks are essentially free.

linux-vserver (http://linux-vserver.org/) already patches ext[234] and I think jfs to enable this sort of COW hardlink breaking functionality by abusing the immutable attribute and an otherwise unused attribute linux-vserver repurposes as "break-hardlink-and-remove-immutable-bit-when-opened-for-writing".

It would be great to have the same feature in zfsonlinux; its lack is one of the only reasons I can't put my vservers volume on zfs (the lack of posix or nfsv4 acl support is the oter reason).

So, to clarify, I'm requesting the following semantics:

1. create a hardlink to a file and set some xattr(?) on it.
2. if the file is opened for reading, everything proceeds as normal.
3. if the file is opened for writing, the instance of the file that was to be opened gets removed, and a new copy of the original file placed there; the open() succeeds and the resulting file handle refers to the new copy.

This wouldn't break existing applications because the feature would not be enabled by default (you'd have to set the special xattr on a file to use it).

[cburroughs]

As was brought up in the thread we currently using http://www.xmailserver.org/flcow.html on ext4 for file/dir level COW. This works, but we would much prefer if we were using ZFS to have the filesystem take care of COW goodness. (For our narrow use case we can probably do everything we need to do with a filesystem per directory, but having our code just work with ``cp` would be nice to have.)

[torn5]

I would like to bump for this feature. When I submitted it 2 years ago there were like 30 issues before it, now there are like 500. It's moving farther and farther with issues being created before it like in the expansion of the universe. I do understand this a feature request and not a bugfix but it would make ZFS a helluva lot more appealing for people.
Snapshotting the whole ZFS filesystem to achieve the clone of 1 file is definitely overkill, cannot substitute.

[ryao]

One of the problems with implementing this is that the directory entries are implemented as name-value pairs, which at a glance, provides no obvious way of doing this. I just noticed today that value is divided into 3 sections. The top 4 bits indicate the file type, the bottom 48-bits are the actual object and the middle 12-bits are unused. One of those unused bits could be repurposed to implement reflinks.

Implementing reflinks would require much more than marking a bit, but the fact that we have spare bits available should be useful information to anyone who decides to work on this.

[torn5]

Hi Ryao, thanks for noticing this :-) If deduplication does not need such bits or a different directory structure, then also reflink should not need them. I see the reflink as a way to do copy+deduplicate a specific file without the costs associated with copy and with deduplication, but with the same final result... Is it not so?

[behlendorf]

@torn5 I believe you're correct, this could be done relatively easily by leveraging dedup. Basically, the reflink ioctl() would provide an user interface for per-file deduplication. As long as we're talking about a relatively small number of active files the entire deduplication table should be easily cachable and performance will be good. If implemented this way we'd inherit the existing dedup behavior for quotas and such. This makes the most sense for individual files in larger filesystems, for directories creating a new dataset would still be best.

[hufman]

Here is a scenario that I think this feature would be very helpful for:

I take regular snapshots of my video collection. Because of COW, these snapshots do not take any space. However, a (young relative|virus|hostile alien) comes for a visit and deletes some videos from my collection, and I would like to recover them from my handy snapshots. If I use cp normally, each recovered video is duplicated in snapshots and in the active space. With cp --reflink, the file system would be signaled to COW the file to a new name, without taking any additional space, along with making recovery instantaneous.

As an aside, is there a way to scan a ZFS volume and run an offline deduplication? If I had copied the data, is there a way to recover the space other than deleting all snapshots that contained the data?

[akorn]

On Thu, Jan 16, 2014 at 05:16:38PM -0800, hufman wrote:

I take regular snapshots of my video collection. Because of COW, these
snapshots do not take any space. However, a (young relative|virus|hostile
alien) comes for a visit and deletes some videos from my collection, and I
would like to recover them from my handy snapshots. If I use cp normally,
each recovered video is duplicated in snapshots and in the active space.
With cp --reflink, the file system would be signaled to COW the file to a
new name, without taking any additional space, along with making recovery
instantaneous.

I'm not sure I see how that would work; it would need cross-filesystem
reflink support (since you'd be copying out of a snapshot and into a real
fs).

Normally, to recover from such situations, you'd just roll back to the
latest snapshot that still has the missing data. Of course, if you'd like to
keep some of the changes made since then, this is less than ideal.

If this is a frequent occurrence, maybe you'd like to turn on deduplication.
In that case, copying the files out of the snapshot will not use extra
space.

As an aside, is there a way to scan a ZFS volume and run an offline
deduplication?

None that I know of. What you could do is: enable dedup, then copy each file
to a new name, remove the original file and rename the new file to the old
name. This obviously does nothing to deduplicate existing snapshots.

If I had copied the data, is there a way to recover the
space other than deleting all snapshots that contained the data?

Other than rolling back to a snapshot, no, I don't think so.

Andras

                            E = mc^2 + 3d6

[hufman]

Thank you for your respose!

[rocallahan]

My use-case for reflink is that we are building a record and replay debugging tool (https://github.com/mozilla/rr) and every time a debuggee process mmaps a file, we need to make a copy of that file so we can map it again later unchanged. Reflink makes that free. Filesystem snapshots won't work for us; they're heavier-weight, clumsy for this use-case, and far more difficult to implement in our tool.

[ThisGuyCodes]

I also have a use for this, I use different zfs filesystems to control differing IO requirements. Currently for my app to move items between these filesystems it must do a full copy, which works, but makes things fairly unresponsive fairly regularly, as it copies dozens of gigabytes of data. I would consider using deduplication, but I'm fairly resource constrained as it is.

[yshui]

I would like to do this as my GSoC 2014 project, but I don't know whether ZFSOnLinux participate in GSoC.