COW cp (--reflink) doesn't work across different datasets: Invalid cross-device link

[darkbasic]

System information

Type	Version/Name
Distribution Name	Arch Linux
Distribution Version
Kernel Version	6.6.0-rc4
Architecture	amd64
OpenZFS Version	git branch zfs-2.2-release + 6.6 compatibility patches

Describe the problem you're observing

Reflinking doesn't work across different datasets.
Since https://lore.kernel.org/linux-btrfs/cover.1645194730.git.josef@toxicpanda.com/T/#mf251325026fe2e15ed5119856bf654ba4f0d298b btrfs allows to reflink across different subvolumes, so it should be possible to achieve something similar in Linux with zfs.
Not being able to reflink across different datasets vastly reduce the utility of reflinking.

Describe how to reproduce the problem

cp -a --reflink=always /path/to/first/dataset/file /path/to/second/dataset/

Include any warning/errors/backtraces from the system logs

[niko@arch-phoenix ~]$ cp --reflink=always ~/.cache/yay/chromium-wayland-vaapi/chromium-117.0.5938.132.tar.xz .
cp: failed to clone './chromium-117.0.5938.132.tar.xz' from '/home/niko/.cache/yay/chromium-wayland-vaapi/chromium-117.0.5938.132.tar.xz': Invalid cross-device link

P.S.
I have been told that --reflink=auto should be able to clone blocks across different datasets, but this isn't the case:

[niko@arch-phoenix ~]$ time cp ~/.cache/yay/chromium-wayland-vaapi/chromium-117.0.5938.132.tar.xz . && rm chromium-117.0.5938.132.tar.xz

real	0m3.136s
user	0m0.000s
sys	0m2.852s

[niko@arch-phoenix .cache]$ time cp yay/chromium-wayland-vaapi/chromium-117.0.5938.132.tar.xz . && rm chromium-117.0.5938.132.tar.xz

real	0m0.127s
user	0m0.000s
sys	0m0.127s

It took 3 seconds compared to 0.1 seconds when the dataset was the same, suggesting that reflinking didn't work across datasets.

[rincebrain]

See here.

[darkbasic]

@rincebrain interesting, but somehow it doesn't work according to the time results.

[robn]

--reflink=always calls ioctl(FICLONE). Linux inspects this call before passing it to the filesystem, and will reject it if source and destination files are not on a filesystem with the same superblock (and before 5.18, the same mountpoint).

This is not an OpenZFS bug as such, because if Linux would pass the call down, we would quite happily service it. Working around Linux's check here is extremely difficult, if its even possible.

(the Btrfs example is not really relevant; OpenZFS and Btrfs have a fundamentally different construction and purpose).

--reflink=auto calls the copy_file_range() syscall, which in this case means "make a new file with the same contents as this existing one and I don't care how you do it". Often OpenZFS can service this with a clone, but not always (for many good reasons). If it can't, it'll fall back to a regular content copy.

The call time is not a very good indicator that a clone vs a copy was done. To tell if it was cloned you currently have to dig around with zdb. But in any case, that's acceptable behaviour, because --reflink=auto (copy_file_range() allows it).

Yes, this sucks. We'll keep working on it but its complicated.

[rincebrain]

btrfs doesn't trip this because they show up subvolumes as the same "mountpoint" which is why this check doesn't bite them.

[darkbasic]

The problem is not only with different datasets, but with snapshots as well:

[niko@arch-phoenix ~]$ cp --reflink=always /home/.zfs/snapshot/zrepl_20231004_074556_000/niko/devel/linux-mainline.tar.gz .
cp: failed to clone './linux-mainline.tar.gz' from '/home/.zfs/snapshot/zrepl_20231004_074556_000/niko/devel/linux-mainline.tar.gz': Invalid cross-device link
[niko@arch-phoenix ~]$ mount | grep home
rpool/home on /home type zfs (rw,nodev,relatime,xattr,posixacl,casesensitive)
rpool/home@zrepl_20231004_074556_000 on /home/.zfs/snapshot/zrepl_20231004_074556_000 type zfs (ro,relatime,xattr,posixacl,casesensitive)

Accessing snapshots create a different mountpoint which triggers the same issue.

[robn]

Filesystems and snapshots are different datasets. Or, put another way, they're different mounts, and so different superblocks, and so Linux rejects the request.

I understand that this is frustrating, but no amount of pointing it out is going to make a quick fix happen. I'm aware of four possible solutions (or shapes of solutions):

• Linux lifting the restriction
• Adding OpenZFS-specific calls that Linux doesn't know about (and so won't intercept), and then adding support for those to common tools (like cp)
• Adding zfs clonefile or similar command to do clones directly inside OpenZFS
• Significantly modify OpenZFS to use the same superblock for all mounts

I've been quietly exploring all of these options for a few weeks now. They are all difficult and/or complicated, for different reasons, and I also have very little time available to look at it. If you've got some other idea, I'm happy to hear it.

[darkbasic]

My only suggestion is to reach Kent Overstreet @koverstreet and ask him what are his plans for bcachefs regarding this.
Lifting the Linux restriction would obviously be the best course of action and working with a (soon-to-be) mainline filesystem would be much easier.

[darkbasic]

The reason why --reflink=auto (which in turn calls the copy_file_range() syscall) might not create clones is encryption. We can't clone encrypted blocks across datasets because the key material is partially bound to the source dataset (actually its encryption root). #14705 has a start on this.

[robn]

If you're going to take something I wrote and pass it off as your own you should at least adjust it to match the context. That's why it didn't work in your case. It can and does create clones in many other situations.

In any case, there's no bug here. These limitations are well understood and will be worked on as time and interest allows.

[darkbasic]

If you're going to take something I wrote and pass it off as your own you should at least adjust it to match the context.

That was not my intention and I'm sorry if it felt that way. I'm just trying my best to juggle the relevant information across the two threads so that anybody who stumbles upon either of them will understand what's going on.

That's why it didn't work in your case

I thought that was clear enough, but I've edited my past message to change "does" with "might".

[oromenahar]

Linux lifting the restriction

I think getting a patch into the linux kernel to lift the cross device link restriction can be pretty hard. There is no usecase in the kernel and no kernel (fs)driver wich need this. I don't think that the linux kernel will accept for example a new ioctl-flag FICLONEC for cross device links without any usecase inside the kernel and I think the existing flags won't be changed as well, because cross reflinking over different datasets doesn't make sense. Don't know for sure but I quess so.

[darkbasic]

Unfortunately even bcachefs won't be a suitable candide because it already supports reflinking across different subvolumes.

[jittygitty]

@darkbasic Perhaps, but regardless, Kent's very smart and knowledgeable and I think nice enough to give some objective and thoughtful opinions that may be helpful.