Use BTRFS reflink for safer deduplication

[DADA30000]

tldr;
BTRFS can copy a file without using additional space on disk, and it's safe to delete a file that it was copied from, basically symlinks but show up as a file, and source file can be deleted safely.

https://youtu.be/15_-hgsX2V0?t=409 I've noticed this video in my recommendations and decided to watch it, but on timecode the guy was talking about deduplication being dangerous, as external modification of actual files can break the symlink. Then I immediatly thought about reflinks in BTRFS
BTRFS supports a feature named "reflinks", it's basically like a symlink, but to a user it shows up as a completely regular file, even if you delete the source file that it was copied from, the copy of the file will still be valid, as the source file is like a "link" to the actual data of file, and only the link gets deleted, the actual data is deleted only if there are no "links" to the data.
So this can be used for safe deduplication, maybe a check for this, or an additional flag for deduplication config.

[9001]

yep, this is an excellent idea -- someone mentioned it in #197 too :>

I plan to add this 👍

[DADA30000]

yep, this is an excellent idea -- someone mentioned it in #197 too :>

I plan to add this 👍

oh wow, an instant response :D
thx for working on this :)
I will then just leave the issue there to let people see if the feature is implemented

[surfaceflinger]

ZFS has/had this quirk when reflinking between datasets, idk if that's still the case tho https://discourse.practicalzfs.com/t/sanity-check-block-cloning/794/2

[DADA30000]

ZFS has/had this quirk when reflinking between datasets, idk if that's still the case tho https://discourse.practicalzfs.com/t/sanity-check-block-cloning/794/2

yep, that's why I only mentioned BTRFS, AFAIK only it has completely working reflink implementation without any problems for few years already

[9001]

it appears python 3.14 will have a battery for reflinking; python/cpython#93152

compiling 3.14rc1 to give it a workout (primarily on btrfs) and will add it as an option for the next version if it seems fine

[motiejus]

FWIW copy_file_range (the syscall behind reflink) is also supported on xfs.

[PotcFdk]

I just wanted to add that ioctl_fideduperange would allow to even share partial duplicate ranges within files. Since copyparty already does partial checksums, the easy way to make use of this might be to notice when a range was seen before, assuming the hashes stick around after the upload has finished, which might not be the case (I did not read the source code).

[9001]

ioctl_fideduperange would allow to even share partial duplicate ranges within files

that's neat!

right now, copyparty is only aware of the individual chunks as the upload is happening - the chunk checksums are discarded when the upload is completed, and only the full combined hash is stored in the db.

if we find other reasons to store the individual checksums at some point, then this would be something to look at for sure.