ZFS ARC significantly exceeds `zfs_arc_max` during `cp --reflink` from snapshot to specific dataset in ZFS 2.3.2-1

[Schuwi]

I have found similar issues describing an unbounded growth of the ARC in ZFS 2.3.x but none specific to the reflink copy and my personal knowledge of ZFS' implementation is little to none, so I could not judge whether they are caused by the same underlying issue.

Transparency note: This issue was interactively composed with Gemini 2.5 Pro. It instructed me in running the preliminary debugging and collecting the following information, and compiled it into this summary form.

System information

Type	Version/Name
Distribution Name	Debian
Distribution Version	13
Kernel Version	6.1.0-31-amd64
Architecture	amd64
OpenZFS Version	zfs-2.3.2-1 (kmod: zfs-kmod-2.3.2-1)

Describe the problem you're observing

The ZFS ARC grows significantly beyond the configured zfs_arc_max limit (32 GiB) when performing a cp -r --reflink=auto operation. This issue has been specifically observed when copying from a directory within a ZFS snapshot (.zfs/snapshot/photos-pre-move/Photos/2019/* of dataset dpool/home/schuwi) to a particular target dataset (dpool/home/schuwi/Photos/2019). Both the source snapshot content and the target dataset reside on the same ZFS pool (dpool) which has the block_cloning feature enabled.

During the problematic reflink copy, the ARC size (reported by /proc/spl/kstat/zfs/arcstats) rapidly grows to over 50 GiB (from a 64 GiB total system RAM), far exceeding the 32 GiB zfs_arc_max limit. This leads to excessive memory consumption, severe system slowdown, and imminent risk of OOM killer intervention. If the operation is not terminated manually the OOM killer terminates most running applications until it catches the terminal running the cp operation, thus ending the copy operation and stopping memory usage from increasing.
The data_size component of the ARC alone is observed to exceed zfs_arc_max. High values for evict_skip and evict_not_enough are also present in arcstats during these operations, indicating the ARC is struggling to evict buffers.

Crucially, this issue exhibits specificity:

• The issue does not occur when performing the same copy operation with cp -r --reflink=never. In this case, the ARC size remains stable around the configured zfs_arc_max.
• The issue does not occur when using cp -r --reflink=auto to copy from the same snapshot source structure (e.g., .zfs/snapshot/photos-pre-move/Photos/2018/*) to a different target dataset (e.g., dpool/home/schuwi/Photos/2018). In this scenario, the ARC also respects the zfs_arc_max limit. Though the transfer speed is very slow, indicating that a normal transfer is happening, without taking advantage of the block_cloning feature.

This suggests an interaction specific to the source data from the photos-pre-move/Photos/2019/ directory within the snapshot and/or the state of the dpool/home/schuwi/Photos/2019 target dataset.

ZFS DKMS modules (zfs-2.3.2-1, zfs-kmod-2.3.2-1) are successfully built and installed for the current kernel.

Describe how to reproduce the problem

1. Configure zfs_arc_max to 32 GiB (34359738368 bytes) in /etc/modprobe.d/zfs.conf. Reboot or ensure the setting is active.

   options zfs zfs_arc_max=34359738368
   
   

2. Use a ZFS pool (e.g., dpool) with the block_cloning feature enabled.

   Bash

   zpool get feature@block_cloning dpool 
   # Expected: dpool  feature@block_cloning   active   local
   
   

3. Have a source dataset, take a snapshot (e.g., dpool/home/schuwi@photos-pre-move). The source data for the copy will be a directory within this snapshot, e.g., /path/to/mountpoint/.zfs/snapshot/photos-pre-move/Photos/2019/*.

4. Have a target dataset where the issue is observed, e.g., dpool/home/schuwi/Photos/2019.

5. Attempt to copy the data from the specific snapshot directory to the specific target dataset using cp with reflink enabled. Assuming the command is run from /home/schuwi/ (which is on dpool/home/schuwi):

   Bash

   cp -r --reflink=auto .zfs/snapshot/photos-pre-move/Photos/2019/* Photos/2019/
   
   

6. Monitor ARC usage via /proc/spl/kstat/zfs/arcstats and system memory (free -m, vmstat).

7. Observe the ARC size and data_size rapidly grow beyond c_max.

Contrastive Test (Does NOT show the issue):

• Copying from .zfs/snapshot/photos-pre-move/Photos/2018/* to a different target dataset, e.g., Photos/2018 (on dpool/home/schuwi/Photos/2018), using cp -r --reflink=auto does NOT exhibit this ARC overgrowth.

Include any warning/errors/backtraces from the system logs

No specific ZFS errors or backtraces appear in dmesg or journalctl during the ARC overgrowth, other than general system slowdown due to memory exhaustion.

System and ZFS Information:

$ lsb_release -is
Debian
$ lsb_release -rs
13
$ zfs version
zfs-2.3.2-1
zfs-kmod-2.3.2-1
$ uname -r
6.1.0-31-amd64

Pool and Dataset Information (Before problematic copy):

$ zpool status dpool
  pool: dpool
 state: ONLINE
  scan: scrub repaired 0B in 05:11:37 with 0 errors on Sun Jun  1 05:11:38 2025
config:
    NAME                                                      STATE     READ WRITE CKSUM
    dpool                                                     ONLINE       0     0     0
      ata-WDC_WD20EZRZ-22Z5HB0_WD-WCC4M0USZJEK                ONLINE       0     0     0
      ata-WDC_WD20EZRZ-22Z5HB0_WD-WCC4M0YVV5JK                ONLINE       0     0     0
    special   
      nvme-Samsung_SSD_980_PRO_1TB_S5GXNF0NC28024N-part5       ONLINE       0     0     0
errors: No known data errors

$ zpool get all dpool | grep block_cloning
dpool  feature@block_cloning         active                          local

$ zfs list -o name,used,avail,refer,mountpoint,primarycache,secondarycache dpool/home/schuwi@photos-pre-move dpool/home/schuwi/Photos/2019
NAME                               USED  AVAIL  REFER  MOUNTPOINT                 PRIMARYCACHE  SECONDARYCACHE
dpool/home/schuwi@photos-pre-move  29.1M      -  2.63T  -                          all           all
dpool/home/schuwi/Photos/2019       377G   287G   108G  /home/schuwi/Photos/2019  all           all

Dataset Information (After problematic copy was stopped manually to avoid process termination):

$ zfs list -o name,used,avail,refer,mountpoint,primarycache,secondarycache dpool/home/schuwi@photos-pre-move dpool/home/schuwi/Photos/2019
NAME                               USED  AVAIL  REFER  MOUNTPOINT                 PRIMARYCACHE  SECONDARYCACHE
dpool/home/schuwi@photos-pre-move  29.1M      -  2.63T  -                          all           all
dpool/home/schuwi/Photos/2019       432G   287G   162G  /home/schuwi/Photos/2019  all           all 

/etc/modprobe.d/zfs.conf:

options zfs zfs_arc_max=34359738368

Memory Usage During Issue:

$ free -m && vmstat 1 5
              total        used        free      shared  buff/cache   available
Mem:          64200       57016        7265          16         648        7183
Swap:             0           0           0
procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu
 1  0      0 7425448      0 664016    0    0 31916 14915 2098    2  0  1 97  2  0  0
 1  0      0 4673012      0 672072    0    0 263368     0 11052 116137  0  7 93  0  0  0
 1  0      0 4276772      0 674000    0    0 237644 14284 8435 34033  0  8 91  1  0  0
 1  0      0 4549092      0 673696    0    0 239680 470628 40485 194893  0 16 83  1  0  0
 0  0      0 4826904      0 673696    0    0 266776     0 3424 5038  0  1 99  0  0  0

arcstats Data:

• arcstats_before.txt: Content of /proc/spl/kstat/zfs/arcstats before the copy
• arcstats_during_reflink_issue.txt: Content of /proc/spl/kstat/zfs/arcstats when ARC size > c_max
• arcstats_after.txt: Content of /proc/spl/kstat/zfs/arcstats after stopping the copy and waiting 1~2 minutes

[amotin]

I think I might have an idea what's going on, but please answer one question: what recordsize do you have on that dataset and was it the same in 2018 and 2019 when those photos were written there, or it changed over time? Unless you remember, you might be able to find it in pool history, unless the pool has quite an active life.

[Schuwi]

Thank you for taking the time to look into this!

I thought I remembered fiddling with the recordsize at some point but checking the zpool history I found no trace of anything like that (at least for this pool).
The only thing I noticed is that I changed the special_small_blocks property of the root dataset shortly after creating the pool.

$ sudo zpool history dpool
2021-01-21.20:04:25 zpool create -o ashift=12 -O acltype=posixacl -O canmount=off -O compression=lz4 -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa -O mountpoint=/home -O special_small_blocks
=32K -R /mnt dpool /dev/disk/by-id/ata-WDC_WD20EZRZ-22Z5HB0_WD-WCC4M0USZJEK /dev/disk/by-id/ata-WDC_WD20EZRZ-22Z5HB0_WD-WCC4M0YVV5JK special /dev/disk/by-id/nvme-Samsung_SSD_980_PRO_1TB_S5GXNF0NC28024N-part5
[...]
2021-01-22.12:18:51 zfs set special_small_blocks=64K dpool
[...]
$ zfs list -o name,recordsize,special_small_blocks,dnodesize,compression dpool/home/schuwi dpool/home/schuwi/Photos/2019 dpool/home/schuwi/Photos/2018
NAME                           RECSIZE  SPECIAL_SMALL_BLOCKS  DNSIZE  COMPRESS
dpool/home/schuwi                 128K                   64K  auto    lz4
dpool/home/schuwi/Photos/2018     128K                   64K  auto    lz4
dpool/home/schuwi/Photos/2019     128K                   64K  auto    lz4

To give you an idea of the files that make up the data in this case:

$ ls -l .zfs/snapshot/photos-pre-move/Photos/2018/2018-01-21/ .zfs/snapshot/photos-pre-move/Photos/2019/2019-01-01/
.zfs/snapshot/photos-pre-move/Photos/2018/2018-01-21/:
total 816390
-rwxr-xr-x 1 schuwi schuwi 16308689 Jan 28  2019 P1210033.ORF
-rw-r--r-- 1 schuwi schuwi      985 May  2  2021 P1210033.ORF.xmp
-rwxr-xr-x 1 schuwi schuwi 16158621 Jan 28  2019 P1210034.ORF
-rw-r--r-- 1 schuwi schuwi      985 May  2  2021 P1210034.ORF.xmp
-rwxr-xr-x 1 schuwi schuwi 16054394 Jan 28  2019 P1210035.ORF
-rw-r--r-- 1 schuwi schuwi      985 May  2  2021 P1210035.ORF.xmp
-rwxr-xr-x 1 schuwi schuwi 15975378 Jan 28  2019 P1210036.ORF
-rw-r--r-- 1 schuwi schuwi      985 May  2  2021 P1210036.ORF.xmp
-rwxr-xr-x 1 schuwi schuwi 15736918 Jan 28  2019 P1210037.ORF
-rw-r--r-- 1 schuwi schuwi      985 May  2  2021 P1210037.ORF.xmp
-rwxr-xr-x 1 schuwi schuwi 15697447 Jan 28  2019 P1210038.ORF
-rw-r--r-- 1 schuwi schuwi      985 May  2  2021 P1210038.ORF.xmp
-rwxr-xr-x 1 schuwi schuwi 14397807 Jan 28  2019 P1210039.ORF
-rw-r--r-- 1 schuwi schuwi      985 May  2  2021 P1210039.ORF.xmp
[...]

.zfs/snapshot/photos-pre-move/Photos/2019/2019-01-01/:
total 2377586
-rwxr-xr-x 1 schuwi schuwi 28161412 Jan  1  2019 DSC_9686.NEF
-rw-r--r-- 1 schuwi schuwi      985 Mar 29  2021 DSC_9686.NEF.xmp
-rwxr-xr-x 1 schuwi schuwi 28179612 Jan  1  2019 DSC_9687.NEF
-rw-r--r-- 1 schuwi schuwi      985 Mar 29  2021 DSC_9687.NEF.xmp
-rwxr-xr-x 1 schuwi schuwi 31510588 Jan  1  2019 DSC_9688.NEF
-rw-r--r-- 1 schuwi schuwi      985 Mar 29  2021 DSC_9688.NEF.xmp
-rwxr-xr-x 1 schuwi schuwi 31354434 Jan  1  2019 DSC_9689.NEF
-rw-r--r-- 1 schuwi schuwi      985 Mar 29  2021 DSC_9689.NEF.xmp
-rwxr-xr-x 1 schuwi schuwi 31332365 Jan  1  2019 DSC_9690.NEF
-rw-r--r-- 1 schuwi schuwi      985 Mar 29  2021 DSC_9690.NEF.xmp
[...]

[amotin]

I don't know if it will be enough for your case, but this should be a step in the right direction: #17431 .

[Schuwi]

Thank you for taking action @amotin.
Though I am not sure how to build and install ZFS from source and whether it is even advisable testing a bleeding edge build on my live system.
I have backups of my data but I would prefer not having to go through the hassle of rebuilding my system 😅

I do have more observations to add however:
Even though the reflink copy is significantly slower on the 2018 data(set) it actually does seem to do block cloning (contrary to my earlier belief). I infer this from the amount of free space on my pool fluctuating during the copy operation but always returning to more or less the same amount as before the copy operation.
I can also observe the very curious behaviour of the amount of copied data (watch du -sh Photos/2018) jumping in quite large increments of several gigabyte at a time with no apparent change in between jumps, which is not what I would expect from a copy operation limited by a HDD write speed of some hundreds of MB per sec.

I think I have made out the significant difference why the system runs out of memory on one dataset but not the other.
It seems to simply be because of the speed the copy operation progresses at. Whatever amount of excess ARC memory is reserved slowly decrements (i.e. I could actually now work around the initial issue and still do a reflink copy by pausing the copy operation [Ctrl+Z] when the system memory is almost at it's limit, and resuming once it calmed down [fg]).
When the data is processed slowly enough that the memory consumption has time to calm down the issue does not occur.

The reason why one of these progresses significantly faster than the other one (or at least fills the ARC above it's limit faster than it can reduce it again) could have to do with one dataset consisting of at least partially compressible files (the slow dataset, 2018, consists mostly out of .ORF files which are at least a little bit compressed):

$ du -bB1K Photos/2018/2018-06-22/P6220727.ORF
14626	Photos/2018/2018-06-22/P6220727.ORF
$ du -B1K Photos/2018/2018-06-22/P6220727.ORF
14461	Photos/2018/2018-06-22/P6220727.ORF

while the .NEF file in the "fast" dataset, 2019, seem to be mostly deemed incompressible by ZFS:

$ du -bB1K Photos/2019/real/2019-01-01/DSC_9699.NEF
30626	Photos/2019/real/2019-01-01/DSC_9699.NEF
$ du -B1K Photos/2019/real/2019-01-01/DSC_9699.NEF
30637	Photos/2019/real/2019-01-01/DSC_9699.NEF

At some point in the 2018 dataset (chronologically, which incidentally seems to coincide with the copy order) the data switched from .ORF files to .NEF files and I once again got problems with my system memory exploding.

[exi]

@amotin

I don't know if it will be enough for your case, but this should be a step in the right direction: #17431 .

I am running OpenZFS 2.3.3 on arch, so your PR should be included already.

I'm still running into massive OOM issues with large cross-dataset cp --reflink=auto copies to the point where I need to do copies in chunks with intermittend zpool sync calls to flush things, otherwise my machine just OOMs and dies.

The zpool sync calls themselves take an extreme amount of time as well. Yesterday I reflink-copied data for 20 minutes and then spent over 4 hours waiting for zpool sync.

My pool:

# zfs list raid
NAME   USED  AVAIL  REFER  MOUNTPOINT
raid  14.4T   350G  2.80T  /mnt/raid

# zpool status -v raid
  pool: raid
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 09:54:25 with 0 errors on Sat Jun 21 05:24:25 2025
config:

        NAME                                    STATE     READ WRITE CKSUM
        raid                                    ONLINE       0     0     0
          raidz1-0                              ONLINE       0     0     0
            wwn-0x50014ee26743478b-part1        ONLINE       0     0     0
            wwn-0x50014ee2bc9919eb-part1        ONLINE       0     0     0
            wwn-0x50014ee2bc9921ef-part1        ONLINE       0     0     0
            wwn-0x50014ee267434631-part1        ONLINE       0     0     0
        cache
          5e674241-a0f0-7049-afd3-2b35b82b4180  ONLINE       0     0     0

errors: No known data errors

Output of zfs list -o all raid -j | jq:
zpool-list.txt

I tried copying everything in one go tonight but that would just OOM my machine.
This morning i started the copy in chunks and it worked:

[amotin]

Huh. Thanks for the update. I've suspected that change, even while being right, might be not enough.