nearly-full vdevs are still being written to even after adding new, empty vdev

[justinpryzby]

(FYI: we run ZFS on top of LVM, on top of VM storage. I know It's against best-practices, but that's what we do.)

Running zfs-2.3.1, with compress=zstd, recordsize=1M,

I've been loading a large 5 TB DB to a new ZFS pool. I started with a single 512 GB vdev, and once it was filled to ~80%, I added another 512 GB vdev. Repeated several times, up to (now) 8, 512 GB vdevs. I was surprised to see that the nearly-full vdevs were still being written to, even after reaching 98% capacity, and 70% fragmentation:

  data/zfs0          512G   503G  8.30G        -         -    74%  98.4%      -    ONLINE
  data/zfs1          512G   499G  9.17G        -         -    73%  98.2%      -    ONLINE
  data/zfs2          512G   499G  9.08G        -         -    71%  98.2%      -    ONLINE
  data/zfs3          512G   499G  9.19G        -         -    71%  98.2%      -    ONLINE
  data/zfs4          512G   503G  5.48G        -         -    72%  98.9%      -    ONLINE
  data/zfs5          512G   469G  38.9G        -         -    18%  92.3%      -    ONLINE
  data/zfs6          512G   311G   197G        -         -     6%  61.2%      -    ONLINE
  data/zfs7          512G   116G   392G        -         -     1%  22.8%      -    ONLINE

None of these were above ~80% full when I added an additional, new vdev.

I tried setting these, but it doesn't seem to have changed the behavior to my expectations.

/sys/module/zfs/parameters/zfs_metaslab_fragmentation_threshold = 60
/sys/module/zfs/parameters/zfs_mg_fragmentation_threshold = 60
/sys/module/zfs/parameters/zfs_mg_noalloc_threshold = 85

Any insight appreciated.

[amotin]

You might be interested in this work: #17020 . Though it is only in master branch now and so will be in upcoming 2.4.

[justinpryzby]

Looks like it's a known issue. Thanks for working on it. Looking forward to 2.4.