ZFS Performance on mirror HDD

[hdiethelm]

Hello

I have a pretty basic ZFS Mirror on Segate Exos X18 drives and I am wondering why I am somewhat away from the theoretical read and write performance. Let's put it that way, it's not bad but it could be better. During the storage expansion, I did some benchmarks. Especially the slower read speed after the change from raid0 to twin two way mirrors is strange.

I read trough a lot of articles but I did not find anything conclusive. According to https://static.ixsystems.co/uploads/2020/09/ZFS_Storage_Pool_Layout_White_Paper_2020_WEB.pdf I should get:
Streaming read speed: N * Streaming read speed of single drive
Streaming write speed: Streaming write speed of single drive
But that's not the case.

Note: I am using numjobs =1 because I am normally just me and myself reading/writing big data files.

Settings:

sudo zpool create \
    -o ashift=12 \
    -O acltype=posixacl -O xattr=sa -O dnodesize=auto \
    -O compression=lz4 \
    -O normalization=formD \
    -O relatime=on \
    -O readonly=on \
    -m /mnt/tank1-new tank1-new /dev/disk1-part1 /dev/disk2-part1

The Disks have ~140...280MB/s, depending on the location:

dd if=/dev/sda1 of=/dev/zero bs=1M count=1024
1073741824 Byte (1.1 GB, 1.0 GiB) kopiert, 3.88505 s, 276 MB/s
dd if=/dev/sda1 of=/dev/zero bs=1M count=1024 skip=8000000
1073741824 Byte (1.1 GB, 1.0 GiB) kopiert, 4.69628 s, 229 MB/s
dd if=/dev/sda1 of=/dev/zero bs=1M count=1024 skip=16000000
1073741824 Byte (1.1 GB, 1.0 GiB) kopiert, 7.34748 s, 146 MB/s

Read:

fio --name=fio-file --ioengine=posixaio --rw=read --bs=1m --size=32g --numjobs=1 --iodepth=1 --runtime=30 --time_based --end_fsync=1
sudo sysctl -w vm.drop_caches=3
fio --name=fio-file --ioengine=posixaio --rw=read --bs=1m --size=32g --numjobs=1 --iodepth=1 --runtime=30 --time_based --end_fsync=1

Write:

fio --name=fio-file --ioengine=posixaio --rw=write --bs=1m --size=32g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1

Config1

	NAME                                                  STATE     READ WRITE CKSUM
	tank1                                                 ONLINE       0     0     0
	  mirror-0                                            ONLINE       0     0     0
	    disk1-part1                      ONLINE       0     0     0
	    disk2-part1                      ONLINE       0     0     0
	cache
	  nvme  ONLINE       0     0     0

./read.sh 
   READ: bw=276MiB/s (289MB/s), 276MiB/s-276MiB/s (289MB/s-289MB/s), io=16.2GiB (17.3GB), run=60004-60004msec
./write.sh
  WRITE: bw=191MiB/s (200MB/s), 191MiB/s-191MiB/s (200MB/s-200MB/s), io=18.0GiB (19.4GB), run=96694-96694msec

I would expect read ~500MB/s / write ~250MB/s

Config 2 (I was brave to break the mirror to expand the size)

pool: tank1-new
state: ONLINE
config:

   NAME                            STATE     READ WRITE CKSUM
   tank1-new                       ONLINE       0     0     0
     disk1-part1  ONLINE       0     0     0
     disk2-part1  ONLINE       0     0     0

./read.sh 
   READ: bw=521MiB/s (546MB/s), 521MiB/s-521MiB/s (546MB/s-546MB/s), io=30.5GiB (32.8GB), run=60003-60003msec
./write.sh 
  WRITE: bw=448MiB/s (469MB/s), 448MiB/s-448MiB/s (469MB/s-469MB/s), io=33.2GiB (35.7GB), run=75963-75963msec

I would expect read ~500MB/s / write ~500MB/s (Pretty much there but not exactly)

Config 3

 pool: tank1
 state: ONLINE
config:

	NAME                                                  STATE     READ WRITE CKSUM
	tank1                                                 ONLINE       0     0     0
	  mirror-0                                            ONLINE       0     0     0
	    disk1-part1                      ONLINE       0     0     0
	    disk2-part1                      ONLINE       0     0     0
	  mirror-1                                            ONLINE       0     0     0
	    disk3-part1                      ONLINE       0     0     0
	    disk4-part1                      ONLINE       0     0     0
	cache
	  nvme  ONLINE       0     0     0

./read.sh
  READ: bw=509MiB/s (533MB/s), 509MiB/s-509MiB/s (533MB/s-533MB/s), io=29.8GiB (32.0GB), run=60004-60004msec
./write.sh 
  WRITE: bw=339MiB/s (355MB/s), 339MiB/s-339MiB/s (355MB/s-355MB/s), io=26.7GiB (28.7GB), run=80806-80806msec

I would expect read ~1000MB/s / write ~500MB/s (But read got slower by changing to a two way mirror instead of faster?! Reads are from all four disk, just halve the speed than without mirrors)

Config 3 with recordsize=1M

./read.sh
   READ: bw=570MiB/s (598MB/s), 570MiB/s-570MiB/s (598MB/s-598MB/s), io=16.7GiB (17.9GB), run=30012-30012msec
./write.sh 
  WRITE: bw=360MiB/s (377MB/s), 360MiB/s-360MiB/s (377MB/s-377MB/s), io=27.3GiB (29.4GB), run=77819-77819msec

Better but still not great.

[amotin]

Single-stream reads from a mirror of HDDs is tricky. Since both disks contain exactly the same data, if you do round-robin in not bit enough chunks, then while one disk read the data, another will just fly the head over. Increased chunk size together with deeper speculative prefetch might help, but also only to a degree. I was thinking to look on it again at some point, but not yet. With SSDs the situation is predictably better.

[hdiethelm]

Thanks, that makes sens. So basically all this theory about the read speed are just right if you take zero seek time. As soon as you take the seek time into account, it gets way slower because you have to skip halve the data on each drive right?

I just figured out setting
primarycache=metadata
makes
sudo sysctl -w vm.drop_caches=3
unneccesary and seams to have no impact on speed.

Just did some more tests. It seams the location of the file on the disk is somewhat random, sometimes it is faster or slower when I delete the test file and recreate it to change recordsize.

Sumary:

• More job's / iodepth don't help for 1M recordsize
• In theory, you could use one halve of the mirror to read one file and the other halve to read the second, but that anyway breaks down as soon as you have several jobs.
• ^ Seams to happen as soon as you set recordsize=16M, interesting...
• recordsize=1M is better
• recordsize=16M is worse for one job but better for two

recordsize=1M / iodepth=1 / numjobs=1
READ: bw=650MiB/s (682MB/s), 650MiB/s-650MiB/s (682MB/s-682MB/s), io=19.0GiB (20.5GB), run=30004-30004msec

recordsize=1M / iodepth=2 / numjobs=1
READ: bw=650MiB/s (681MB/s), 650MiB/s-650MiB/s (681MB/s-681MB/s), io=19.0GiB (20.5GB), run=30025-30025msec

recordsize=1M / iodepth=4 / numjobs=1
READ: bw=647MiB/s (679MB/s), 647MiB/s-647MiB/s (679MB/s-679MB/s), io=19.0GiB (20.4GB), run=30082-30082msec

recordsize=1M / iodepth=32 / numjobs=1
READ: bw=644MiB/s (676MB/s), 644MiB/s-644MiB/s (676MB/s-676MB/s), io=19.0GiB (20.4GB), run=30194-30194msec

recordsize=1M / iodepth=1 / numjobs=2
READ: bw=589MiB/s (618MB/s), 294MiB/s-295MiB/s (309MB/s-309MB/s), io=17.3GiB (18.5GB), run=30001-30016msec

recordsize=1M / iodepth=2 / numjobs=2
READ: bw=577MiB/s (605MB/s), 284MiB/s-295MiB/s (297MB/s-310MB/s), io=17.0GiB (18.3GB), run=30023-30185msec

recordsize=1M / iodepth=4 / numjobs=2
READ: bw=586MiB/s (614MB/s), 289MiB/s-298MiB/s (303MB/s-313MB/s), io=17.3GiB (18.6GB), run=30134-30252msec

recordsize=1M / iodepth=32 / numjobs=2
READ: bw=585MiB/s (614MB/s), 284MiB/s-302MiB/s (298MB/s-317MB/s), io=17.5GiB (18.8GB), run=30525-30612msec

recordsize=16M / iodepth=1 / numjobs=1 / bs=16m
READ: bw=408MiB/s (428MB/s), 408MiB/s-408MiB/s (428MB/s-428MB/s), io=12.0GiB (12.9GB), run=30027-30027msec
Slower?! Now I just see reads on two of the four disks.

recordsize=16M / iodepth=1 / numjobs=2 / bs=16m
READ: bw=738MiB/s (773MB/s), 357MiB/s-381MiB/s (374MB/s-399MB/s), io=21.7GiB (23.3GB), run=30035-30069msec
Best yet!

[amotin]

As soon as you take the seek time into account, it gets way slower because you have to skip halve the data on each drive right?

Right. Theoretically if speculative prefetcher is able to prefetch many megabytes ahead, then mirror code could potentially make different disks to read different tracks at the same time instead of flying over, but this is not exactly trivial and could likely be improved in the current code.

[hdiethelm]

Thanks a lot for the fast answers! So I'm looking forward for new ZFS versions. Anyway, it's fast enough, I was just wondering if more is possible.

I guess a 4 way stripe (with good backups) or a 4 disk raidz1 would be a bit faster for streaming, but not worth the effort to rebuild everything again. It takes ages with this huge disks and it gets complicated / risky when you want to do it in place.

Otherwise ZFS is working great! I got now 3 PC's with Debian + Root on ZFS. Full system backups are now a matter of minutes. Once I had to restore my workstation due to a broken main SSH, which worked perfectly. Or just roll back the whole machine a few hours because I wanted to test the new Nvidia driver and broke everything... ;-)