metadata caching does not work as expected - repeatedly getting lost in arc and l2arc with primary|secondarycache=metadata

[devZer0]

Distribution Name | debian
Distribution Version | 10.9 (buster)
Linux Kernel | 5.4.106-1
Architecture | x86_64
ZFS Version | 2.0.4-pve1
SPL Version | 2.0.4-pve1

Describe the problem you're observing

i have metadata intensive workload, so i set primarycache=metadata and secondarycache=metadata because otherwise, data would evict metadata too soon.

when doing "find /dir/tree/with/1mio/dirs/files" or "rsync -av --dry-run /dir/tree/with/1mio/dirs/files /tmp" i see arc and l2arc filling, but after a while, arc is collapsing and file metadata is not even served from l2arc, or at least only being served particularly from there.

even doing find / rsync a multiple times does not reliable fix this, i see lot's of reads being served from disk instead from arc or l2arc on ssd, so this renders using arc/l2arc for metadata caching pretty useless for me.

in arcstat, i see typical arc miss% at 5 and dm% at 4-5 - and suddenly pm% goes up from 0 to 100 and is see read i/o hitting the clasic harddisks instead being served from ram/ssd . that's very frustrating.

system has 4gb of ram, btw.

i'm trying to optimize two different system (rsync backup machine and proxmox backup server) for metadata centric type of workload, because otherwise backup job runtime gets too long. i'm aware of special vdev feature for relocating metadata, but i do not see the point why i should invest into enterprise quality, mirrored ssd when darn cheap consumer ssd should be able to act as a metadata read caching device, which is not not causing any data loss when it breaks.

[gamanakis]

I think you have a misunderstanding on how L2ARC works. By default it scans a few hundred megabytes at the end of ARC and caches whatever buffers are eligible. Meaning if your ARC content is changing fast, then L2ARC caches only part of the ARC.

Moreover, by default prefetches are not cached in L2ARC. You can turn this on by l2arc_noprefetch=0 in zfs module parameters.

Have a look at l2arc_write_max, l2arc_write_boost, l2arc_headroom, l2arc_noprefetch parameters in man 5 zfs-module-parameters.

[devZer0]

yes, i know that only part of evicted pages go into l2arc.

i did some tuning so that l2arc is getting filled more agressively, but even after re-running "find /path" in a loop for hours, i'm not able to getting most of the metadata reads getting served from l2arc when i put some little memory pressure on the arc.

there is always at least 10-20% of metadata getting served from disk/hdd and not from l2arc/ssd.

i think l2arc is really behaving totally strange and for me it's far away from expectation. at least with param "primary|secondarycache=metadata" it's completely nonsense if it's only caching parts of the metadata. as we know every metadata read from hdd adds noticeable latency, even if it's <20%. it's still most of the time spent.

so, when main role of l2arc is "to boost the performance of random read workloads" i really wonder why i need special vdev / metadata allocation classes to get file/dir metadata getting served from ssd realiably.

there are so many metadata centric workloads , especially when using backup....

[shodanshok]

@devZer0 have you tried setting l2arc_noprefetch=0 ?

[devZer0]

yes. does not help

[malventano]

The larger issue here is that folks should not have to limit their arc/l2arc to metadata-only just as a workaround, as it trades one performance problem for another. One example is that with metadata-only caching, a process sequentially reading blocks of a smaller size than recordsize will effectively amplify those reads. 1MB of 16KB sequential reads on a dataset with recordsize=1M results in 64x1MB reads, resulting in that read taking 64x as long as it should. Perhaps that's a separate optimization that should be made elsewhere (you should still at least buffer the last record read even with metadata-only caching to handle this case), but it does show how workarounds like this should not be needed in the first place and can have unintended consequences.

Users desiring more persistent metadata should be able to specify an arc_meta_min (on systems with sufficient DRAM), and that should be enough to keep that metadata in the cache. FreeNAS pre-configures arc_meta_min as several GB (vs. openzfs default of 16MB) to try and (unsuccessfully with high data throughputs) address metadata eviction. Even without arc_meta_min or other parameter tweaks, the default behavior should have metadata take sufficient priority over data that metadata is very hard to evict. I do believe that was the intent, but given the reports spanning across multiple distros and versions, it appears this is an actual bug that needs squashing, and it has existed since at least as far back as 0.8.4.

[shodanshok]

The larger issue here is that folks should not have to limit their arc/l2arc to metadata-only just as a workaround, as it trades one performance problem for another. One example is that with metadata-only caching, a process sequentially reading blocks of a smaller size than recordsize will effectively amplify those reads. 1MB of 16KB sequential reads on a dataset with recordsize=1M results in 64x1MB reads, resulting in that read taking 64x as long as it should. Perhaps that's a separate optimization that should be made elsewhere (you should still at least buffer the last record read even with metadata-only caching to handle this case), but it does show how workarounds like this should not be needed in the first place and can have unintended consequences.

Yeah, for this reason, when I have both a metadata-rich and sequential read heavy workloads, I generally set primarycache=all (to avoid read amplification) and secondarycache=metadata (to store as much metadata as possible).

Users desiring more persistent metadata should be able to specify an arc_meta_min (on systems with sufficient DRAM), and that should be enough to keep that metadata in the cache. FreeNAS pre-configures arc_meta_min as several GB (vs. openzfs default of 16MB) to try and (unsuccessfully with high data throughputs) address metadata eviction. Even without arc_meta_min or other parameter tweaks, the default behavior should have metadata take sufficient priority over data that metadata is very hard to evict. I do believe that was the intent, but given the reports spanning across multiple distros and versions, it appears this is an actual bug that needs squashing, and it has existed since at least as far back as 0.8.4.

True, for this reason I am (for what it is worth) against removing the arc_meta_min as was recently discussed.

[grahamperrin]

#12028 (comment)

… l2arc_write_max, l2arc_write_boost, l2arc_headroom, l2arc_noprefetch parameters in man 5 zfs-module-parameters.

I can't find that manual page:

https://openzfs.github.io/openzfs-docs/man/5/index.html

% man 5 zfs-module-parameters
No manual entry for zfs-module-parameters
% man zfs-module-parameters
No manual entry for zfs-module-parameters
% apropos zfs-module-parameters
apropos: nothing appropriate
% zfs --version
zfs-2.1.99-FreeBSD_g6c8f03232
zfs-kmod-2.1.99-FreeBSD_g6c8f03232
% uname -srKU
FreeBSD 14.0-CURRENT 1400041 1400041
% 

[gamanakis]

Try a "man 4 zfs", they were moved to section 4 some time ago.

…

On Tue, Nov 23, 2021 at 7:56 AM Graham Perrin ***@***.***> wrote: #12028 (comment) <#12028 (comment)> … l2arc_write_max, l2arc_write_boost, l2arc_headroom, l2arc_noprefetch parameters in man 5 zfs-module-parameters. I can't find that manual page: https://openzfs.github.io/openzfs-docs/man/5/index.html % man 5 zfs-module-parameters No manual entry for zfs-module-parameters % man zfs-module-parameters No manual entry for zfs-module-parameters % apropos zfs-module-parameters apropos: nothing appropriate % zfs --version zfs-2.1.99-FreeBSD_g6c8f03232 zfs-kmod-2.1.99-FreeBSD_g6c8f03232 % uname -srKU FreeBSD 14.0-CURRENT 1400041 1400041 % — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#12028 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB2Y2IMTTTHNXF4WSGPKYNDUNM3K7ANCNFSM44XDFXXQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[devZer0]

it's extremely frustrating to see filling arc with data leads to eviction of data structures which refer to metadata in l2arc.

for example i can see sustaining 100% l2arc hits for repeated "rsync -av --dry-run /path /tmp" and as soon as other data hits the arc, metadata reads from l2arc go down to a low rate and io from metadata reads hits spinning disk again.

so this renders secondarycache=metadata param completely unusable.

anyhow, caching metadata in arc and l2arc is severely broken imho.

[shodanshok]

@devZer0 do you have l2arc prefetch enabled? Can you share the output of arc_summary (with tunables)? Do you have any other dataset with secondarycache=all?