Feat: expose per-dataset metrics in the ZFS collector on FreeBSD

[asomers]

Host operating system: output of uname -a

FreeBSD, all versions

node_exporter version: output of node_exporter --version

1.5.0

Feature

PR #1632 exposed ZFS's per-dataset performance metrics, but only on Linux. The same metrics are available on FreeBSD. However, whereas on Linux they are presented as files in /proc , on FreeBSD they are presented as sysctls. The ZFS collector already publishes non-dataset-specific metrics on FreeBSD, so I would think that it shouldn't be too hard to add the dataset-specific ones too. See https://github.com/asomers/ztop/blob/master/src/app.rs for an example (in Rust) of a project that parses both Linux's and FreeBSD's ZFS per-dataset metrics.

[conallob]

Exploring the sysctl metrics of my FreeBSD system with ZFS, I see:

sysctl kstat.zfs.pool.dataset
...
kstat.zfs.pool.dataset.objset-0x40.nunlinked: 2
kstat.zfs.pool.dataset.objset-0x40.nunlinks: 2
kstat.zfs.pool.dataset.objset-0x40.nread: 34942
kstat.zfs.pool.dataset.objset-0x40.reads: 18
kstat.zfs.pool.dataset.objset-0x40.nwritten: 792
kstat.zfs.pool.dataset.objset-0x40.writes: 1
kstat.zfs.pool.dataset.objset-0x40.dataset_name: pool/home
...

I'm happy to iterate across such entries and plumb them into collector/zfs_freebsd.go

[seeplusplus]

Posting an update here for anyone looking for the feature in the future:

The original implementation in #2753 calls sysctl (3), which is - as OP pointed out - the only way to get this information in FreeBSD. At least, I haven't found any other way to get it. The current blocker (as far as I can estimate) is that sysctl (3) (actually sysctlbyname) requires fully qualified mibs e.g., kstat.zfs.pool.dataset.objset-0x40.nread, and doesn't accept partials e.g., kstat.zfs.pool.dataset. This may be confusing because partials do work in sysctl (8). However, this means that for us to add this feature to node_exporter we need a way to get the pool labels (names?) prior to calling sysctl (3).

I don't know how to do this. OP's example (https://github.com/asomers/ztop/blob/master/src/app.rs) seems to be a CLI that requires the user to specify pool names a priori, so it's not exactly useful in that regard (someone can check me on this, I may have missed something).

I would try taking inspiration for how sysctl (8) solves this problem:
https://github.com/freebsd/freebsd-src/blob/main/sbin/sysctl/sysctl.c

Which appears to be (somehow?) recursively searching the MIB tree for leaf-nodes.

I am looking into this - as well as any other Go zfs libs that might make this easier - but wanted to leave all of this info here for others to pick up or help if possible.

[asomers]

Actually, ztop does not require the user to specify a pool name; that's optional. What it does is it walks the whole tree of kstat.zfs and looks for any mibs ending in dataset_name to figure out what the objset-0x70 fields mean. It's been a while since i wrote it, but I think it relies on the mibs being returned in a specific order.

[asomers]

Also, because I lack the Go skills necessary to complete this task, I wrote a separate Prometheus exporter in Rust (based on the ztop code) to export this data. It's closed-source for now, but if there's community interest we would release it.

[seeplusplus]

Interesting. I'll have to read more into ztops implementation then. Thanks for sharing!

What it does is it walks the whole tree of kstat.zfs and looks for any mibs ending in dataset_name to figure out what the objset-0x70 fields mean

I think this is similar to what sysctl(8) is doing. So I think if I spend some time this weekend looking at both I should have a good understanding of how to solve this (final) missing piece.

It's closed-source for now, but if there's community interest we would release it.

I can't speak for the community, but I would certainly take any help I can get in figuring this out. I have only been tinkering with Go/FreeBSD for the last week or so all help is good help. Let me know if you'd rather share the code in a private channel like IRC/Matrix/Discord/Email.

[asomers]

It's closed-source for now, but if there's community interest we would release it.

I can't speak for the community, but I would certainly take any help I can get in figuring this out. I have only been tinkering with Go/FreeBSD for the last week or so all help is good help. Let me know if you'd rather share the code in a private channel like IRC/Matrix/Discord/Email.

As far as the sysctl-parsing part goes, the code is the same in the open source ztop as in the closed source ztop-exporter.

[dswarbrick]

I suspect that the kstat interface is fine if you already know what objset ID you are looking for, but for enumerating these in the first place, a journey into the world of ioctl is probably necessary. A truss of a command such as zfs get -Hpo value objsetid zroot/home indicates that that is likely the canonical method of getting a lot of this kind of information:

$ zfs get -Hpo value objsetid zroot/usr
75

openat(AT_FDCWD,"/dev/zfs",O_RDWR|O_EXCL|O_CLOEXEC,00) = 3 (0x3)
mmap(0x0,135168,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 82089848340480 (0x4aa908400000)
openat(AT_FDCWD,"/dev/zfs",O_RDWR|O_CLOEXEC,00)  = 4 (0x4)
__sysctlbyname("vfs.zfs.version.ioctl",21,0x374c5d4aab64,0x374c5d4aab68,0x0,0) = 0 (0x0)
ioctl(3,0xc0185a12 { IORW 0x5a('Z'), 18, 24 },0x374c5d4aab68) = 0 (0x0)
ioctl(3,0xc0185a05 { IORW 0x5a('Z'), 5, 24 },0x374c5d4a98c8) = 0 (0x0)
fstat(1,{ mode=crw--w---- ,inode=112,size=0,blksize=4096 }) = 0 (0x0)
ioctl(1,TIOCGETA,0x374c5d4ab230)                 = 0 (0x0)
75
write(1,"75\n",3)                               = 3 (0x3)
ioctl(3,0xc0185a3f { IORW 0x5a('Z'), 63, 24 },0x374c5d4ab2b8) ERR#1 'Operation not permitted'
close(3)                                         = 0 (0x0)
close(4)                                         = 0 (0x0)

hex(75) = 0x4b ...

$ sysctl kstat.zfs.zroot.dataset.objset-0x4b
kstat.zfs.zroot.dataset.objset-0x4b.zil_itx_metaslab_slog_alloc: 0
kstat.zfs.zroot.dataset.objset-0x4b.zil_itx_metaslab_slog_write: 0
kstat.zfs.zroot.dataset.objset-0x4b.zil_itx_metaslab_slog_bytes: 0
kstat.zfs.zroot.dataset.objset-0x4b.zil_itx_metaslab_slog_count: 0
kstat.zfs.zroot.dataset.objset-0x4b.zil_itx_metaslab_normal_alloc: 10747904
kstat.zfs.zroot.dataset.objset-0x4b.zil_itx_metaslab_normal_write: 10723328
kstat.zfs.zroot.dataset.objset-0x4b.zil_itx_metaslab_normal_bytes: 10528600
kstat.zfs.zroot.dataset.objset-0x4b.zil_itx_metaslab_normal_count: 87
kstat.zfs.zroot.dataset.objset-0x4b.zil_itx_needcopy_bytes: 10038112
kstat.zfs.zroot.dataset.objset-0x4b.zil_itx_needcopy_count: 2465
kstat.zfs.zroot.dataset.objset-0x4b.zil_itx_copied_bytes: 0
kstat.zfs.zroot.dataset.objset-0x4b.zil_itx_copied_count: 0
kstat.zfs.zroot.dataset.objset-0x4b.zil_itx_indirect_bytes: 503804
kstat.zfs.zroot.dataset.objset-0x4b.zil_itx_indirect_count: 5
kstat.zfs.zroot.dataset.objset-0x4b.zil_itx_count: 2472
kstat.zfs.zroot.dataset.objset-0x4b.zil_commit_writer_count: 3
kstat.zfs.zroot.dataset.objset-0x4b.zil_commit_count: 3
kstat.zfs.zroot.dataset.objset-0x4b.nunlinked: 517
kstat.zfs.zroot.dataset.objset-0x4b.nunlinks: 517
kstat.zfs.zroot.dataset.objset-0x4b.nread: 53573101
kstat.zfs.zroot.dataset.objset-0x4b.reads: 29654
kstat.zfs.zroot.dataset.objset-0x4b.nwritten: 17462677
kstat.zfs.zroot.dataset.objset-0x4b.writes: 3470
kstat.zfs.zroot.dataset.objset-0x4b.dataset_name: zroot/home

Something like zfs list -o name,objsetid also clearly uses ioctls to enumerate the available filesystems and their objset ID.