ZFS: io stats removed

[chrisrd]

Host operating system:

OpenZFS as of commit openzfs/zfs@371f88d

node_exporter version

$ /usr/bin/prometheus-node-exporter --version
node_exporter, version 0.17.0+ds (branch: debian/sid, revision: 0.17.0+ds-3+b11)
  build user:       pkg-go-maintainers@lists.alioth.debian.org
  build date:       20190608-10:00:14
  go version:       go1.11.6

...through to master, current HEAD (3762191)

node_exporter command line flags

--collector.zfs

What did you do that produced an error?

Upgraded to OpenZFS HEAD openzfs/zfs@HEAD

What did you expect to see?

$ wget -qO - http://server:9100/metrics | grep ^node_zfs_zpool || echo "NOT FOUND"
node_zfs_zpool_nread{zpool="pool"} 2.49897949175808e+14
node_zfs_zpool_nwritten{zpool="pool"} 2.65591973699584e+14
node_zfs_zpool_rcnt{zpool="pool"} 0
node_zfs_zpool_reads{zpool="pool"} 2.729451643e+09
node_zfs_zpool_rlentime{zpool="pool"} 4.728605718663304e+16
node_zfs_zpool_rtime{zpool="pool"} 5.775579362149045e+15
node_zfs_zpool_rupdate{zpool="pool"} 1.3886342123071748e+16
node_zfs_zpool_wcnt{zpool="pool"} 0
node_zfs_zpool_wlentime{zpool="pool"} 2.8960276949205395e+17
node_zfs_zpool_writes{zpool="pool"} 3.436014883e+09
node_zfs_zpool_wtime{zpool="pool"} 1.319409654928622e+15
node_zfs_zpool_wupdate{zpool="pool"} 1.3886342121834806e+16

What did you see instead?

$ wget -qO - http://server:9100/metrics | grep ^node_zfs_zpool || echo "NOT FOUND"
NOT FOUND

Node_exporter uses /proc/spl/kstat/zfs/*/io to collect these stats. This proc file has been removed as of OpenZFS openzfs/zfs@371f88d.

See Also: openzfs/zfs#12212

[discordianfish]

Ok, looks like someone needs to implement the "more efficient ways .. to obtain detailed statistics".

[chrisrd]

The "more efficient ways" are probably as done by zpool iostat. That uses an zfs ioctl to pull in at least the nread, nwritten, reads and writes part of the stats (i.e. the stuff I'm most interested in). E.g.:

# zpool iostat -p
                        capacity                         operations                         bandwidth           
pool                  alloc             free             read            write             read            write
----------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------
pool        126300985387008   71611107612672             1646             3625         17295293          8371479
fast          4126006828544    3845452472832               12               92           516551          8286794
----------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------

(And more stats are available, e.g. -q shows queue stats, -v shows per-vdev stats.)

The question would be how to pull this stuff in:

• run the external command and parse the output
• use the libzfs2 functions as zpool iostat does, e.g. zpool_refresh_stats()
• use the ioctl directly
  I'm not sure if libzfs2 is supposed to be internal only or meant to be used by external projects.

The other fields in the now-defunct "io" file are from struct kstat_io described here:

https://github.com/openzfs/zfs/blob/master/lib/libspl/include/sys/kstat.h#L582

I don't know which of these are more or less useful to try to get.

Sadly, my knowledge of Go is practically nonexistent, as is the time I have to put towards this.

[discordianfish]

We don't allow running external commands. Technically using libzfs2 might be an option, but we'd try to avoid linking to c libraries. So ideally we can use the ioctl directly but dunno how involved that would be.

[chrisrd]

Looks like it's not straight forward (you've gotta pull apart "nvlists" and ...stuff), but someone's started on it: https://github.com/lorenz/go-zfs

[chrisrd]

There's another "go-zfs" project at https://github.com/mistifyio/go-zfs but that one appears to be a wrapper around the command line utilities so not suitable given "We don't allow running external commands".

[chrisrd]

Huh. And another at https://github.com/bicomsystems/go-libzfs which links to the libzfs c library so less suitable.

[discostur]

@discordianfish @chrisrd anything new on this? Encountered the same bug after upgrading openzfs to the latest release ...

[chrisrd]

@discostur Nothing new from me - I'm not in a position to fix this.

In the meantime I'm just monitoring one of the disks in my pool with:

irate(node_disk_read_bytes_total{device="sdf"}[5m])
irate(node_disk_written_bytes_total{device="sdf"}[5m])

This obviously doesn't provide actual pool-wide read/write stats, but it's enough to track how busy the pool is over time.

[discordianfish]

@discostur No unfortunately not. Someone should look into using to get these stats from https://github.com/lorenz/go-zfs. I might get to that eventually but can't commit to when.

[lorenz]

FYI I (the go-zfs author) am working on getting a ZFS exporter up and running. Once I have something we can work on integrating it into node_exporter.

[lorenz]

I invite everyone here to give feedback on https://github.com/lorenz/zfs_exporter. As we get consensus on which metrics to expose, how and under what names we can start merging this back into node_exporter, but I think we can iterate this faster outside.

[discordianfish]

Awesome! For the node-exporter we're trying to not break existing metrics before the next major release, so ideally in the version for the node-exporter the change shouldn't change the metrics generated for now. Or it could be a new collector that is disabled by default.

Alternatively, I was also thinking about adding versioned metrics endpoint. But I guess that requires some further discussion which I should finally kick off.

[jakubvojacek]

Hello

was this resolved somehow, are there new metrics/collector available that exposes node_zfs_zpool_*?

Thanks

[retzkek]

Just a note, since I'm not sure it was clear from the conversation so far, this is breaking the collection of all zpool metrics, since if the */io procfiles aren't found the collector doesn't try any of the others. https://github.com/prometheus/node_exporter/blob/master/collector/zfs_linux.go#L77=