add avgRTT to NFS operation stats

[ksankeerth]

Hi Team/ @discordianfish,

Please check this PR and let me know if anything has to be changed.
Fixes #2550

We have to merge prometheus/procfs#487 before merging this PR.

Thanks

[ksankeerth]

Build and tests failed because of missing field in struct. Once we merge procfs PR, I guess these errors won't come.

[ksankeerth]

Hi Team/ @discordianfish,

Since this PR is merged prometheus/procfs#487, Can you please run CI again?

[discordianfish]

@ksankeerth You need to update the procfs dependency here first

[SuperQ]

This needs a rebase in order to pick up the procfs changes.

collector/mountstats_linux.go:760:15: op.AverageRTTMilliseconds undefined (type procfs.NFSOperationStats has no field or method AverageRTTMilliseconds)

[ethercflow]

@ksankeerth @discordianfish @SuperQ Sorry to bother you, any plan to merge this?

[discordianfish]

I've rebased, let see

[discordianfish]

@ksankeerth Looks like this still needs work

[ksankeerth]

@ksankeerth Looks like this still needs work

Sorry for missing this. I'll update tonight

[ksankeerth]

It seems, there are some e2e test failures. I'm looking at them.

[ksankeerth]

Hi @discordianfish / @SuperQ ,

I tested locally and can see NFS mountstats that I've added. But e2e tests fail because of old fixtures. Can You help me to update fixtures?

I've tried to update fixtures using command make update_fixtures but it doesn't update fixtures with new metrics.

 # HELP node_mountstats_nfs_operations_average_round_trip_time_seconds Duration from the time that the client's kernel sends the RPC request until the time it receives the reply.
# TYPE node_mountstats_nfs_operations_average_round_trip_time_seconds counter
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="ACCESS",protocol="tcp"} 0.003
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="ALLOCATE",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="BIND_CONN_TO_SESSION",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="CLONE",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="CLOSE",protocol="tcp"} 0.00075
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="COMMIT",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="COPY",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="COPY_NOTIFY",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="CREATE",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="CREATE_SESSION",protocol="tcp"} 0.001
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="DEALLOCATE",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="DELEGRETURN",protocol="tcp"} 0.0006666666666666666
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="DESTROY_CLIENTID",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="DESTROY_SESSION",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="EXCHANGE_ID",protocol="tcp"} 0.0015
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="FREE_STATEID",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="FSID_PRESENT",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="FSINFO",protocol="tcp"} 0.001
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="FS_LOCATIONS",protocol="tcp"} 0.001
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="GETACL",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="GETATTR",protocol="tcp"} 0.0009210526315789473
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="GETDEVICEINFO",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="GETDEVICELIST",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="GETXATTR",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="GET_LEASE_TIME",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LAYOUTCOMMIT",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LAYOUTERROR",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LAYOUTGET",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LAYOUTRETURN",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LAYOUTSTATS",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LINK",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LISTXATTRS",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LOCK",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LOCKT",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LOCKU",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LOOKUP",protocol="tcp"} 0.001
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LOOKUPP",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LOOKUP_ROOT",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="NULL",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="OFFLOAD_CANCEL",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="OPEN",protocol="tcp"} 0.0038333333333333336
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="OPEN_CONFIRM",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="OPEN_DOWNGRADE",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="OPEN_NOATTR",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="PATHCONF",protocol="tcp"} 0.001
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="READ",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="READDIR",protocol="tcp"} 0.002
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="READLINK",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="READ_PLUS",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="RECLAIM_COMPLETE",protocol="tcp"} 0.009
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="RELEASE_LOCKOWNER",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="REMOVE",protocol="tcp"} 0.0036666666666666666
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="REMOVEXATTR",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="RENAME",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="RENEW",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SECINFO",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SECINFO_NO_NAME",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SEEK",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SEQUENCE",protocol="tcp"} 0.003142857142857143
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SERVER_CAPS",protocol="tcp"} 0.001
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SETACL",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SETATTR",protocol="tcp"} 0.00425
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SETCLIENTID",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SETCLIENTID_CONFIRM",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SETXATTR",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="STATFS",protocol="tcp"} 0.00175
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SYMLINK",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="TEST_STATEID",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="WRITE",protocol="tcp"} 0.0065

Thanks,
Sankeerthan

[ksankeerth]

Hi Team,

Updated PR with required changes. Please check and let me know if more changes have to be done.

Thanks

[SuperQ]

I was just reviewing this, and the original procfs PR.

This is not a metric we can accept, as it is a computation against two metrics we already expose.

node_mountstats_operations_response_time_seconds_total / node_mountstats_operations_requests_total

Worse, this is actually an invalid and misleading new metric, because it produces a "since beginning of time" average, rather than the current average response time which is provided when you do the correct PromQL:

rate(node_mountstats_operations_response_time_seconds_total[1m]) / rate(node_mountstats_operations_requests_total[1m])