Add an incus top command

[stgraber]

It would be nice to have an incus top command which accesses the metrics API (/1.0/metrics) and renders a list of instances with their CPU, memory and disk usage columns (similar to top).

That should all be possible with our existing APIs, so it's just CLI work to implement such a command and provide some hotkeys to switch the column sorting and refresh frequency.

[CharanSriramUni]

Hey! @stgraber I'm a student at UT Austin (same virtualization class as other posters). Could I claim this issue?

[stgraber]

Sure!

[CharanSriramUni]

Hey @stgraber! I've gotten some work on this issue and had a few questions:

• NVM - forgot to have an instance running
• How should CPU usage be recorded? /1.0/metrics displays CPU usage per core in term of seconds(?) utilized (incus_cpu_seconds_total{cpu="0",mode="user",name="second",project="default",type="container"} 0.534083444 for example), and the total number of effective CPUs
• Are there any arguments this command should take?

[stgraber]

Are there any arguments this command should take?

Not initially no. I suspect that down the line it will be expanded with more options on what columns to show, what columns to sort by, refresh interval, ... but none of that is needed for this initial implementation.

How should CPU usage be recorded? /1.0/metrics displays CPU usage per core in term of seconds(?) utilized (incus_cpu_seconds_total{cpu="0",mode="user",name="second",project="default",type="container"} 0.534083444 for example), and the total number of effective CPUs

I think we should just compute the total CPU time in seconds, basically adding up user and system on all CPUs.

[CharanSriramUni]

Got it. Sounds good, I'll modify it accordingly.

Relatedly I see that d.GetMetrics returns a string instead of the Metric type that is defined in the codebase. Is there a way to unmarshal the string into a Go struct of this type? Can't find any examples of string -> MetricSet

Lastly, for disk usage, the disk only reports information on reads/writes completed/ & total bytes read/written. The filesystem has information on "free space" and "total space" so I'm going to assume utilization is calculated as (total space - free space) / total space unless there's a different metric to use.

[stgraber]

Relatedly I see that d.GetMetrics returns a string instead of the Metric type that is defined in the codebase. Is there a way to unmarshal the string into a Go struct of this type? Can't find any examples of string -> MetricSet

MetricSet is an internal representation of those metrics on the server side, it's not a struct that's available to clients.

Parsing those lines shouldn't be too difficult to do by hand.
Alternatively using https://pkg.go.dev/github.com/prometheus/common/expfmt may be an option, we just want to be pretty careful not to bring in a lot of extra external code just for this.

[stgraber]

Lastly, for disk usage, the disk only reports information on reads/writes completed/ & total bytes read/written. The filesystem has information on "free space" and "total space" so I'm going to assume utilization is calculated as (total space - free space) / total space unless there's a different metric to use.

Yeah, that's right, the relevant thing is likely to be the filesystem metrics and you indeed get usage by deducting free from total.