[hostmetrics] change cpu.load.average metric units

[povilasv]

Description:

The system.cpu.load_average.5m uses 1 as unit, which is reserved for utilization metrics - https://opentelemetry.io/docs/specs/semconv/general/metrics/#instrument-units

CPU Load Average does not track utilization and can go above

Link to tracking Issue:

#29914

Testing:

Documentation:

• generated

[dmitryax]

Given the definition

CPU load is the number of processes which are being executed by CPU or waiting to be executed by CPU. So CPU load average is the average number of processes being or waiting executed over past 1, 5 and 15 minutes.

Should the unit be {cpu}? @braydonk @mx-psi WDYT?

[bogdandrutu]

"default unit 1" it is default for all the metrics not only for utilization. See https://github.com/open-telemetry/semantic-conventions/blob/b2e2c64b3f16b3d70b867cbaefc01936f23c1f4c/docs/general/metrics.md?plain=1#L195 and can read also https://unitsofmeasure.org/ucum.

[bogdandrutu]

@dmitryax we can have it as "{cpu}" but we should not have as empty string.

[braydonk]

Should the unit be {cpu}?

My instinct is probably not? {cpu} is used elsewhere to represent a number of CPUs, which doesn't feel appropriate for load average.

1 actually would make sense as a unit in a system with a single CPU (a good explanation is in the uptime manpage). The reason this can exceed 1 is because it can go up to 1 for every CPU on the system. This has similarities to a utilization metric, but it's not exactly the same. I wonder if there are other scenarios like this, it feels unique and nothing in the linked semantic conventions docs are standing out to me as a rule to follow here.

I am not sure how to handle this scenario as this is a strange metric. If I were to suggest something now, I would say it should be its own unit {load_avg}. I'm curious to hear other suggestions on this.

[povilasv]

Linux Load averages don't measure CPU, I've seen load averages in 1000s and stuff like that, when there a lot of processes in D state, due to waiting for i/o or release of kernel lock.

Good article about this is -> https://tanelpoder.com/posts/high-system-load-low-cpu-utilization-on-linux/

Also found a good post that explain what actually Linux Load Averages measure -> https://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html

Linux load averages are "system load averages" that show the running thread (task) demand on the system as an average number of running plus waiting threads. This measures demand, which can be greater than what the system is currently processing. Most tools show three averages, for 1, 5, and 15 minutes:

So the unit here seems to be threads?

For Windows, we seem to compute the average using Processor Queue Length, which is:

The Processor Queue Length is the number of threads that are ready to run on the processor but cannot because of another active thread.

Unit also seems to be threads.

So my suggestion for unit is {thread}.

WDYT?

[mx-psi]

+1 to doing {thread} after reading @povilasv's links

[braydonk]

Thanks for the additional documentation, it seems I misunderstood what the load average really meant.

{thread} is already a unit in process metrics for the process.threads metric.* In that metric, the unit refers to an actual exact number of threads, whereas this metric refers to an average. So similarly to {cpu} as a unit, using {thread} here would be a clash with the other usage of that as a unit.

* The unit there is {threads} plural, which I'm pretty sure it's not supposed to be. That will probably be changed in semantic conventions updates along with the name of the metric to process.thread.count.

[povilasv]

I don't understand the issue with clashing units? Why it's an issue?

Taking simple example if one metric gives my count of current threads running, the unit is {thread}, if another metrics gives me average amount of threads in 5 minutes window the unit is still {thread}?

Ref: https://stats.stackexchange.com/questions/187918/does-average-have-units

[braydonk]

Maybe it's fine then. I can't see any other example in semantic conventions of this case ever occurring (using the same unit for a specific number or average) but if that's not against the rules then {thread} is fine as a unit.

[povilasv]

FYI I've changed the unit to {thread}, let me know what you think @dmitryax :)

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[dmitryax]

{thread} sounds good to me