Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

metrics: improve accuracy of CPU gauges #26793

Merged
merged 9 commits into from
Mar 6, 2023
Merged

metrics: improve accuracy of CPU gauges #26793

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
b70fe66
metrics: measure collection frequency
turboboost55 Mar 1, 2023
3882520
keep the linter happy
turboboost55 Mar 2, 2023
3731a89
avoid loss of precision due to truncation
turboboost55 Mar 3, 2023
272b00d
linter issues
turboboost55 Mar 5, 2023
66ed572
metrics: use float64 gauge
fjl Mar 6, 2023
7d10a5b
Merge branch 'ethereum:master' into metrics1
turboboost55 Mar 6, 2023
0c9e714
metrics: compute CPU percentage metrics using float64 as base unit
fjl Mar 6, 2023
66677d3
metrics: move update of lastCollectionTime
fjl Mar 6, 2023
25a5eff
metrics: read time only once
fjl Mar 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions metrics/cpu.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,9 @@
package metrics

// CPUStats is the system and process CPU stats.
// All values are in seconds.
type CPUStats struct {
GlobalTime int64 // Time spent by the CPU working on all processes
GlobalWait int64 // Time spent by waiting on disk for all processes
LocalTime int64 // Time spent by the CPU working on this process
GlobalTime float64 // Time spent by the CPU working on all processes
GlobalWait float64 // Time spent by waiting on disk for all processes
LocalTime float64 // Time spent by the CPU working on this process
}
4 changes: 2 additions & 2 deletions metrics/cpu_enabled.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ func ReadCPUStats(stats *CPUStats) {
}
// requesting all cpu times will always return an array with only one time stats entry
timeStat := timeStats[0]
stats.GlobalTime = int64((timeStat.User + timeStat.Nice + timeStat.System) * cpu.ClocksPerSec)
stats.GlobalWait = int64((timeStat.Iowait) * cpu.ClocksPerSec)
stats.GlobalTime = timeStat.User + timeStat.Nice + timeStat.System
stats.GlobalWait = timeStat.Iowait
stats.LocalTime = getProcessCPUTime()
}
2 changes: 1 addition & 1 deletion metrics/cputime_nop.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,6 @@ package metrics

// getProcessCPUTime returns 0 on Windows as there is no system call to resolve
// the actual process' CPU time.
func getProcessCPUTime() int64 {
func getProcessCPUTime() float64 {
return 0
}
4 changes: 2 additions & 2 deletions metrics/cputime_unix.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,11 +26,11 @@ import (
)

// getProcessCPUTime retrieves the process' CPU time since program startup.
func getProcessCPUTime() int64 {
func getProcessCPUTime() float64 {
var usage syscall.Rusage
if err := syscall.Getrusage(syscall.RUSAGE_SELF, &usage); err != nil {
log.Warn("Failed to retrieve CPU time", "err", err)
return 0
}
return int64(usage.Utime.Sec+usage.Stime.Sec)*100 + int64(usage.Utime.Usec+usage.Stime.Usec)/10000 //nolint:unconvert
return float64(usage.Utime.Sec+usage.Stime.Sec) + float64(usage.Utime.Usec+usage.Stime.Usec)/1000000 //nolint:unconvert
}
19 changes: 13 additions & 6 deletions metrics/metrics.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,8 +127,6 @@ func CollectProcessMetrics(refresh time.Duration) {
return
}

refreshFreq := int64(refresh / time.Second)

// Create the various data collectors
var (
cpustats = make([]CPUStats, 2)
Expand Down Expand Up @@ -163,14 +161,23 @@ func CollectProcessMetrics(refresh time.Duration) {
diskWriteBytesCounter = GetOrRegisterCounter("system/disk/writebytes", DefaultRegistry)
)

// Avoid divide-by-zero the first time through the loop.
lastCollectionTime := time.Now().Add(-refresh)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's very wonky to me. You are adding a negative refresh nano seconds.

Copy link
Contributor Author

@turboboost55 turboboost55 Mar 3, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know, that does looks weird. There is a Sub function, but it only subtracts a Time value, not a Duration. According to the documentation, that's how you subtract a Duration: To compute t-d for a duration d, use t.Add(-d).

https://pkg.go.dev/time#Time.Sub


// Iterate loading the different stats and updating the meters.
now, prev := 0, 1
for ; ; now, prev = prev, now {
// CPU
// Gather CPU times.
ReadCPUStats(&cpustats[now])
cpuSysLoad.Update((cpustats[now].GlobalTime - cpustats[prev].GlobalTime) / refreshFreq)
cpuSysWait.Update((cpustats[now].GlobalWait - cpustats[prev].GlobalWait) / refreshFreq)
cpuProcLoad.Update((cpustats[now].LocalTime - cpustats[prev].LocalTime) / refreshFreq)
refreshFreq := time.Since(lastCollectionTime).Seconds()
lastCollectionTime = time.Now()
sysLoad := (cpustats[now].GlobalTime - cpustats[prev].GlobalTime) / refreshFreq
sysWait := (cpustats[now].GlobalWait - cpustats[prev].GlobalWait) / refreshFreq
procLoad := (cpustats[now].LocalTime - cpustats[prev].LocalTime) / refreshFreq
// Convert to integer percentage.
cpuSysLoad.Update(int64(sysLoad * 100))
cpuSysWait.Update(int64(sysWait * 100))
cpuProcLoad.Update(int64(procLoad * 100))

// Threads
cpuThreads.Update(int64(threadCreateProfile.Count()))
Expand Down