Node-Exporter : memory usage too high (OOME)

[SpeedBlack]

Host operating system: output of uname -a

3.10.0-862.3.2.el7.x86_64

node_exporter version: output of node_exporter --version

sh-4.2$ node_exporter --version
node_exporter, version 0.16.0 (branch: HEAD, revision: d42bd70)
build user: root@node-exporter-binary-3-build
build date: 20180606-16:48:15
go version: go1.10

node_exporter command line flags

--path.procfs=/host/proc --path.sysfs=/host/sys

Are you running node_exporter in Docker?

Yes, in Openshift

Hi,

I use node-exporter (openshift/prometheus-node-exporter:v0.16.0) in Openshift with Prometheus and Grafana.
I have a problem with memory recycling.
The Pod is killed each time after at an OOME (OutOfMemory). Memory usage increases continuously without being recycled.

By default, the limits were (template here):

          resources:
            requests:
              memory: 30Mi
              cpu: 100m
            limits:
              memory: 50Mi
              cpu: 200m

I tested several configurations without success.
Today, the configuration is:

          resources:
            limits:
              cpu: 250m
              memory: 250Mi
            requests:
              cpu: 100m
              memory: 75Mi

Do you have any idea ?
An adjustment to make ?
Do you have recommended resources limits ?

Thanks in advance for your help !

[SuperQ]

The easiest thing to do is get a pprof memory profile.

See #859 for some more tips.

It would be useful to look at the the node_exporter's metric, go_goroutines over time to see if there is a leak.

[SpeedBlack]

Hi SuperQ,

Below, the memory usage of the "node-exporter" pod deployed on a node :

below the graph with the metrics go_goroutines for the same node : go_goroutines{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="xxxxxx",job="kubernetes-nodes-exporter",kubernetes_io_hostname="xxxxxx",.....}

What is this metric ?

I'm looking at how to get the pprof.

Thanks,
Best regards.

[SuperQ]

Yes, that looks like a goroutine leak. Can you post any pod logs to see if there are any errors?

[SpeedBlack]

Below the pod logs :

$ oc logs -f prometheus-node-exporter-krvrb
time="2018-07-19T03:57:03Z" level=info msg="Starting node_exporter (version=0.16.0, branch=HEAD, revision=d42bd70f4363dced6b77d8fc311ea57b63387e4f)" source="node_exporter.go:82"
time="2018-07-19T03:57:03Z" level=info msg="Build context (go=go1.10, user=root@node-exporter-binary-3-build, date=20180606-16:48:15)" source="node_exporter.go:83"
time="2018-07-19T03:57:03Z" level=info msg="Enabled collectors:" source="node_exporter.go:90"
time="2018-07-19T03:57:03Z" level=info msg=" - arp" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - bcache" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - bonding" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - conntrack" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - cpu" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - diskstats" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - edac" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - entropy" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - filefd" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - filesystem" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - hwmon" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - infiniband" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - ipvs" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - loadavg" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - mdadm" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - meminfo" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - netdev" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - netstat" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - nfs" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - nfsd" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - sockstat" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - stat" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - textfile" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - time" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - timex" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - uname" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - vmstat" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - wifi" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - xfs" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - zfs" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg="Listening on :9100" source="node_exporter.go:111"

Thanks

[SuperQ]

Thanks, hopefully the pprof dump will give us the information we need.

[SpeedBlack]

Hi Superq,

I could run the command but Graphviz is not installed and I can't install it in the pod:

go tool pprof -svg http://localhost:9090/debug/pprof/heap > heap.svg                                                                                                                      
Fetching profile over HTTP from http://localhost:9090/debug/pprof/heap                                                                                                                            
Could not use temp dir /pprof: mkdir /pprof: permission denied                                                                                                                                    
Saved profile in /tmp/pprof.prometheus.alloc_objects.alloc_space.inuse_objects.inuse_space.003.pb.gz                                                                                              
Failed to execute dot. Is Graphviz installed? Error: exec: "dot": executable file not found in $PATH

I have to run the command in the pod because I have an OCP auth-proxy for prometheus.

But I have the raw data ("pprof.prometheus.alloc_objects.alloc_space.inuse_objects.inuse_space.003.pb.gz" file).
Do you know how I can generate the graph afterwards with Graphviz on another system ? I haven't found the command yet... or the tool.

Thanks,
Best regards.

[SpeedBlack]

Sorry, it's Ok :

it's Ok ?

Thanks,
Best regards

[SuperQ]

That is a profile for Prometheus, not the node exporter. You need to use port 9100.

[SpeedBlack]

Hoo sorry,
Please, find attachment pprof svg for node-exporter.

pprof_node_exporter.zip

Thanks
Best regards

[SpeedBlack]

Below, the memory usage of the node-exporter since the restart of the pod:

[SuperQ]

It would help to take a pprof sample when it's using over 200MB of memory, this way we can more easily find the memory leak.

[SpeedBlack]

I took a pprof sample at 12:24:00. The node-exporter used 248MiB :

is it better to lower the MEMORY limit and take a pprof sample ? (Currently 250Mi | By default 50Mi)

Thanks,
Best regards.

[SuperQ]

Strange. The pprof only shows 9.6MiB fo memory in use.

I would probably eliminate the CPU limit. I have seen CPU limits in K8s cause bad performance stalling. But I haven't looked at the current implementation to see if that's improved or not.

/cc @juliusv Do you know if there is something specific we can do to trace this goroutine leak?

[SpeedBlack]

Hi,
I tried to remove CPU limits without result.
I'll take another pprof sample.

Thanks,
Best regards

[SuperQ]

One additional debugging option, get a dump of the goroutines with http://localhost:9100/debug/pprof/goroutine?debug=1. Normally it would be only a few, like this.