Permission denied on statfs() #66

Zikoel · 2017-10-17T10:44:31Z

When I start docker-composer on log output I see a bunch of error every time metrics was called from prometeus. The error was permissions denied:

node-exporter_1 | time="2017-10-17T10:40:38Z" level=error msg="Error on statfs() system call for \"/rootfs/run/docker/netns/default\": permission denied" source="filesystem_linux.go:57"

node-exporter_1 | time="2017-10-17T10:40:38Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay2/38801ac617091d009b3767fffd86acf3c3a8bd676065ed13ee04826cd2294a5e/merged\": permission denied" source="filesystem_linux.go:57"
There is a solution for this? I run docker-compose on debian 9

The text was updated successfully, but these errors were encountered:

vegasbrianc · 2017-10-17T19:31:34Z

Hi @Zikoel Let me look into this.

vegasbrianc · 2017-10-18T05:44:43Z

I disabled the node-exporter in the stack until we figure out why it is not behaving correctly across the different OS platforms. In the mean time cAdvisor is still working.

llitfkitfk · 2017-11-02T02:32:35Z

prometheus/node_exporter#392
prometheus/node_exporter#703
prometheus/node_exporter#697

sradnev · 2017-11-10T15:57:11Z

Refer to this comment.

You'll have to run as root or ignore those mountpoints, i.e. run node_exporter with something like
--collector.filesystem.ignored-mount-points "^/rootfs/(var/lib/docker/)|(run/docker/netns/).*".

vegasbrianc · 2017-11-13T15:46:59Z

@Zikoel Is this working for you based on the information provided?

llitfkitfk · 2017-11-14T04:29:53Z

fixed in #74

ntelisil · 2017-11-28T17:32:25Z

I copied the fix (the new regex for collector.filesystem.ignored-mount-points) in my docker-compose file for node_exporter and still getting the "Error on statfs() system call... permissions denied" messages:
time="2017-11-28T17:29:34Z" level=error msg="Error on statfs() system call for "/rootfs/var/lib/docker/aufs/mnt/07f0b6acbe62194ce8b5fd1fb73be8a478979e455040e0495a4f74a59ca4d13f/sys/fs/cgroup/blkio": permission denied" source="filesystem_linux.go:57"
time="2017-11-28T17:29:34Z" level=error msg="Error on statfs() system call for "/rootfs/var/lib/docker/aufs/mnt/07f0b6acbe62194ce8b5fd1fb73be8a478979e455040e0495a4f74a59ca4d13f/sys/fs/cgroup/freezer": permission denied" source="filesystem_linux.go:57"
...

Any ideas?

llitfkitfk · 2017-11-29T03:14:07Z

@ntelisil

Can you replace

- --collector.filesystem.ignored-mount-points
- "^/(sys|proc|dev|host|etc|rootfs/var/lib/docker/containers|rootfs/var/lib/docker/overlay2|rootfs/run/docker/netns|rootfs/var/lib/docker/aufs)($$|/)"

to

- --collector.filesystem.ignored-mount-points
- "^/(sys|proc|dev|host|etc|rootfs/var/lib/docker/containers|rootfs/var/lib/docker/overlay2|rootfs/run/docker/netns)($$|/)"
- --collector.filesystem.ignored-fs-types
- "^/(sys|proc|auto|rootfs/var/lib/docker/aufs)($$|/)"

and retry.

ntelisil · 2017-11-29T12:35:32Z

@llitfkitfk

Unfortunately that didn't work either.
After playing a little bit I found out that the affected mount point should be part of the ignored-mount-points regex":

- --collector.filesystem.ignored-mount-points
- "^/(sys|proc|dev|host|etc|rootfs/var/lib/docker/containers|rootfs/var/lib/docker/overlay2|rootfs/run/docker/netns|rootfs/var/lib/docker/aufs)($$|/)"

After adding this I no longer see the error messages.

Does this by any means affect the metrics that node_exporter gather? Or it is totally irrelevant...?

In another experiment I changed the user to root in node_exporter's Dockerfile (USER root) and the error message was not present even when the aforementioned mount point was not in ignored-mount-points.

So I can see to options:
If metrics gathering is not affected go with the ignored-mount-points update
else
set the user inside the container to root (or investigate further, e.g. container capabilities...)

gabrielstein · 2017-12-12T10:48:56Z

Hello!

I'm using:
CentOS Linux release 7.4.1708 (Core)
Docker? No
SeLinux Enable: Tested with Enforcing and Permissive
Node Exporter v0.15.1
Systemd(Service) mit:

[Unit]
Description=Prometheus node_exporter
Wants=basic.target
After=basic.target network.target

[Service]
User=node-exporter
Group=node-exporter
ExecStart=/usr/local/bin/node_exporter \
--collector.filesystem.ignored-mount-points  "^/(sys|proc|dev|host|etc|rootfs/var/lib/docker/containers|rootfs/var/lib/docker/overlay2|rootfs/run/docker/netns|rootfs/var/lib/docker/aufs)($$|/)"
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=always

[Install]
WantedBy=multi-user.target

I already tested all options from the answers above, doesn't works.

Error:

Dez 12 11:46:22 chuck-norris.always.survives node_exporter[17479]: time="2017-12-12T11:46:22+01:00" level=error msg="Error on statfs() system call for \"/var/lib/docker/containers/3b904624f3f78c0a0bea28242d71b058649e496cd370f61dfd914d611dec669c/shm\": permission denied" source="filesystem_linux.go:57"
Dez 12 11:46:22 chuck-norris.always.survives node_exporter[17479]: time="2017-12-12T11:46:22+01:00" level=error msg="Error on statfs() system call for \"/var/lib/docker/containers/98cf0acb0a216b90c829beada7f07ae0ffa8c718563f1bc0a417e9dbf976b6cf/shm\": permission denied" source="filesystem_linux.go:57"
Dez 12 11:46:22 chuck-norris.always.survives node_exporter[17479]: time="2017-12-12T11:46:22+01:00" level=error msg="Error on statfs() system call for \"/var/lib/docker/containers/0c7fcfe251dbf2c963f5a2f6ebd188f8520b492409666c8879269817ac06c2d8/shm\": permission denied" source="filesystem_linux.go:57"
Dez 12 11:46:22 chuck-norris.always.survives node_exporter[17479]: time="2017-12-12T11:46:22+01:00" level=error msg="Error on statfs() system call for \"/var/lib/docker/containers/df231cc28c1a9600c5941bdb6ee0485d890ddab7782f9ccca6f508943214e52b/shm\": permission denied" source="filesystem_linux.go:57"
Dez 12 11:46:22 chuck-norris.always.survives node_exporter[17479]: time="2017-12-12T11:46:22+01:00" level=error msg="Error on statfs() system call for \"/run/docker/netns/ecf6554bda8d\": permission denied" source="filesystem_linux.go:57"
Dez 12 11:46:22 chuck-norris.always.survives node_exporter[17479]: time="2017-12-12T11:46:22+01:00" level=error msg="Error on statfs() system call for \"/run/docker/netns/d0d90749145b\": permission denied" source="filesystem_linux.go:57"
Dez 12 11:46:22 chuck-norris.always.survives node_exporter[17479]: time="2017-12-12T11:46:22+01:00" level=error msg="Error on statfs() system call for \"/run/docker/netns/91a83012616c\": permission denied" source="filesystem_linux.go:57"
Dez 12 11:46:22 chuck-norris.always.survives node_exporter[17479]: time="2017-12-12T11:46:22+01:00" level=error msg="Error on statfs() system call for \"/run/docker/netns/b16cf914fc6e\": permission denied" source="filesystem_linux.go:57"
Dez 12 11:46:22 chuck-norris.always.survives node_exporter[17479]: time="2017-12-12T11:46:22+01:00" level=error msg="Error on statfs() system call for \"/run/docker/netns/ingress_sbox\": permission denied" source="filesystem_linux.go:57"
Dez 12 11:46:22 chuck-norris.always.survives node_exporter[17479]: time="2017-12-12T11:46:22+01:00" level=error msg="Error on statfs() system call for \"/run/docker/netns/1-k5hvwdpqsj\": permission denied" source="filesystem_linux.go:57"

Am I doing something wrong?

Should I use --collector.filesystem.ignored-mount-points = .regex* or --collector.filesystem.ignored-mount-points .regex*

I'm implementing using Puppet.

Thanks in Advance!

gabrielstein · 2017-12-12T11:07:50Z

And one last thing: Version 0.14(Node Exporter) works without Problems.

llitfkitfk · 2017-12-12T14:37:26Z

@gabrielstein
Can you add option: --no-collector.hwmon to node-exporter and retry?

prometheus/node_exporter#697 (comment)

psistorm · 2018-01-08T12:56:04Z

Same problem here as described by @gabrielstein. Version 0.14 works perfect. The @llitfkitfk mentioned workaround with option --no-collector.hwmon does not work for me.

nsaud01 · 2018-01-08T17:15:15Z

I'm still getting time="Error on statfs() system call for \"/rootfs/var/lib/docker/containers/565b0b1afccf274158cb6366b99079907663193d957b193816f346400eadebde/shm\": permission denied" source="filesystem_linux.go:57"

Anything rootfs related is giving permission errors. This wasn't an issue in v14. My docker run command is:

docker run -d -p 9100:9100 \
  -v "/proc:/host/proc" \
  -v "/sys:/host/sys" \
  -v "/:/rootfs" \
  --net="host" \
  quay.io/prometheus/node-exporter:v0.15.2 \
    --path.procfs /host/proc \
    --path.sysfs /host/sys \
    --collector.filesystem.ignored-mount-points "^/(sys|proc|dev|host|etc|rootfs/var/lib/docker/containers|rootfs/var/lib/docker/overlay2|rootfs/run/docker/netns)($$|/)"

Is there something I'm missing?

MaxenceG2M · 2018-06-29T17:01:49Z

For information, I solve the problem with the option:

--collector.filesystem.ignored-mount-points "^/(var/lib/docker/)|(run/docker/netns/).*"

like suggest @sradnev (#66 (comment)) without rootfs part.

Hope this help.

yinbangzhong · 2019-04-10T09:25:24Z

level=error msg="ERROR: cpu collector failed after 0.000392s: open /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq: permission denied" source="collector.go:132"

idevai · 2019-06-09T12:07:05Z

If you are launching node exporter via systemd, the standard way to modify the startup parameters is by editing file

/etc/default/prometheus-node-exporter

You need to modify this line:

--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|run)($|/) \

Into this:

--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|run|var/lib/docker|tmp)($|/) \

and then restart by:

service prometheus-node-exporter restart

llitfkitfk mentioned this issue Nov 14, 2017

Update node exporter ignored mount points #74

Merged

llitfkitfk closed this as completed in #74 Nov 14, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Permission denied on statfs() #66

Permission denied on statfs() #66

Zikoel commented Oct 17, 2017

vegasbrianc commented Oct 17, 2017 •

edited

Loading

vegasbrianc commented Oct 18, 2017

llitfkitfk commented Nov 2, 2017 •

edited

Loading

sradnev commented Nov 10, 2017 •

edited

Loading

vegasbrianc commented Nov 13, 2017

llitfkitfk commented Nov 14, 2017

ntelisil commented Nov 28, 2017

llitfkitfk commented Nov 29, 2017 •

edited

Loading

ntelisil commented Nov 29, 2017 •

edited

Loading

gabrielstein commented Dec 12, 2017

gabrielstein commented Dec 12, 2017

llitfkitfk commented Dec 12, 2017 •

edited

Loading

psistorm commented Jan 8, 2018

nsaud01 commented Jan 8, 2018 •

edited

Loading

MaxenceG2M commented Jun 29, 2018 •

edited

Loading

yinbangzhong commented Apr 10, 2019

idevai commented Jun 9, 2019

Permission denied on statfs() #66

Permission denied on statfs() #66

Comments

Zikoel commented Oct 17, 2017

vegasbrianc commented Oct 17, 2017 • edited Loading

vegasbrianc commented Oct 18, 2017

llitfkitfk commented Nov 2, 2017 • edited Loading

sradnev commented Nov 10, 2017 • edited Loading

vegasbrianc commented Nov 13, 2017

llitfkitfk commented Nov 14, 2017

ntelisil commented Nov 28, 2017

llitfkitfk commented Nov 29, 2017 • edited Loading

ntelisil commented Nov 29, 2017 • edited Loading

gabrielstein commented Dec 12, 2017

gabrielstein commented Dec 12, 2017

llitfkitfk commented Dec 12, 2017 • edited Loading

psistorm commented Jan 8, 2018

nsaud01 commented Jan 8, 2018 • edited Loading

MaxenceG2M commented Jun 29, 2018 • edited Loading

yinbangzhong commented Apr 10, 2019

idevai commented Jun 9, 2019

vegasbrianc commented Oct 17, 2017 •

edited

Loading

llitfkitfk commented Nov 2, 2017 •

edited

Loading

sradnev commented Nov 10, 2017 •

edited

Loading

llitfkitfk commented Nov 29, 2017 •

edited

Loading

ntelisil commented Nov 29, 2017 •

edited

Loading

llitfkitfk commented Dec 12, 2017 •

edited

Loading

nsaud01 commented Jan 8, 2018 •

edited

Loading

MaxenceG2M commented Jun 29, 2018 •

edited

Loading