Network namespace awareness

[bluecmd]

Currently node_exporter silently ignores network namespaces.

Steps to reproduce:

modprobe dummy

At this point you should see a dummy0 interface being exported by node_exporter.

ip netns add test
ip link set dummy0 netns test

Actual behavior: The interface is gone.
Expected behavior: The interface is still listed.

This also applies to all the netstat metrics - they are different between network namespaces and should be exported as such.

To make this nice maybe node_exporter could have a label which network namespace the metric belongs to? Alternatively support querying a given network namespace would also work but not be as nice.

[SuperQ]

I think the normal advice is that cAdvisor is recommended to monitor metrics in Linux namespaces.

[bluecmd]

These are not cgroups, so I'm not sure it's 100% applicable, or that cAdvisor even has support for network namespaces.

[discordianfish]

I don't think just supporting network stats across namespaces makes sense. Either we support slicing all metrics by namespace or only report system wide metrics as we do right now.

If you think we should have general support for namespaces and/or cgroups, we should open an issue for that. But in general cAdvisor should expose this IIRC.

PS: You can also run the exporter in the network namespace, then it will see only that namespaces devices. Something like ip netns exec test node_exporter ...

[bluecmd]

Either we support slicing all metrics by namespace

That's what I suggested.

ip netns exec test node_exporter

That assumes that the node_exporter would be reachable. I have namespaces that their only routes are internal to the network they serve, no external connectivity. So it would be quite painful to make that work.

Too bad that you don't see this as a feature worth having, I really think it would be a good thing.
Again, cAdvisor doesn't do namespaces but as far as I know only cgroups. Namespaces are just totally different.

[discordianfish]

Well, so far the node-exporter always exposes metrics about all of the host, no per process/container/namespace/cgroup stuff.

If we want to change that, we should discuss it in a dedicated issue first and implement it for all namespaces (network, mounts, ipc, pid etc), but as I understand it, that's what cadvisor is about.

Have you tried using it? Just did and it exposes network metrics:

# HELP container_network_receive_bytes_total Cumulative count of bytes received
# TYPE container_network_receive_bytes_total counter
container_network_receive_bytes_total{id="/",interface="wlp58s0"} 1.688142346e+09
container_network_receive_bytes_total{id="/docker/1bb3210c7d854569fb45ba925a758d726d420698f6b1ec3b6c85a359b3013d21",image="google/cadvisor:latest",interface="eth0",name="cadvisor"} 113984
container_network_receive_bytes_total{id="/docker/6fcf1a3e409bbd80ceff224f82f62993f8d7a5a5fab16e07aa4e6b830b6e558b",image="ubuntu",interface="eth0",name="happy_banach"} 14184

If you have a compelling reason why you want to have the node-exporter expose these, feel free to open a new issue about supporting namespace metrics in general.

[bluecmd]

As the issue is quite old I don't remember the details, but for example in your case above you're using docker, which uses cgroups AND namespaces. What you get there (IIRC) is the cgroups numbers, not the namespace counters.

Try doing what I stated in my first comment at the top. If you at that time see dummy0 in cAdvisor I'm happy, but I'm pretty sure that isn't the case. If I were in a place where I could make sure of it, I would - but I trust past me.

Well, so far the node-exporter always exposes metrics about all of the host,

No, that's my point: You're only exporting part of the host metrics - the part that is in the current namespace. All other things running on the machine are invisible

[discordianfish]

What you get there (IIRC) is the cgroups numbers, not the namespace counters.

Hrm that's quite possible.

No, that's my point: You're only exporting part of the host metrics - the part that is in the current namespace. All other things running on the machine are invisible

You're right, I assumed that the interfaces hidden from the exporter are always sub-interfaces but that doesn't need to be the case. Now I'm wondering though if we have similar issue with other namespaces. Like does node_filefd_allocated include fds allocated in a IPC namespace?

Either way, this should be fixed somehow.

[cirocosta]

Hey, maybe I'm missing something (I'm pretty new to networking) but my 2cents is that I think there's a great use case for docker folks:

• docker service create --publish target=80,published=8080 nginx ... exposes nginx at ingress
• IPVS is set to route requests to nginx when you hit <iface>:8080
• the IPVS specifics are all hidden under the ingress_sbox netns such that:

#cat /proc/net/ip_vs_stats
   Total Incoming Outgoing         Incoming         Outgoing
   Conns  Packets  Packets            Bytes            Bytes
       0        0        0                0                0

 Conns/s   Pkts/s   Pkts/s          Bytes/s          Bytes/s
       0        0        0                0                0


# ip netns exec ingress_sbox cat /proc/net/ip_vs_stats
   Total Incoming Outgoing         Incoming         Outgoing
   Conns  Packets  Packets            Bytes            Bytes
    2925    24FA2    25B48         107DA9A5          8A30D89

 Conns/s   Pkts/s   Pkts/s          Bytes/s          Bytes/s
       1        5        4              470              2E5

having namespace awareness would bring these stats.

(I assume that you can't just ip netns exec ingress_sbox node_exporter ... as I suppose it wouldn't route the port to outside).

Please correct me if I'm wrong!

Thx!

[discordianfish]

Yes, we should surface this somehow and are open to accept PRs. Just nobody had time to look into this yet.

[discordianfish]

Related #66

[frittentheke]

@SuperQ @discordianfish I was wondering if you would still consider implementing network namespace support or even accept PRs for this?

I just ran into this wile looking at node_exporter to help debugging an issue with OpenStack Neutron (virtual networking component) which has one driver, Linuxbridge, which makes heavy use of network namespaces to provide tenant networks, routers and other network services (https://docs.openstack.org/neutron/latest/admin/deploy-lb-selfservice.html#architecture).
But as other have also mentioned network namespaces are also heavily used with containers and container networks.
Having support for network namespaces in the node_exporter would be a great addition to also collect data from those, otherwise invisible interfaces.

Just having a switch to enable exporting interfaces with their namespace as an additional label and maybe an include/exlude filter on namespace names as usually automation creates namespaces with a certain pattern and one might only want a certain set of network namespaces exported (just like iftype or fstype filter)

As for the implementation I suppose either an addition to https://github.com/prometheus/procfs/blob/master/sysfs/net_class.go would be required to iterate and switch through all (selected) network namespaces when accessing /sys/class/net as the kernel does use the network namespace of the calling process to chose which devices to show, see https://github.com/torvalds/linux/blob/28c20cc73b9cc4288c86c2a3fc62af4087de4b19/fs/sysfs/mount.c#L35 or for a more comprehensive description https://unix.stackexchange.com/a/457384.

Another approach would be to allow the selection of the network namespace when calling NewFS https://github.com/prometheus/procfs/blob/5162bec877a860b5ff140b5d13db31ebb0643dd3/sysfs/fs.go#L37 and do the iteration within the node_exporter.

[discordianfish]

@frittentheke I'm not sure if we want full support for namespaces, only support for network namespaces or just get host wide metrics. I'm open to discuss any option but still want to avoid too much overlap with cadvisor. There is a case to be made though to say these metrics should come from node-exporter and not cadvisor. Either way, this needs some more discussion. Would love if some more kubernetes people would chime in as well.

[frittentheke]

@discordianfish thanks for the quick response. Yes, there should be no explosive duplication in metrics I fully agree.

CAdvisor does look at (network) namespaces from a container (read: namespaced and cgrouped process) point of view though and identifies things this way: https://github.com/google/cadvisor/blob/ef7e64f9efab1257e297d7af339e94bb016cf221/container/libcontainer/handler.go#L71

My case for having namespace support was more for cases (i.e. with OpenStack Neutron) when the linux kernel IP stack functionality (tunneling, routing, iptables for filtering / NAT, IPSEC, ...) was used via network namespaces. In this case there is no container or no process involved, just the linux kernel doing it's thing. As I wrote, I am convinced that having a config switch and a namespace include / exclude filter is a must in any case, to not have node_exporter gather data on unwanted interfaces.

[discordianfish]

Yeah I get that, I just think ideally there would be one standard way to get cgroup and namespaces metrics.

So if we decide to expose namespaced network metrics, we should probably also consider exposing other namespaced metrics. e.g mounts and processes.

[yusufgungor]

We have the same situation on openstack site like @frittentheke

[SuperQ]

To echo the same comment I made on #2264. I'm not sure this is going to be possible in the node_exporter. I'm pretty sure "nsenter" is going to require root. Which we do not want to have in the node_exporter.