Ability to scrape filesystem host metrics from a container

[dmitryax]

Use case:

Scraping underlying host metrics from a container

Existing approach to scrape host metrics from container:

1. Mount host filesystem / path to some path inside container, e.g. /hostfs.
2. Setup env variables supported by gosutil to point it to different locations, e.g. HOST_PROC=/hostfs/proc

Problem description:

The described approach works for all hostmetrics receiver scrapers except for filesystem.

filesystem scraper workflow (assuming HOST_PROC=/hostfs/proc) is the following:

1. Fetch list of disc partitions with gopsutil/disk.Partitions: all the paths in the returned list are plain host paths, not prefixed with /hostfs.
2. Read usage stats of the partitions returned at the step 1 with gopsutil/disk.Usage using mount points paths: since the mount point paths are used as is (without /hostfx prefix), the function reads container partition stats instead of host partition stats. Obviously, most of the host mount points are not available in the container, and it results in throwing the following errors:

2021-10-23T16:39:03.988Z	error	scraperhelper/scrapercontroller.go:198	Error scraping metrics	{"kind": "receiver", "name": "hostmetrics", "error": "no such file or directory; no such file or directory; no such file or directory; no such file or directory; no such file or directory; no such file or directory; no such file or directory; no such file or directory; no such file or directory; no such file or directory; no such file or directory; no such file or directory; no such file or directory; no such file or directory; no such file or directory; no such file or directory; no such file or directory; no such file or directory; no such file or directory; no such file or directory; no such file or directory; no such file or directory", "scraper": "filesystem"}
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport
	go.opentelemetry.io/collector@v0.37.1-0.20211015233822-bd87fb628058/receiver/scraperhelper/scrapercontroller.go:198
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1
	go.opentelemetry.io/collector@v0.37.1-0.20211015233822-bd87fb628058/receiver/scraperhelper/scrapercontroller.go:173

The issue was uncovered by this change to goputils library shirou/gopsutil#1133. Before that gopsutil/disk.Partitions used to return container disk partitions, so it looked like it worked well, but it didn't.

Suggested solution:

Add new configuration option to hostmetrics receiver: root_path with / as default. The receiver behavior stays the same with default value, but if the root_path set to something else:

• Make sure that all the env variables needed for gopsutil are automatically set (only if not provided by user)
• Handle all the paths translation between gopsutil/disk.Partitions and gopsutil/disk.Usage calls in filesystem scraper to fix the described issue. We cannot expect users to always mount the whole / host path to the container, so any files misses should throw debug logs instead of errors.

[rmfitzpatrick]

@dmitryax does using HOST_PROC_MOUNTINFO introduced in shirou/gopsutil#1272 provide a mechanism for this that doesn't necessarily require changes to the scraper?

[dmitryax]

Yes looks like it does. We can use it as a workaround for now. It should fix metrics that don't depend on /proc/ in our distro. But other metrics using /proc/self instead of /proc/1 will be still reporting container value instead of host. I don't remember how critical is that. Probably should be close to accurate.

[dmitryax]

Please test it in kubernetes to make sure that no errors are thrown. Thanks for looking into that!

[rmfitzpatrick]

Oh my mistake, the disk.Usage() call will still fail since the host mount points need to still be coerced to ~/hostfs.