Rewrite host_metrics

[Voldemat]

There are a lot of pending issues right now related to problems with host_metrics. Main issues arise from statically defined paths to /proc and /sys files in heim package(unmaintained). My proposition is to basically copy any necessary code from heim into vector and allow to configure paths for each metric like filesystem, disk, network etc. I`m gonna link all open issues I find related to host_metrics for reference.

#23412 #22487 #21598 #21389 #20016 #6924 #7538 #6923 #5038

[Voldemat]

First step would be to just basically copy all needed heim code into repository and remove this dependency. After that I would add additional configuration options for all paths that we use to get metrics information from /proc and /sys filesystem, because if I vector is run in a container, it should request some info not from root like "/proc" but from "/proc/1/" to get actual host info, trying to just set PROCFS_ROOT to /proc/1 doesn`t work and throws errors, probably because not all metrics can be retrieved from this subpath.

[pront]

Hi @Voldemat, we should be able to completely remove the heim dependency.

Related: #20414

[Voldemat]

Hi @pront , I checked sysinfo crate source code and it also uses hardcoded paths for /proc filesystem so it wont solve "vector in docker container" problem, but it is at least maintained so I guess it is possible to fix it upstream.

[0323pin]

Hi!
Any news on this? I've tried to compile vector on NetBSD, just to hit compile_error!("Unsupported OS") while compiling your fork of heim.

Upstream heim is abandonware so, I came here looking for a future plan and found this and some of the linked issues.

What's the roadmap looking like? I see 3 possible paths:

1. The one being discussed in this issue
2. Shift to sysinfo which currently doesn't support NetBSD but has a wip branch ongoing
3. Take over heim and become the main upstream. Something like starship did with rust-battery as a fork of https://github.com/svartalf/rust-battery (oh, coincidence ... svartalf again). Then people could contribute OS support to an active project which, doesn't just pile-up PRs that will bit rot overtime.

Tagging @he32 as he also has some interest in this. Thanks for any pointers.

[thomasqueirozb]

I checked sysinfo crate source code and it also uses hardcoded paths for /proc filesystem so it wont solve "vector in docker container" problem

Shift to sysinfo which currently doesn't support NetBSD but has a wip branch ongoing

From the issue discussion it seems like sysinfo might be a good idea. Some patches may be needed to achieve all we need from it though.

It is also worth mentioning that, at the moment, we don't have plans to officially support any OS/architectures other than x86 Linux, x86 Windows and ARM MacOS

Take over heim and become the main upstream. Something like starship did with rust-battery as a fork of svartalf/rust-battery (oh, coincidence ... svartalf again). Then people could contribute OS support to an active project which, doesn't just pile-up PRs that will bit rot overtime.

If you or anyone else is interested in taking over and maintaining heim then I'd say go for it. Our team however doesn't have any plans of maintaining any dependencies or other projects at the moment.

Unfortunately a rewrite/update of host_metrics will not be prioritized in the foreseeable future because there are a quite few higher priority projects pending. However, we absolutely welcome community contributions and do our best to provide guidance and timely reviews for those. If you’re interested in contributing, we’d be happy to support you through the process.

[0323pin]

@thomasqueirozb Thank you for the pointers regarding the roadmap. I'll revisit this in the future and check which path you have chosen.

It is also worth mentioning that, at the moment, we don't have plans to officially support any OS/architectures other than x86 Linux, x86 Windows and ARM MacOS

For the record, this is only partially true. heim doesn't build on my Linux system either as heim only builds when using glibc, my Linux system is built using musl-libc.

[thomasqueirozb]

For the record, this is only partially true. heim doesn't build on my Linux system either as heim only builds when using glibc, my Linux system is built using musl-libc.

We do build and publish x86 Linux musl. You might be having an issue with your system. GitHub Actions runners probably have some config/setup that makes it so that isn't an issue. Feel free to open a separate issue with more information about this.

[mushrowan]

Hi! I'm interested in picking this issue up. I've been doing a bit of work today to get inodes working as this is kinda a deal breaker for the company I work at in using vector (currently we're supplementing vector with node_metrics to get that data, but obviously it's less than ideal)
just wanted to leave a note here in case someone is coincidentally working on the issue at the same time as me.