[inputs.systemd_units] cpu usage goes through the roof!

[electrofloat]

Relevant telegraf.conf

[[inputs.systemd_units]]

Logs from Telegraf

-

System info

Ubuntu 22.04

Docker

No response

Steps to reproduce

Start telegraf with [inputs.systemd_units] used.

Expected behavior

Same cpu usage as before.

Actual behavior

With the new implementation of systemd_units, every time telegraf runs (which is every 10s), a systemd process eats a lot of cpu.

Attached 2 picture of grafana graphs, I think you can guess when I upgraded to 1.30.1. I've just reverted it back again to 1.29.5-1.

Additional info

No response

[srebhan]

@electrofloat thanks for reporting the issue! During the rewrite we also added some features like reporting disabled/unloaded units and my guess would be that this causes the issue. Could you assist in debugging the issue by testing some PR binaries?

[electrofloat]

@srebhan I can try, yes.

[srebhan]

@electrofloat thanks! Can you please test the binary in PR #15108 available after CI finished the tests?!? Please set the new option loaded_only to true and let me know if this helps!

[electrofloat]

It is better. Instead of about 100%, It only causes the PID1 systemd process to use about 15% cpu every 10 seconds, but v1.29.5-1 uses 0%. (I've just checked)

[srebhan]

@electrofloat I added another optimization, can you please test the binary in the PR when CI finished the tests!? Let me know if this reduces the load further...

[electrofloat]

It seems much better now. Seems the systemd process stays at 0%, with the loaded_only true option. I haven't tested whether it still provides the exact same metrics as before or not, only tested the cpu usage.

Also.. shouldn't the new option default to true? So that it works exactly like before the rewrite, and people don't have to change anything in the config to stay with the previous (before rewrite) behavior?

[srebhan]

@electrofloat will discuss it with the team as unfortunately we released 1.30. with the behavior being to also report non-loaded units...

[srebhan]

@electrofloat we decided to revert to the pre-v1.30.0 behavior and I adapted the PR accordingly. Could you please test it again, just to be sure!? Please note that the option now is inverted and no longer called loaded_only but collect_disabled_units.

[electrofloat]

Yes, it seems to be working as expected now, without any config change.