New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Add systemd (unit) receiver #37169

Draft

pieterlexis-tomtom wants to merge 6 commits into open-telemetry:main from pieterlexis-tomtom:systemd-receiver

pieterlexis-tomtom commented Jan 13, 2025

Description

This PR adds an initial systemd unit receiver, as requested in #33532.
I would describe the receiver a very early alpha, but usable.

Comments are welcome.

Link to tracking issue

Fixes #33532

Testing

This receiver has had some light testing and for this config:

---
receivers:
  systemd:
    units:
      - named.service
      - cups.service

exporters:
  debug:
    verbosity: detailed
  prometheus:
    endpoint: 127.0.0.1:8080
    resource_to_telemetry_conversion:
      enabled: true

service:
  telemetry:
    logs:
      level: debug
  pipelines:
    metrics:
      receivers:
        - systemd
      exporters:
        - prometheus
        - debug

it outputs this as metrics:

# HELP systemd_failed_jobs_total How many jobs have ever failed in total
# TYPE systemd_failed_jobs_total counter
systemd_failed_jobs_total 0
# HELP systemd_installed_jobs_total How many jobs have ever been queued in total
# TYPE systemd_installed_jobs_total counter
systemd_installed_jobs_total 2136
# HELP systemd_jobs How many jobs are currently queued
# TYPE systemd_jobs gauge
systemd_jobs 0
# HELP systemd_system_state The current state of the service manager
# TYPE systemd_system_state gauge
systemd_system_state{architecture="x86-64",system_state="running",systemd_version="249.11-0ubuntu3.12",virtualization=""} 3
# HELP systemd_unit_errno The errno (exit code) of the last error/exit
# TYPE systemd_unit_errno gauge
systemd_unit_errno{systemd_unit_name="cups.service"} 0
systemd_unit_errno{systemd_unit_name="named.service"} 0
# HELP systemd_unit_restarts_total Amount of time the unit was restarted this boot
# TYPE systemd_unit_restarts_total counter
systemd_unit_restarts_total{systemd_unit_name="cups.service"} 0
systemd_unit_restarts_total{systemd_unit_name="named.service"} 0
# HELP systemd_unit_state The full state of the unit. The gauge value is the sub state, but all states (load, active, sub) are exposed as attributes
# TYPE systemd_unit_state gauge
systemd_unit_state{active_state="active",load_state="loaded",sub_state="running",systemd_unit_name="cups.service"} 6
systemd_unit_state{active_state="active",load_state="loaded",sub_state="running",systemd_unit_name="named.service"} 6

Apart from that, there are several go tests of the scraper.

pieterlexis-tomtom added 6 commits

January 13, 2025 16:22


          [receiver/systemd] feat: Initial systemd receiver

8b0627e


          [receiver/systemd] chore: make client pluggable

1da17bb


          [receivers/systemd] feat: Add systemd-wide metrics

56d39cb

This also refactors the `scrape` function somewhat.


          [receivers/systemd] tests: Add tests

f1a51a8


          [receivers/systemd] feat: use a function to create the client

433656a


          [receivers/systemd] feat: Ensure an empty client is used on !linux

d201127

Signed-off-by: Pieter Lexis <pieter.lexis@tomtom.com>

linux-foundation-easycla bot commented Jan 13, 2025

❌ - login: @pieterlexis-tomtom / name: Pieter Lexis (TomTom) . The commit (8b0627e, 1da17bb, f1a51a8, 433656a, 56d39cb, d201127) is not authorized under a signed CLA. Please click here to be authorized. For further assistance with EasyCLA, please submit a support request ticket.

github-actions bot added cmd/otelcontribcol receiver/systemd labels

atoulme reviewed

View reviewed changes

receiver/systemdreceiver/metadata.yaml

@@ @@ -4,8 +4,172 @@ status: @@
                 class: receiver
                 stability:
                   development: [metrics]
-                distributions: []
+                distributions: [contrib]

Contributor

atoulme Jan 18, 2025

no it's not in contrib yet, please remove for now

atoulme reviewed

View reviewed changes

receiver/systemdreceiver/metadata.yaml

+                    - active_state
+                    - sub_state
+                systemd.unit.restarts:
+                  description: Amount of time the unit was restarted this boot

Contributor

atoulme Jan 18, 2025

the unit {restarts} doesn't match the description

atoulme reviewed

View reviewed changes

receiver/systemdreceiver/metadata.yaml

+                  unit: "{system_state}"
+                  attributes:
+                    - systemd_version
+                    - system_state

Contributor

atoulme Jan 18, 2025

no that doesn't really work, you are sending a metric but only care for its attribute. You cannot make the unit match the attribute name.

The value of the metric must itself be meaningful, or just send 1 if you're just going to create a time series with the attributes.

atoulme reviewed

View reviewed changes

receiver/systemdreceiver/metadata.yaml

+                    - systemd_version
+                    - system_state
+                    - architecture
+                    - virtualization

Contributor

atoulme Jan 18, 2025

I would disable those 2 attributes by default

atoulme reviewed

View reviewed changes

receiver/systemdreceiver/metadata.yaml

+                  gauge:
+                    value_type: int
+                  unit: "{jobs}"
+                systemd.installed_jobs:

Contributor

atoulme Jan 18, 2025

is installed jobs the right name? How about systemd.jobs.total ?

atoulme reviewed

View reviewed changes

receiver/systemdreceiver/metadata.yaml

+                    monotonic: true
+                    value_type: int
+                  unit: "{jobs}"
+                systemd.failed_jobs:

Contributor

atoulme Jan 18, 2025

Suggested change

      
              systemd.failed_jobs:
          
              systemd.jobs.failed:

atoulme reviewed

View reviewed changes

receiver/systemdreceiver/metadata.yaml

+                  enabled: false
+                  gauge:
+                    value_type: int
+                  unit: "{active_state}"

Contributor

atoulme Jan 18, 2025

What value are we reporting?

atoulme reviewed

View reviewed changes

receiver/systemdreceiver/metadata.yaml

+                  attributes:
+                    - load_state
+                    - active_state
+                    - sub_state

Contributor

atoulme Jan 18, 2025

this might create high cardinality as your metric will have changing attribute values, creating multiple time series.

Author

pieterlexis-tomtom commented Jan 22, 2025

@atoulme I'll fix the issues mentioned somewhere the coming week

Contributor

github-actions bot commented Feb 6, 2025

This PR was marked stale due to lack of activity. It will be closed in 14 days.

github-actions bot added the Stale label

Author

pieterlexis-tomtom commented Feb 11, 2025

@atoulme I have some problems with rebasing. Is there a guide somewhere where there are some descriptions on how to fix the gomod files, builder-config etc.?

github-actions bot removed the Stale label

Contributor

github-actions bot commented Feb 27, 2025

This PR was marked stale due to lack of activity. It will be closed in 14 days.

github-actions bot added the Stale label

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cmd/otelcontribcol receiver/systemd Stale