[Feature]: Plugin output with more details or customised

[wattebausch]

Describe the feature request

We alert our developers via Icinga. Via e-mail or a messager. Unfortunately, the developers can't get much information from the message and have to log in to Prometheus. A big advantage would be if we could get more or info or customised like ‘namespace’ or ‘pod’. The info on Instance: [IP:Port] is not necessary from our point of view. The cluster name is covered by the host from Icinga.

Output from check_prometheus

[CRITICAL] - 3 Alerts: 1 Firing - 2 Pending - 0 Inactive
\_ [WARNING] [FluxKustomizationNotApplied] - Job: [kube-state-metrics] on Instance: [10.42.0.58:8080] is pending - value: 1.00
\_ [CRITICAL] [InfoInhibitor] on Instance: [10.XX.XX.XX:10250] is firing - value: 1.00
\_ [WARNING] [CPUThrottlingHigh] on Instance: [10.XX.XX.175:10250] is pending - value: 0.38

Output from Prometheus

FluxKustomizationNotApplied
gotk_resource_info{container="kube-state-metrics", customresource_group="kustomize.toolkit.fluxcd.io", customresource_kind="Kustomization", customresource_version="v1", endpoint="http", exported_namespace="flux-system", instance="10.42.1.10:8080", job="kube-state-metrics", name="apps", namespace="infra", pod="infra-metrics-kube-state-metrics-6d9c96cf74-95ggq", ready="Unknown", revision="main@sha1:40e136d9cd5c81d45bc9293751778a79a9cb795c", service="infra-metrics-kube-state-metrics", source_name="flux-system"} 1

InfoInhibitor
ALERTS{alertname="CPUThrottlingHigh", alertstate="pending", container="container", instance="10.XX.XX.175:10250", namespace="apps-dev", pod="r-deployment-78d78f556d-cwsl9", service="infra-metrics-kube-prometh-kubelet", severity="info"} 1

CPUThrottlingHigh
{container="container", instance="10.XX.XX.XX:10250", namespace="apps-dev", pod="r-deployment-78d78f556d-cwsl9", service="infra-metrics-kube-prometh-kubelet"} 0.38

what do you think about this? how do you deal with messages?
Greetings and thanks
Marcel

[martialblog]

Hi,

would make sense to have additional information in the output.

However, not sure yet if and where we can get the data from the Prometheus API.
The /api/v1/alerts returns the Labels with the summary and description of the alert rule. But since we check the rules the actual data might not be included.

Example when I add the summary and description to the output:

[CRITICAL] - 4 Alerts: 1 Firing - 0 Pending - 3 Inactive
\_ [OK] [PrometheusTargetMissing] is inactive - Prometheus target missing (instance {{ $labels.instance }}) - A Prometheus target has disappeared. An exporter might be crashed.
  VALUE = {{ $value }}
  LABELS = {{ $labels }}
\_ [CRITICAL] [PrometheusAlertmanagerJobMissing] - Job: [alertmanager] is firing - value: 1.00
\_ [OK] [HostOutOfMemory] is inactive - Host out of memory (instance {{ $labels.instance }}) - Node memory is filling up (< 10% left)
  VALUE = {{ $value }}
  LABELS = {{ $labels }}

But I just did some quick testing, there might be a way.

[martialblog]

Hi, feel free to test this: https://github.com/NETWAYS/check_prometheus/releases/tag/v0.4.0-rc1