Make severity configurable in PrometheusRules

[sebastiangaiser]

Is your feature request related to a problem? Please describe.
Currently the severity is hardcoded in the PrometheusRules like here

Describe the solution you'd like
It would be great to have them configurable via the Logging CR e.g.:

spec:
  fluentd:
    metrics:
      prometheusRules: true
      severity: my-custom-severity
  fluentbit:
    metrics:
      prometheusRules: true
      severity: my-custom-severity
  syslogNG:
    metrics:
      prometheusRules: true
      severity: my-custom-severity

Describe alternatives you've considered
My current workaround is to disable the PrometheusRules and deploy them in the way I want

Additional context
Logging Operator docs: https://kube-logging.dev/docs/operation/alerting/

[sebastiangaiser]

@pepov sorry to ping you but could you please mind have a look :D

[pepov]

Good idea, but I don't have any free cycles currently. We are looking for contributors to take on "good first issues" like this!

[aditya7302]

@pepov I would like to work on this issue. Can you assign it to me ?

[pepov]

@aditya7302 sure, thanks!

[aditya7302]

@pepov is this crd is where severity metric should be added to make it configurable ?

[pepov]

@aditya7302 that is generated from the go code. I recommend to check the Metrics type that could be extended with a severity field and applied everywhere where a prometheusrule is defined, for example here

By looking at the code, my problem is, that using this method we would have to use the same severity for all the prometheusrules that are defined for a single metrics endpoint. For example in case of fluentbit there are two rules defined, one with warning and one with critical severity. How would the proposed severity setting work in this case? 🤔