Skip to content

Loki Grafana Alerts are not working via Prometheus datasource #2020

Closed
@cyriltovena

Description

What happened:
In Loki we don't have yet an alerting storing, so we took advantages of the similarities between Loki and Prometheus API to create Grafana Alert on top of Loki but added as a Prometheus datasource, see below.

image

We even have a video explaining how to do that https://www.youtube.com/watch?v=GdgX46KwKqo

Now it seems that since 6.6 alert are failing to be evaluated, although graphing still works see below:

image

It seems that something is now parsing the PromQL query and since Loki has some differences it fails. Although the query is actually the same as PromQL (sum(rate({cluster="dev-us-central1", job="loki-boltdb-shipper/distributor", level!~"debug|info"}[5m])) by (level)).

Response from the test endpoint:

{"firing":true,"state":"pending","conditionEvals":" = true","timeMs":"12.647ms","error":"tsdb.HandleRequest() error bad_response: readObjectStart: expect { or n, but found p, error found in grafana/grafana#1 byte of ...|parse error|..., bigger context ...|parse error : syntax error: unexpected $end|...","logs":[{"message":"Condition[0]: Query","data":{"from":1588256991975,"queries":[{"refId":"A","model":{"expr":"sum(rate({cluster=\"dev-us-central1\", job=\"loki-boltdb-shipper/distributor\", level!~\"debug|info\"}[5m])) by (level)","interval":"","intervalFactor":3,"legendFormat":"{{level}}","refId":"A"},"datasource":{"id":3,"name":"Loki-Prometheus"},"maxDataPoints":0,"intervalMs":0}],"to":1588257291975}}]}

What you expected to happen:

Alert to still works with Loki Prometheus datasource.

How to reproduce it (as minimally and precisely as possible):

See the video.

Environment:

  • Grafana version: seems to break from 6.6 and forward
  • Data source type & version: Loki as Prometheus datasource

/cc @aocenas

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions