QueryFrontend error cannot be parsed correctly in Grafana

[yeya24]

Describe the bug
When query frontend throws error, Grafana will show error message like below.

readObjectStart: expect { or n, but found z, error found in #1 byte of ...|zero or neg|..., bigger context ...|zero or negative query resolution step widths are n|...

This is not very readable and not user friendly. The issue is that Grafana expects errors to be returned with Prometheus' Error format. However, query frontend doesn't format error using the Prometheus error response format, it just returns the raw error string like https://github.com/cortexproject/cortex/blob/master/pkg/querier/tripperware/queryrange/query_range.go#L183.

{
  "status": "success" | "error",
  "data": <data>,

  // Only set if status is "error". The data field may still hold
  // additional data.
  "errorType": "<string>",
  "error": "<string>",

  // Only if there were warnings while executing the request.
  // There will still be data in the data field.
  "warnings": ["<string>"]
}

Same issue happened at past as well, like #5260. Now we always forwards the request to querier to return error and this is very inefficient. Also there are errors happened in QueryFrontend only and returning to querier won't work.

We should format the error response correctly in query frontend, like Prometheus does.

[alvinlin123]

I think we should fix this for 1.16 release. Let me know if there are other thoughts.

[yeya24]

Hi @dogukanteber, it is easy to reproduce this issue. Steps are:

1. Have a Cortex cluster running. It would be good to use the latest master version
2. Have a Grafana pointing to Cortex query frontend component
3. Run some queries in Grafana and you are able to see these errors. Depends on different queries you run, you might be able to see different error messages so that's might be the reason you couldn't reproduce it yourself.

I tried up[30d:1s] which corresponds to the error we threw here https://github.com/cortexproject/cortex/blob/master/pkg/querier/tripperware/subquery.go#L38.

If you are using curl directly, you can also try to reproduce this error https://github.com/cortexproject/cortex/blob/master/pkg/querier/tripperware/queryrange/query_range.go#L202 with a very small step size.

[yeya24]

Let's move this to release 1.17. Unfortunately we couldn't make it for 1.16

[yeya24]

Not sure if it is the behavior got changed at newer version of Grafana. I am testing with Grafana v10.1.5.

Looks like it doesn't display error at this version.

When doing it via postman it is a 400 error.

We should work on this ASAP. Actually we already have required types defined in Ruler https://github.com/cortexproject/cortex/blob/master/pkg/ruler/api.go#L104. We should start using it at QFE.