Add user(tenant) info in the metrics exposed by query frontend

[achilles42]

Currently, for the metrics which are exposed by query frontend does not have user(tenant) info in the labels, it gets really difficult to figure out which tenant's queries are getting stuck.

I would suggest, we should add user info on the following metrics-

1. query_frontend_queue_duration_seconds - Time spend by requests queued.
2. query_frontend_queue_length - number of queries in the queue.
3. frontend_query_range_duration_seconds - Total time spent in seconds doing query range requests.

Please let me know your thoughts on this. I would love to raise PR.

[csmarchbanks]

I think it definitely makes sense for queue_length to be per tenant so it is easy to see which tenant is backing up.

I am less convinced by the other metrics, but not ruling them out. An alternative would be slow query logging so it is also easier to see which queries are slow as well.

[achilles42]

@csmarchbanks Sure, Thanks for the comment.

For the query_frontend_queue_duration_seconds and frontend_query_range_duration_seconds , I agree that the slow query logs can give us exact info about which query is not performing as expected or taking more time.
But I am thinking from a consistency point of view where one metric exposes user_id and others don't. IMO, if we add the user_id info on other metrics, then the users can have cumulative details by doing sum(query_frontend_queue_duration_seconds{}) as well as per tenant. Please let me know WDYT.

For enabling a slow query, I am thinking of having a flag called log-queries-after(s) and print the queries in the logs which are taking more time than defined on log-queries-after.

[bboreham]

FWIW we use Jaeger tracing to see slow queries. We don't get that many, compared to everything else, so can sample a high percentage.

[gouthamve]

So, one of the reasons we are not keen on adding user-id to the other metrics is that they are histograms and things quickly add up with user-ids.

The log-queries-longer-than flag makes sense tbh. I see a lot of use-cases for it internally.

[gouthamve]

Closing this as we now have slow query logigng.