metrics.go support for metadata queries(labels and series)

[kavirajk]

What this PR does / why we need it:
We use metrics.go (a convention) to record some useful statistics about each of the instance and range queries.

So that we can query something like
{foo="bar"}|= "metrics.go" |logfmt|latency=slow

Where latency comes from statistics recorded in pkg/logql/metric.go.

This PR add support those experience for series and labels API as well.

Examples:

1. Labels API (names)

curl -G -s  "http://127.0.0.1:3100/loki/api/v1/label"

Generates following log lines via metrics.go

level=info ts=2022-05-03T19:51:37.417252626Z caller=metrics.go:170 component=frontend org_id=145265 traceID=5bdfb2e229890ea2 latency=fast query_type=labels length=1h0m0s duration=65.140762ms status=200 label= throughput=0B total_bytes=0B total_entries=14

2. Labels API (values)

curl -G -s  "http://127.0.0.1:3100/loki/api/v1/label/container/values"

Generates following log lines via metrics.go

level=info ts=2022-05-03T19:50:07.710590524Z caller=metrics.go:170 component=frontend org_id=145265 traceID=11349be1207cbdba latency=fast query_type=labels length=1h0m0s duration=11.239903ms status=200 label=container throughput=0B total_bytes=0B total_entries=19

3. Series API

curl -G -s  "http://127.0.0.1:3100/loki/api/v1/series" --data-urlencode 'match[]={app="loki"}'

Generates following log lines via metrics.go

level=info ts=2022-05-03T19:51:40.272480868Z caller=metrics.go:215 component=frontend org_id=145265 traceID=396c8cdce34bb53e latency=fast query_type=series length=1h0m0s duration=85.968193ms status=200 match="{app=\"loki\"}" throughput=0B total_bytes=0B total_entries=249

Which issue(s) this PR fixes:
Fixes #4503

Special notes for your reviewer:

Checklist

• Documentation added
• Tests updated
• Add an entry in the CHANGELOG.md about the changes.

Future Work

• Get rid of latency=slow|fast label from metric.go entries.
• deprecate and remove returned_lines when query_type=filtered. Because now we have total_entries common for all the query_type.

[cstyan]

Would we want any additional stats fields for these queries? the # of results for example

[kavirajk]

Would we want any additional stats fields for these queries? the # of results for example

Good point. I think that would really be helpful. Let me try adding it.

[trevorwhitney]

Looks great!

[cstyan]

LGTM as well, there's just a lint issue it looks like.

Thanks for implementing this Kavi. I learned a thing or two about how the stats collection is implemented while reviewing.

[kavirajk]

I'm facing one weird panic during some series API call. Investigating.

[owen-d]

The approach is looking good. As you mentioned, we'll need to find/fix the panic and do some linting.

[kavirajk]

This is how the metrics.go entries looks like for following metadata queries.(tested it on some internal loki clusters)

1. series
2. label names
3. label values (notice a value for label in metrics.go)

[cstyan]

can we include a link to the docs for this endpoint? https://grafana.com/docs/loki/latest/api/#get-lokiapiv1labelnamevalues

[cyriltovena]

LGTM

[owen-d]

LGTM