Add exporter for the overrides

[codesome]

The exporter would expose the overrides as Prometheus metrics which can be then displayed on a dashboard.

I already have it implemented. Would it make sense to have a separate repo under cortexproject for this exporter?

[pstibrany]

Would it make sense to make it part of Cortex metrics?

[pracucci]

Also related to #1433.

[codesome]

Would it make sense to make it part of Cortex metrics?

Would be a separate module in Cortex? (because we don't want it to be exposed by >1 pod)

[pstibrany]

Would be a separate module in Cortex? (because we don't want it to be exposed by >1 pod)

Good point. Separate module would work in my opinion. Even if it is an extra pod, it would be tiny, and doing no work other than reloading config and publishing it via metrics. I think that's no different than having extra exporter. And in single-binary mode, it would just be part of "big" Cortex.

[pracucci]

And in single-binary mode, it would just be part of "big" Cortex.

If we run it in the -target=all, doesn't this introduce the high cardinality issue which prevented us from exporting these metrics directly from Cortex? Even in a small 10 nodes cluster, it will generate 10x series which I'm not sure it's what we want.

[pstibrany]

If we run it in the -target=all, doesn't this introduce the high cardinality issue which prevented us from exporting these metrics directly from Cortex? Even in a small 10 nodes cluster, it will generate 10x series which I'm not sure it's what we want.

You're right. I forgot that we encourage people to use multiple single-binary instances.

[weeco]

The number of tenants is your major concern what causes the high numbers of reported metric serieses right? What about an option where users can configure in what tenants' metrics the user is interested in? All tenants which aren't part of this list will be grouped together as "other_tenants" or so?

[simonswine]

Another solution I could think of would be to expose the limits under a completely different path like /metrics/limits by default and then users can decide to scrape it through a config that would only scrape it once per cluster. (Like a kubernetes role: service discovery rule)

[pracucci]

Another solution I could think of would be to expose the limits under a completely different path like /metrics/limits by default and then users can decide to scrape it through a config that would only scrape it once per cluster.

I'm not sure it solves the problem. To conditionally expose these metrics we don't need a separate endpoint, but just a configuration option. However, if you enable it and these metrics are exposed by every single Cortex service then in a large cluster you will have very high cardinality metrics. To solve this, a solution would be creating a new target (which is one of the proposed solution).

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[jtlisi]

still valid

Any thoughts on closing this issue since an external exporter for this purpose exists here grafana/cortex-tools#91?