Add metrics to properly monitor validation limits

[weeco]

Use case

When running Cortex for multiple tenants (e. g. multiple departments within a larger company) one still wants to set reasonable validation limits to ensure a healthy Cortex cluster.

The problem

Due to the fact that all limits multiply with the number of distributor/ingester replicas it's not easy to understand the current validation limits. Additionally overriding the limits for specific users makes it more complex.

It would be very nice if I could monitor all validation limits (and their usage) per user, so that I know which user is close to it's validation limits. This way I can reach out to users ask them if they need higher limits, before they run into rejections which would cause lost metrics.

Proposed solution

We could add prometheus metricses for all limits which we can easily sum in a Grafana dashboard. This way we know the exact validation limits (e. g. total ingestion rate and burst size per user). This might be a problem for those who run Cortex for hundreds of users I assume as each user means a new metric series.

[bboreham]

Sounds like the metric cortex_discarded_samples_total, which has labels user and reason

[weeco]

That's indeed helpful, however when samples are discarded they are lost, I'd like to prevent that and know beforehand if we are anything near the limits, to prevent loss of metricses. Does that make sense?

[pracucci]

Due to the fact that all limits multiply with the number of distributor/ingester replicas it's not easy to understand the current validation limits.

We currently support some global limits which solve this specific problem for some limits. Global limits are supported for:

• Ingestion rate
• Max series per user
• Max series per metric

Please reference to the limits doc for more information about how it works.

@weeco Does this solve your original issue or not?

[weeco]

@pracucci No I don't think so.

The problem is that I can not comfortably monitor how close each tenant is to the set limits. I only set limits to make sure that accidential excessive writes do not take down the whole cluster. Thus if I know a tenant is close to the configured limits I can raise the limits before metrics will be rejected due to exceeding the limits.

[pracucci]

We could add prometheus metricses for all limits which we can easily sum in a Grafana dashboard

At Grafana Labs we've built a custom exporter which reads the Cortex config file (limits and overrides) and expose them. The reason why we did it through a custom exporter instead of exposing these metrics from Cortex is to keep the cardinality lower (because in a large cluster, if any single Cortex node exposes the same limits, the cardinality could easily explode).

The custom exporter is not OSS yet, but I will circle back internally to make it happen.

[weeco]

is to keep the cardinality lower (because in a large cluster, if any single Cortex node exposes the same limits

Couldn't we just make it an option then where users can choose to enable or disable it? I think it's desirable to avoid introducing further services