Ingesters should not grow without bound

[awh]

The ingesters would be more manageable if they could refuse service/exert backpressure instead of growing to the point where they are OOM killed when they can't flush or are exposed to a traffic spike. Ideally they would accept a configuration parameter to control this behaviour, which we would use in conjunction with a slightly higher memory request/limit in k8s.

[cboggs]

Out of curiosity, what circumstances are leading to this behavior (OOM killing ingesters)?

I'd imagine some sort of slow-down on the backing store would be the most likely culprit, but I'd love to hear specific situations if you have any. :-)

[bboreham]

Some info on store slow-downs at #733, but we also get cases where the incoming data blows up - e.g. someone starts sending loads more timeseries.

I'm not clear that "backpressure" is available - if we slow down, the sending Prometheus will queue the data and eventually OOM itself. We could discard data in the ingester; we could error back to the sender so it will discard it.

[tomwilkie]

if we slow down, the sending Prometheus will queue the data and eventually OOM itself.

Prometheus queue is bounded, so the sending prometheus with start dropping samples.

We could discard data in the ingester; we could error back to the sender so it will discard it.

There is already a limiter on sample/s per user https://github.com/weaveworks/cortex/blob/6191d41c5a99e6f89fbd1877afc8805ab0c3ffbd/pkg/distributor/distributor.go#L298, but its not coordinated between distributors so it "scales" up with the number of replicas. #255 is for that. The downstream prometheus should understand and backoff: https://github.com/prometheus/prometheus/blob/master/storage/remote/queue_manager.go#L495

Re: bounded queue, I want to make the prometheus remote write code tail the TSDB WAL so we can "buffer" against this kind of problem client side much more robustly, and support usecases with spotty internet connections.

[bboreham]

Samples/sec needn't be a problem, so long as the samples compress well. Number of timeseries is a problem, since they get 1K plus metadata even if you only send one sample. I though there was a limit for that but also I think it isn't entirely effective. Again, need better tools to understand the problem.

[bboreham]

There are also DefaultMaxSeriesPerMetric = 50000 and DefaultMaxSeriesPerUser = 5000000; the latter is per-ingester. Empirically each timeseries costs about 4K of Go heap, so letting one user have 20GB is brave.

[bboreham]

It occurred to me we could inspect Go's runtime.MemStats.HeapAlloc and refuse to create a new series if that number is over a configurable limit, e.g. 10GB. This would get closer to the desired behaviour and protect against OOM-kills.

[awh]

Maybe relevant: golang/go#16843

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.