Ingestion latency spikes when discarding metrics due to limits

[gouthamve]

We're seeing consistent spikes in ingester /cortex.Ingester/Push latency when there is an uptick in the samples dropped due to limits.

Needs to be investigated further.

[bboreham]

@gouthamve do you see any improvement after #1497 ?

[bboreham]

I have a theory what causes this.

For each rejection, userState.getSeries() calls httpgrpc.ErrorFromHTTPResponse(), which does a lot of memory-allocating, then returns it to Ingester.Push() which calls httpgrpc.HTTPResponseFromError() which does even more memory-allocating. Then Push() throws it all away except for the last error.

I recommend we use an error object that just holds a reference to the labels until called upon to format itself.