ingesters grind to a halt if > 1 week old samples are pushed

[ekimekim]

If any significant number of samples are pushed to cortex which map to an 'inactive' dynamo table (ie. not the current week), they can only be written at a global throughput limit of 1 chunk/second. In the meantime, all ingesters are trying and failing to write these bad chunks, resulting in a large queue and very little useful work being performed.

To fix this, we need to do one of the following:

• Reject samples that are too old
• Increase limits dynamically on older tables (an extension of the work in Enable write autoscaling for active DynamoDB tables #507)

[jml]

@aaron7 & @bboreham have been looking into this. I have not, so de-assigning.

[aaron7]

The current plan to migrate the problem is to enable write auto scaling for the last N inactive tables.

We haven't decided whether to reject old samples yet.

[jml]

@aaron7 can we close this bug now?

[aaron7]

Yes, #556 will migrate the problem for us.