Reprocessing workflow cause many DLQ messages

[iliapolo]

Whenever our reprocess workflow is triggered, we see a bunch of DLQ message that say:

Rate exceeded (Service: AmazonECS; Status Code: 400; Error Code: ThrottlingException; Request ID: 1db00da3-2470-4c18-b97a-ffca0c6aa602; Proxy: null)

This happens because reprocessing means spawning around 40,000 ECS tasks, while our current service limit stands on 1000.

I think we need a two vector solution here:

1. Increase service limits to make reprocessing faster.
2. Tweak the retry policy or some other mechanism to prevent this from happening.

[Chriscbr]

I wonder if it would make sense to fix the throttling using a queue?

We could put all of these requests into an SQS queue first, and then either:

• create a lambda to process requests into ECS tasks (perhaps in a way that checks that there aren't too many ECS tasks running)
• or, configure our ECS service to auto-scale based on the size of the SQS queue, and automatically poll the SQS queue for messages

[iliapolo]

@Chriscbr

create a lambda to process requests into ECS tasks (perhaps in a way that checks that there aren't too many ECS tasks running)

We were trying to avoid implementing our own back-pressure mechanism, I think that tweaking the SFN retry policies pretty much gives us the same benefit.

or, configure our ECS service to auto-scale based on the size of the SQS queue, and automatically poll the SQS queue for messages

I don't think auto-scaling is relevant here (ignoring the fact we don't have an ECS service, only tasks, and there is no auto-scaling mechanism for tasks). At any point in time our queue will have more messages than ECS can consume in parallel according to our service limits, so what we need here is back-pressure. If we can configure ECS to poll from a queue and perform that back-pressure on its own, that would be fantastic, but I'm not sure this is possible.

Having said that, I do agree we need to rethink this architecture and come up with something more robust, this will be scheduled later on as a bigger operational excellence project.