fix(transliterator): excessive throttling from ECS during reprocessing

[iliapolo]

During the reprocessing workflow, step functions tries to start a burst of 60,000 (current number of package versions) ECS tasks. Since our account limit is only 1000 parallel tasks, we need to apply a retry policy so the throttled tasks don't end up in the DLQ.

Currently, our retry policy allows for a total wait time of roughly 2.5 hours. Lets do some math to see if this is enough.

Since tasks also have boot time, we don't really run 1000 in parallel. In practice what we normally see is:

So for simplicity sake lets assume 500 parallel tasks. If every task takes about 2 minutes (empirically and somewhat based on jsii-docgen test timeouts) we are able to process 1000 tasks in 4 minutes.

This means that in order to process 60,000 tasks, we need 4 hours. The current retry policy of 2.5 hours allows us to process only about 35,000 tasks. And indeed, most recent execution of the workflow resulted in the remaining 25,000 tasks being sent to the DLQ.

The retry policy implemented in this PR gives us 7 hours.

TODO

• 5 hours might still a bit too close. Run the reprocess workflow again to see if the numbers have changed following chore: cache expensive lookups jsii-docgen#553. Follow up: jsii-docgen improvements did make it better but not enough to put a significant dent. I've updated the PR to give us 7 hours.

Fixes #708

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license