Description
The code that slows down writes to Parameter Store (so they don't hit AWS's rate limit) had two problems:
- It only started slowing writes down once a single Lambda invocation's own batch reached the store's
maxWritesPerSecond number, and it based the delay only on that one invocation's batch size. But Parameter Store's write limit applies to the whole AWS account, not to one invocation: several pools can each run their own scale-up/pool Lambda at the same time, each staying under its own limit, while their writes added together go over the account limit. A batch smaller than the threshold got no slowdown at all.
- The write limit itself was hardcoded to 40 writes/second (SSM's default), with no way to say "we turned on SSM's higher-throughput setting, which allows a lot more."
Impact
Running several pools at once (a normal setup that's actually encouraged) can push the total Parameter Store writes over the account-wide limit, even though no single pool's Lambda ever looks like it's over the limit by itself. This shows up as ThrottlingException errors in account-wide metrics, not in any one Lambda's own metrics. Accounts that turned on SSM's higher-throughput setting get no benefit from it, because the code still slows writes down to a lower, fixed number regardless.
Proposed fix
- Always slow down writes once a write limit is set, not just for large batches in a single invocation.
- Make both the assumed number of invocations running at once, and the write-rate limit itself, configurable — so the delay between writes matches the account's real concurrent-invocation count and its real SSM throughput setting.
See PR (to follow) for the implementation.
Description
The code that slows down writes to Parameter Store (so they don't hit AWS's rate limit) had two problems:
maxWritesPerSecondnumber, and it based the delay only on that one invocation's batch size. But Parameter Store's write limit applies to the whole AWS account, not to one invocation: several pools can each run their own scale-up/pool Lambda at the same time, each staying under its own limit, while their writes added together go over the account limit. A batch smaller than the threshold got no slowdown at all.Impact
Running several pools at once (a normal setup that's actually encouraged) can push the total Parameter Store writes over the account-wide limit, even though no single pool's Lambda ever looks like it's over the limit by itself. This shows up as
ThrottlingExceptionerrors in account-wide metrics, not in any one Lambda's own metrics. Accounts that turned on SSM's higher-throughput setting get no benefit from it, because the code still slows writes down to a lower, fixed number regardless.Proposed fix
See PR (to follow) for the implementation.