Skip to content

Parameter Store write-pacing guard is scoped per-invocation, but the limit is account-wide #5410

Description

@atsikham

Description

The code that slows down writes to Parameter Store (so they don't hit AWS's rate limit) had two problems:

  1. It only started slowing writes down once a single Lambda invocation's own batch reached the store's maxWritesPerSecond number, and it based the delay only on that one invocation's batch size. But Parameter Store's write limit applies to the whole AWS account, not to one invocation: several pools can each run their own scale-up/pool Lambda at the same time, each staying under its own limit, while their writes added together go over the account limit. A batch smaller than the threshold got no slowdown at all.
  2. The write limit itself was hardcoded to 40 writes/second (SSM's default), with no way to say "we turned on SSM's higher-throughput setting, which allows a lot more."

Impact

Running several pools at once (a normal setup that's actually encouraged) can push the total Parameter Store writes over the account-wide limit, even though no single pool's Lambda ever looks like it's over the limit by itself. This shows up as ThrottlingException errors in account-wide metrics, not in any one Lambda's own metrics. Accounts that turned on SSM's higher-throughput setting get no benefit from it, because the code still slows writes down to a lower, fixed number regardless.

Proposed fix

  • Always slow down writes once a write limit is set, not just for large batches in a single invocation.
  • Make both the assumed number of invocations running at once, and the write-rate limit itself, configurable — so the delay between writes matches the account's real concurrent-invocation count and its real SSM throughput setting.

See PR (to follow) for the implementation.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions