Rate limiting issues when pulling LWA image during container builds

[enomoto11]

Issue Description

When trying to build a container image with Lambda Web Adapter (LWA), we're encountering rate limiting issues while pulling the base image from public ECR.

Current Behavior

When building a container with the following Dockerfile instruction:

COPY --from=public.ecr.aws/awsguru/aws-lambda-adapter:0.9.0 /lambda-adapter /opt/extensions/lambda-adapter

We get the following error:

Error: buildx failed with: ERROR: failed to solve: public.ecr.aws/awsguru/aws-lambda-adapter:0.9.0: failed to resolve source metadata for public.ecr.aws/awsguru/aws-lambda-adapter:0.9.0: failed to copy: httpReadSeeker: failed open: unexpected status code https://public.ecr.aws/v2/awsguru/aws-lambda-adapter/manifests/sha256:xxx: 429 Too Many Requests - Server message: toomanyrequests: Data limit exceeded

Expected Behavior

The image should be pulled successfully without hitting rate limits, especially since this is an official AWS component required for Lambda container deployments.

Questions

1. Are there any recommended workarounds for this rate limiting issue?
2. Is there an alternative way to include LWA in container builds that won't be affected by public ECR rate limits?
3. Could the image be made available through alternative distribution channels that don't have such strict rate limits?

Environment

• Building through GitHub Actions
• Using Docker Buildx
• Target platform: AWS Lambda

Impact

This issue affects CI/CD pipelines where containers need to be built frequently, as the rate limits can cause build failures and deployment interruptions.

[bnusunny]

@enomoto11 thanks for raising this issue. Using a pull through cache is an immediate workaround to allow your pipelines to work. For example, here is the document on how to setup pull through cache with ECR Private repo.

https://docs.aws.amazon.com/AmazonECR/latest/userguide/pull-through-cache.html

[enomoto11]

@bnusunny
Thank you for your quick response!

I initially worked around this by modifying the source code's entry point to be a Lambda handler instead of using the Lambda Web Adapter. The concept of pull-through cache is new to me, and I'd love to understand it better.

From my current understanding:

1. The issue we're facing is that we can't pull the base image from the public container repository due to rate limiting, even after waiting for a sufficient cool-down period
2. Pull-through cache seems to be a mechanism to reduce the number of accesses to public container repositories to avoid hitting rate limits in the first place

I apologize if my understanding of pull-through cache is incomplete, but I'm curious: How would implementing pull-through cache help resolve this situation where we're already receiving rate limit errors? Would really appreciate if you could explain the mechanics behind this solution, as it would be a great learning opportunity for me and others who might face similar issues.

Also, would the pull-through cache solution work even if we're already in a state where we're getting rate-limited by the public ECR? I'm particularly interested in understanding the relationship between ECR's rate limiting mechanism and how pull-through cache interacts with it.

[bnusunny]

Let me back up a bit. We may have simplier solutions for you.

The public ECR has limts on both data transfer and image pull.

For data transfer, you can anonymously (without using an AWS account) transfer 500 GB of data to the Internet from a public repository each month for free (can't pull more than 500GB per month anonymously. This is limited by source IP). If you sign up for an AWS account, or authenticate to Amazon ECR with an existing AWS account, you can transfer 5 TB of data to the Internet from a public repository each month for free. You also get unlimited bandwidth at no cost when transferring data from a public repository to AWS compute resources in any AWS Region. See the details on the ECR pricing page.

For image pull, the maximum number of unauthenticated image pulls per second is 1 per region. You can get 10 tims image pull per second if you authenticate to Amazon ECR. Details are on this page.

So if you pull from public ECR repo anonymously, you can transfer 500 GB of data per month with 1 image pull per second. You can get 5 TB of data transfer by login to Amazon ECR or ulimited data transfer by pulling from AWS compute resources in any AWS region.

So the simplist solution to your problem is login to Amazon ECR before using LWA images. You can do it with the command below.

aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws

Now, let's talk about pull through cache. In case that you need to pull LWA images more than 10 time per second (this is unusual), you can add private ECR repo and setup pull through cache rule to cache LWA images in your own private ECR repo. In this way, you only need to pull down the LWA images once. You should be able to do that even if you are rate limited, because you still can do 10 image pulls per second.

My suggestion is to add login ECR step in your CI/CD pipeline to increase the rate limit 10x, before considering pull through cache.

[enomoto11]

Sorry for the late reply, and thank you! I didn't know there was a difference in rate limits between anonymous users and authenticated ones. This is new to me. So if we authenticate with an AWS account when pulling, we should be able to use LWA without hitting the rate limits. This is very helpful information!