Dockerhub rate limit broke the www.jenkins.io CI build

[MarkEWaite]

Service(s)

ci.jenkins.io

Summary

The ci.jenkins.io job that builds the www.jenkins.io web site failed its most recent build with the message:

You have reached your pull rate limit.
You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

I've restarted the build in hopes that it will not hit the rate limit.

Reproduction steps

1. Open the ci.jenkins.io job and review the log file

[basil]

FYI it broke an https://github.com/jenkinsci/acceptance-test-harness PR build as well, but I was able to successfully retry about an hour and a half later.

[dduportal]

The ci.jenkins.io job that builds the www.jenkins.io web site failed its most recent build with the message:

You have reached your pull rate limit.
You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

About www.jenkins.io (I'll focus on ATH on a second time):

• The build scripts are pulling Docker Library (e.g. "Official") images which ARE subject to rate limit, contrary to the jenkins/* and jenkinsciinfra/* images

  • https://github.com/jenkins-infra/jenkins.io/blob/387ad60415c543e440cc8be06934559227ab251e/scripts/ruby#L13
  • https://github.com/jenkins-infra/jenkins.io/blob/387ad60415c543e440cc8be06934559227ab251e/scripts/node#L14
  • etc.

• ci.jenkins.io agents only have 2 outbound IPs as per https://github.com/jenkins-infra/azure-net/blob/6637c0b38bf0614335375f92c385a3da452e45e0/gateways.tf#L91

• The DockerHub documentation tells us that anonymous pulls are limited to 100 pulls per 6 hours per IP, so ~200 pulls (1 pull == 1 layer OR 1 manifest)

• The pipeline at https://github.com/jenkins-infra/jenkins.io/blob/master/Jenkinsfile never uses any kind of credential to log in to DockerHub (and increasing the available rate limit)

=> it was prone to happen since we moved to NAT gateways a few month agos. Let us open a PR to run the docker login and push the limit forward

[dduportal]

FYI it broke an https://github.com/jenkinsci/acceptance-test-harness PR build as well, but I was able to successfully retry about an hour and a half later.

@basil can you confirm that the rate limit issue, with the ATH build, was with the test "additional" Docker images (and not the jenkins/ath image itself)? The 2 days ago PRs have their build logs already purged, but I see some tests failures on the master branch (build https://ci.jenkins.io/job/Core/job/acceptance-test-harness/job/master/1178/) that might be related.

I'm asking to think about an eventual ACP-like for Docker Engine with a "pull-through" cache as per https://docs.docker.com/docker-hub/mirror/ for ci.jenkins.io

[dduportal]

The ci.jenkins.io job that builds the www.jenkins.io web site failed its most recent build with the message:

You have reached your pull rate limit.
You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

About www.jenkins.io (I'll focus on ATH on a second time):

* The build scripts are pulling Docker Library (e.g. "Official") images which ARE subject to rate limit, contrary to the `jenkins/*` and `jenkinsciinfra/*` images
  
  * https://github.com/jenkins-infra/jenkins.io/blob/387ad60415c543e440cc8be06934559227ab251e/scripts/ruby#L13
  * https://github.com/jenkins-infra/jenkins.io/blob/387ad60415c543e440cc8be06934559227ab251e/scripts/node#L14
  * etc.

* ci.jenkins.io agents only have 2 outbound IPs as per https://github.com/jenkins-infra/azure-net/blob/6637c0b38bf0614335375f92c385a3da452e45e0/gateways.tf#L91

* The DockerHub documentation tells us that anonymous pulls are limited to 100 pulls per 6 hours per IP, so ~200 pulls (1 pull == 1 layer OR 1 manifest)

* The pipeline at https://github.com/jenkins-infra/jenkins.io/blob/master/Jenkinsfile never uses any kind of credential to log in to DockerHub (and increasing the available rate limit)

=> it was prone to happen since we moved to NAT gateways a few month agos. Let us open a PR to run the docker login and push the limit forward

@MarkEWaite : jenkins-infra/jenkins.io#7421

[timja]

FYI it broke an jenkinsci/acceptance-test-harness PR build as well, but I was able to successfully retry about an hour and a half later.

@basil can you confirm that the rate limit issue, with the ATH build, was with the test "additional" Docker images (and not the jenkins/ath image itself)? The 2 days ago PRs have their build logs already purged, but I see some tests failures on the master branch (build ci.jenkins.io/job/Core/job/acceptance-test-harness/job/master/1178) that might be related.

Likely the same issue from a quick look, the actual build logs of the docker image aren't archived though

[basil]

Can you confirm that the rate limit issue, with the ATH build, was with the test "additional" Docker images (and not the jenkins/ath image itself)?

Yes, this was a rate limit error while fetching containers for use during tests. I didn't encounter any problems building or fetching the jenkins/ath image itself.

[dduportal]

Thanks @basil @timja !

I've opened jenkinsci/acceptance-test-harness#1634 to set up authenticated Docker Engine during tests

[dduportal]

Closing as:

• The issue was temporary and short fix was to relaunch pipeline manually
• Both projects have been updated to use Docker auth. to increase the API rate limit on each.

Thanks folks!

[dduportal]

Reopening as we saw a collection of 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit errors in the builds of jenkinsci/docker-agent and jenkinsci/docker on ci.jenkins.io in the past hour.

Example: https://ci.jenkins.io/job/Packaging/job/docker-agent/job/PR-843/1/pipeline-console/?start-byte=0&selected-node=100#log-170

[dduportal]

Reopening as we saw a collection of 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit errors in the builds of jenkinsci/docker-agent and jenkinsci/docker on ci.jenkins.io in the past hour.

Example: https://ci.jenkins.io/job/Packaging/job/docker-agent/job/PR-843/1/pipeline-console/?start-byte=0&selected-node=100#log-170

jenkinsci/docker-agent#844

[dduportal]

A solution to limit this kind of impact would be for us to run registry pull through caches (see. https://docs.docker.com/docker-hub/mirror/) in the ci.jenkins.io agent networks (all VMs and Linux containers)

[dduportal]

@basil @timja I'm continuing the discussion from jenkinsci/acceptance-test-harness#1640 (comment) but here:

I'm not sure how to identify the failure error, I would need help navigating the ATH build and test results. With this I should be more autonomous to find failures, understand them and provide solutions.

• The PR 1640 on the ATH looks like it is using the Docker login. But even the authenticated users have limits: https://docs.docker.com/docker-hub/download-rate-limit/#whats-the-download-rate-limit-on-docker-hub so it could explain any toomanyrequest (or HTTP/429) errors as all ci.jenkins.io agents share the same outbound IP
  • Jenkins Infra task: ensure that user currently configured on ci.jenkins.io is allowed to have 5000 / day /IP (and not 200 / 6h / IP)
    • If we are reaching the authentication limit (wether 200 or 5000), then I'll create additional outbound IPs for ci.jenkins.io to spread the load and try to push the limit a bit more. But it would be a short term patch to unblock the ATH builds.
  • It looks like the ATH pulls a bunch of Docker images which are subject to rate limit (in https://github.com/jenkinsci/acceptance-test-harness/tree/2e6a75edc6d04b3bc38fbe548f2db958d3f0866b/src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures). Would that be ok if we copy these images under a jenkinsciinfra/* (or jenkins/*) DockerHub organization to make them avoiding the rate limit (as these 2 DockerHub organization do not enforce rate limit for there images as per the sponsorship)?

[dduportal]

Yes it is setup as DinD

🤔 what is the reason to use nested containers?

(btw DinD is a nightmare to configure regarding docker login but at least it explains why my PR did not look to work as it only sets up the outer Docker engine.

[dduportal]

Yes it is setup as DinD

I see https://github.com/jenkinsci/acceptance-test-harness/blob/4904fec29f49dedca64214757f8a7898ffa9a329/ath-container.sh#L37 and it looks like it is not DinD (e.g. nested container engine) but DonD (Docker on Docker, e.g. sharing the socket) is my understanding correct?
=> I'm not sure if the authentication is expected to work though (as it might be on the client side).

[timja]

Yes your understanding is correct.

I'm not sure either would need testing

[dduportal]

Yes your understanding is correct.

I'm not sure either would need testing

If it is DonD, then the ACR will be a good solution as the cache through setup is on engine side \o/

[dduportal]

Actually it does look like they are archived:

You can get there from the test report: e.g. https://ci.jenkins.io/job/Core/job/acceptance-test-harness/job/PR-1645/1/testReport/plugins/AntPluginTest/latest_linux_jdk21_firefox_split5___testWithAntPipelineBlock/

https://ci.jenkins.io/job/Core/job/acceptance-test-harness/job/PR-1645/1/testReport/plugins/AntPluginTest/latest_linux_jdk21_firefox_split5___testWithAntPipelineBlock/attachments/docker-SshAgentContainer.build.log

ERROR: permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Head "http://%2Fvar%2Frun%2Fdocker.sock/_ping": dial unix /var/run/docker.sock: connect: permission denied
ERROR: permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Head "http://%2Fvar%2Frun%2Fdocker.sock/_ping": dial unix /var/run/docker.sock: connect: permission denied

All the failed tests in that build seem to have the same error

But I fail to see the relation between these errors and the rate limit :|

[timja]

Might be this fix that was just pushed: jenkinsci/acceptance-test-harness@04f64ef

[dduportal]

Might be this fix that was just pushed: jenkinsci/acceptance-test-harness@04f64ef

Ow yeah, this change might fix it!

[basil]

https://ci.jenkins.io/job/Core/job/acceptance-test-harness/job/PR-1660/2/testReport/junit/plugins/LdapPluginTest/lts_linux_jdk17_firefox_split1___enable_cache/

#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 3.19kB done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/debian:bullseye
#2 ERROR: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/debian/manifests/sha256:7aef2e7d061743fdb57973dac3ddbceb0b0912746ca7e0ee7535016c38286561: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
------
 > [internal] load metadata for docker.io/library/debian:bullseye:
------
Dockerfile:2
--------------------
   1 |     # Sets up
   2 | >>> FROM debian:bullseye
   3 |     
   4 |     # Viewvc is not part of bullseye repos anymore but oldstable https://github.com/viewvc/viewvc/issues/310
--------------------
ERROR: failed to solve: debian:bullseye: failed to resolve source metadata for docker.io/library/debian:bullseye: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/debian/manifests/sha256:7aef2e7d061743fdb57973dac3ddbceb0b0912746ca7e0ee7535016c38286561: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 3.19kB done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/debian:bullseye
#2 ERROR: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/debian/manifests/sha256:907e428c7d1dd4e3a2458d22da8193e69878d3a23761d12ef9cd1a1238214798: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
------
 > [internal] load metadata for docker.io/library/debian:bullseye:
------
Dockerfile:2
--------------------
   1 |     # Sets up
   2 | >>> FROM debian:bullseye
   3 |     
   4 |     # Viewvc is not part of bullseye repos anymore but oldstable https://github.com/viewvc/viewvc/issues/310
--------------------
ERROR: failed to solve: debian:bullseye: failed to resolve source metadata for docker.io/library/debian:bullseye: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/debian/manifests/sha256:907e428c7d1dd4e3a2458d22da8193e69878d3a23761d12ef9cd1a1238214798: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

[dduportal]

Thanks @basil for the details. In order to tackle down these HTTP/429, I propose this course of actions:

• Short term: add 2 more outbound IPs for ci.jenkins.io to immediatly spread the workload to DockerHub (quick, no impact, imemdiate effect)
• Medium term: add an ACR registry only for ci.jenkins.io agents (priority on VMs as they have Docker Engine, eventually on the AKS ci.jenkins.io cluster later) and set up Azure VM agents Docker engines to use it a pull through mirror
  • Then remove the docker login pipeline steps on ATH

=> once these setup are in place, we'll look at the result

[timja]

@dduportal and I got the ACR option working and have tested on ci.jenkins.io.

@dduportal is going to finish off the terraform automation and update the jcasc config.

It looks like our users aren't rate limited and were probably hitting some anti-abuse protection, this should help with that and is expected to get rid of any rate limiting issues.

It will also mean that anything on ci.jenkins.io Azure doesn't need to login anymore, as the docker daemons are going to have a mirror-registry set to point it at the acr cache

[dduportal]

Update:

• New ACR is up and running (see feat(dockerhub-mirror) set up a new dedicated ACR to mirror DockerHub inside the Jenkins Azure infrastructure azure#794)
• ci.jio but also cert.ci, trusted.ci and infra.ci are setup to use it (fix(jenkinscontrollers) switch to the correct ACR jenkins-infra#3575 and feat(infra.ci) use DockerHub ACR mirror for Azure VM agents kubernetes-management#5507)

We can now roll back the docker login steps, along with all the "pullonly" logic

[dduportal]

All changes have been reverted, but I'll keep this issue opened until the 13 in case we see other issues

[dduportal]

Update:

• We were able to build and release 3 core releases for the Jenkins Security 2024-08-07 advisory using the ACR caching
• Checked today: we went from ~12.9 Gb (Tuesday 06 at 13:00 UTC) to ~ 31.2 Gb right now
• Monitoring mentions around succesful 6.6k pulls in the past 24h:

My only concern is that some images or tags are still absent unless we explicitly docker pull them with a ci.jenkins.io pipeline replay. I don't see any errors in the Docker Engine logs so I guess there might be a slight delay between an initial request to a new image reference (which fails as it's not cached yet) so Docker CE falls back to the DockerHub, and the second try once the "ACR cache rule" routine has collected the image tags in the ACR.

Does it make sense @timja ? Have you already seen this behavior in your own infrastructure?

[timja]

Hmm not sure we use it slightly differently and explicitly use the cached version.

It won’t show up in the cache unless one pull has been completed:
https://learn.microsoft.com/en-us/azure/container-registry/container-registry-artifact-cache?pivots=development-environment-azure-portal#limitations

But if it’s increasing in size it’s definitely caching some

[dduportal]

Closing as we did not had any more errors. Feel free to reopen if you see some