Skip to content

ci: Docker Hub anonymous pull rate limit fails integration and e2e jobs on hosted runners #805

Description

@EricAndrechek

Problem

CI's integration, e2e and coverage jobs pull their test containers (Redis, NATS, ClickHouse and others) from Docker Hub anonymously, from GitHub-hosted ubuntu-latest runners. Docker Hub rate-limits anonymous pulls per source IP, and hosted runners share IPs, so a job can fail before any test runs because a container can't be created. Those failures look like test failures and turn main red for reasons unrelated to the change.

Evidence (measured, 2026-10-09)

  • main at c01b912 (ci: add make ci-remote and make the repo's hooks fail closed #798's merge): CI failed in Integration tests (app) and Coverage. The integration (app) job log contains toomanyrequests 6 times.

  • PR ci(e2e): cache the oldest-Node download of the SDK dist smoke #803 (a CI-cache change touching no Go code), the same evening: Integration tests (app), Integration tests (backends), E2E tests and Coverage failed. In integration (backends), 18 tests failed, starting with internal/cache (TestRedis_Conformance and its siblings), each with:

    create container: Error response from daemon: toomanyrequests: You have reached your unauthenticated pull rate limit. https://www.docker.com/increase-rate-limit
    
  • The three main runs before c01b912 passed, so this is intermittent, depending on which runner IP a job lands on and how many pulls that IP has already made.

Where the images come from

Testcontainers pulls images by Docker Hub name, for example redis:8.10.2-alpine in the cache tests and clickhouse/clickhouse-server:<version> in the orchestrator, with no registry prefix and no docker login step in the workflows. Each job that starts containers pulls on its own, so one workflow run makes several anonymous pulls per image.

Part of #740.

Activity

  1. EricAndrechek commented on Oct 9, 2026

    @EricAndrechek
    MemberAuthor

    Re-run result (measured): re-running only the failed jobs, on fresh runners, passed both for main at c01b912 (run 37989407513, attempt 2) and for #803 (run 37990130879, attempt 2). Same code, so the failures were the pull limit, not the change.


    — Posted by Claude Code on behalf of @EricAndrechek

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions