bug(ansible): PPA/repo add fetches its signing key from the keyserver with no retry — one transient failure aborts provisioning (8 call sites) #17230

Symptom (live)

Provisioning aborts during the python_interpreter role at approximately task 18:

gpg: keyserver receive failed: End of file

This is the current provisioning blocker. It fires long before the nginx/cert
failure chain that #17172 / #16020 addressed (fixed in #17184), so clearing that
chain does not clear this one.

Root cause

roles/_shared/tasks/add_apt_repository_idempotent.yml:89-96 is the final task of
the shared repo-add helper:

- name: "{{ apt_repo_label }} | Add apt repository (not previously configured)"
  ansible.builtin.apt_repository:
    repo: "{{ apt_repo_spec }}"
    state: present
    filename: "{{ apt_repo_filename }}"
  become: true
  when: _apt_repo_present.stdout | trim == 'MISSING'

When apt_repo_spec is a ppa: specification, ansible.builtin.apt_repository
retrieves the repository's signing key from the Launchpad keyserver at
provisioning time. That is a network fetch with no local fallback.

The task declares no retries, no until, no delay, and no timeout. A single
transient keyserver failure therefore aborts the entire play.

Caller that trips it today — roles/python_interpreter/tasks/main.yml:47-55:

- name: "python_interpreter | Add deadsnakes PPA (idempotent, #7218)"
  ansible.builtin.include_tasks: ../../_shared/tasks/add_apt_repository_idempotent.yml
  vars:
    apt_repo_match: "deadsnakes/ppa"
    apt_repo_spec: "ppa:deadsnakes/ppa"

The asymmetry that makes this a defect rather than bad luck

The role already anticipates exactly this flakiness — one task too late.
roles/python_interpreter/tasks/main.yml:57-71 bounds the cache refresh against
an unreachable PPA:

# Only fires when the interpreter is missing, but when it does it must tolerate
# flaky PPAs (the deadsnakes PPA was just added; if it's unreachable, the
# next deploy retry will still need to install the interpreter).
- name: "python_interpreter | Refresh apt cache (best-effort, bounded; #6719)"
  ansible.builtin.command:
    cmd: >-
      timeout 60 apt-get update -qq
      -o Acquire::Retries=0
      -o Acquire::http::Timeout=10
  failed_when: false

So the step after the PPA add tolerates the PPA being unreachable, while the step
that adds the PPA — the one that actually performs the network key fetch — does
not. The hardening was applied to the wrong task.

Blast radius

The helper is shared. 8 call sites across 7 roles inherit the same unretried
network fetch:

All paths relative to autobot-slm-backend/ansible/.

Only the ppa: call sites do a keyserver fetch; the deb https://… ones still
perform an unretried network operation against the repository host, so the
fragility is shared even where the key path differs.

Acceptance criteria

• The apt_repository task in roles/_shared/tasks/add_apt_repository_idempotent.yml
  retries on failure with a bounded backoff (retries / delay / until), so a
  single transient keyserver or repository-host failure does not abort the play.
• Failure after the retry budget is exhausted still fails loudly — this must not
  become failed_when: false. A repo that genuinely cannot be added has to stop the
  deploy, matching the existing fail-loud stance at
  roles/python_interpreter/tasks/main.yml:81-87.
• The retry behaviour is covered by a guard test under repo_tests/, following the
  pattern of repo_tests/ansible_shared_cert_become_16020_test.py and
  repo_tests/ansible_nginx_cert_before_install_17172_test.py.
• Verified against all 8 call sites — the helper is shared, so the fix lands once and
  the guard asserts it once.

Out of scope

Pinning/shipping repository signing keys locally instead of fetching them at
provisioning time would remove the network dependency entirely rather than making it
tolerant. That is a larger supply-chain change and belongs in its own issue if the
owner wants it.

Related

• fix(ansible): cert provisioning chain — own privilege, and before nginx's install (#16020, #17172) #17184 — slice 1 of the provisioning chain (cert become + ordering), merged
• ansible.cfg puts become under [defaults], where ansible ignores it — the global escalation default has never applied #14699 — provisioning umbrella (ansible.cfg become placement)
• guards(ansible): 13 shared tasks touch root-owned paths without declaring become — privilege is ambient #17185 — shared tasks touching root paths without declaring become
• ci(docker): an unreachable Ubuntu mirror surfaces as a misleading 'held broken packages' error; apt steps neither retry nor fail on index errors #16291 — the CI-Docker analogue: unreachable apt mirror, no retry, misleading error
• tech-debt(install.sh+ansible): audit ALL apt_repository sites for noble-conflict pattern (#7144 follow-up) #7218 — the issue this helper was written for

Correction (2026-09-21, same day as filing). Two line ranges were off.
:56-70 started on a blank line — the comment block begins at :57 and the task it describes
runs :61-71, so the range is now :57-71. :81-86 stopped one line short of the task, which
ends at :87 (the last line of the file). Substance unchanged; both ranges re-derived from the
file with explicit line numbering rather than from a sed window, which is what hid the
leading blank line the first time.