fix(provision): install the LLC agent CLIs on ansible-deployed hosts (#12681)

[mrveiss]

Refs #12681 — item 1 of the three the issue covers. Not Closes; item 2 remains.

Thinking Path

The issue reports that setup-wizard provisioning fails invisibly. Item 1 is the
blunt half: nothing installs the claude CLI onto the service PATH.

docker/backend/Dockerfile does it properly — both agent CLIs from official signed
apt repos, with gh --version && claude --version at build time so an install
regression fails the build instead of silently breaking LLC dispatch.

The ansible path had nothing. Searching the whole ansible tree for claude
returned zero files. So on a systemd-deployed node:

• resolve_cli_binary("claude", ...) returns None
• the agent was already hired successfully — llc/api/agent_hires.py validates
  that the adapter type is registered, never that it is available
• the run surfaces as SKIPPED, not as a provisioning error

Which is precisely the invisible-provisioning symptom, and why item 3 (rendering
the skip reason, merged as #14761) was worth doing first: it makes this failure
legible while it exists.

gh is included deliberately. copilot_local_adapter._resolve_gh_cli resolves
it exactly the same way, so installing only claude would leave that adapter's
agents equally unrunnable — half-fixing the same defect. The image installs both
together for the same reason.

What Changed

autobot-slm-backend/ansible/roles/backend/tasks/main.yml — the same signed repos
as the image:

• keyring directory, then both signing keys via get_url (not piped into a
  shell), dpkg --print-architecture read for the GitHub source line
• both apt sources written, apt refreshed only when a key or source changed
• gh + claude-code installed with install_recommends: false
• a resolve check on the host (gh --version, claude --version)

backend_claude_cli_version (defaults, empty) pins an exact package version,
mirroring the image's existing CLAUDE_CLI_VERSION build arg.

The resolve check is the point rather than a nicety: installing without checking
lets a broken PATH look identical to a clean provision, which is the failure mode
this issue is about.

Verification

I cannot run ansible here, so this is verified structurally and by rendering — not
against a host.

Jinja compiles and renders, with Ansible's filters stubbed. I checked this
explicitly because on my previous PR (#14784) I validated files as YAML, which says
nothing about Jinja, and shipped a nested-brace TemplateSyntaxError that a
security review caught:

compiled 8 new expressions, 0 failures
unpinned -> ['gh', 'claude-code']
pinned   -> ['gh', 'claude-code=2.1.89']

The empty pin yields a bare package name rather than a trailing =, which would
otherwise be an apt error on every unpinned provision.

The guard fails on base and passes here:

CONTROL — base (pre-fix) ansible tasks:
  installs claude-code: False   <- must be False
  proves it resolves  : False   <- must be False
control result: PASS — the guard would have failed on base

It asserts the provisioning, not the resolver: a test over resolve_cli_binary
passes whether or not anything ever installs the binary, which is how this stayed
invisible. It contains no shutil.which gate anywhere — a test skipped for a
missing binary is indistinguishable from a pass (#14550).

What this does not fix

Item 2 — llc/api/agent_hires.py:365-372 still accepts an unavailable adapter.
That is deliberate and the ordering matters: today an operator can hire an agent
and hand-install the CLI afterwards. Rejecting the hire while nothing installs the
binary would make provisioning strictly worse. This PR is what makes item 2 safe
to land, and item 2 should follow it, not precede it.

Nothing here touches a live host — it changes what a future provision installs.

Model Used

Claude Opus 5

[github-actions]

✅ SSOT Configuration Compliance: Passing

🎉 No hardcoded values detected that have SSOT config equivalents!

[mrveiss]

Both HIGH findings fixed in adc0ee118b

1. Blast radius — tasks moved out of the restart-guarded window

Correct and I had it wrong. The tasks sat inside the #12139 block, which has no
rescue: and whose always: only force-starts the old binary. They reach two
third-party apt repos, so an outage at either would have aborted the whole deploy —
code sync, config, workers, nginx, health check — after the service was already
stopped. That is precisely the incident the block exists to prevent.

My PR body argued "a host that cannot install the CLI should fail provisioning". I
still think that is right, but I was reasoning about whether to fail and never
about where the failure lands. Moved before the block: a repo hiccup now fails
this feature loudly and leaves a running backend running.

2. The guard test was vacuous — confirmed by mutation, not inspection

Both cases reproduced exactly as reported:

• "gh" in text is satisfied by Copyright in the licence header (Co-py-ri-gh-t)
• the claude --version regex matched my own explanatory comment, not the
  templated cmd: "{{ item }} --version"

The assertions now parse the task file and inspect the apt package list and the
verify loop structurally. Every one is mutation-tested:

BASELINE                      installs gh/claude-code: True/True   resolves gh/claude: True/True
drop gh from package list     installs gh: False          claude-code: True  (unaffected)
delete the verify task        resolves gh: False          claude: False
drop claude from verify loop  resolves claude: False      gh: True           (unaffected)
old check on a gh-less file   "gh" in text: True          <- the original bug

Worth naming plainly: I shipped this having run a "control" that only checked the
assertions pass, and against base I checked two of the four. A control that
never exercises the failing direction is not a control — which is the same defect
I have been flagging in other people's tests all session, in my own safety net.

MEDIUM findings

Version-pin cache staleness — fixed. The refresh now also fires when
backend_claude_cli_version is non-empty, since keys and sources only change on
first install and a converged host would otherwise never refresh.

lock_timeout — added to both apt tasks.

failed_when: false + bounded timeout on update_cache — deliberately not
applied, and I want to be explicit rather than silently skip it. That convention
(main.yml:317-326) exists because a failure inside the restart-sensitive block is
catastrophic. These tasks are no longer in that block, so the rationale no longer
holds — and making the refresh best-effort would let a stale cache proceed to an
install that fails anyway with a worse message. Happy to add it if you disagree.

Unchanged

Jinja and security posture, both confirmed clean in review. I recompiled all 11
CLI-related expressions after the move: 0 failures.

[mrveiss]

Re-review verdict accepted; the one open nit is now fixed too

Bounded the key fetch with timeout: 120, following this file's own #2835
precedent. The point stands on its own merits: lock_timeout bounds the dpkg
lock, not a hung TCP connection to downloads.claude.ai / cli.github.com, and an
unbounded fetch stalls the entire play indefinitely — worse than either outcome the
placement change was arguing between.

One limitation worth recording rather than leaving implied: ansible.builtin.apt
takes no timeout parameter, so the two apt calls remain bounded by lock_timeout
alone. The file's other bounded example (timeout 60 apt-get update -qq) achieves
it by shelling out, which I have not adopted here — the module form is the better
default and the fetch that actually reaches a third-party host is the get_url,
which is now bounded.

On failed_when: false: agreed and unchanged. Fail-loud beats a stale cache
producing "unable to locate package version X" three tasks later.