ci: system packages ansible installs on hosts are absent from CI runners, so their tests skip silently

Summary

setup-python-suite installs Python dependencies only. The system packages ansible installs on every backend host are not installed on CI runners, so any test needing one either skips or fails for the wrong reason — and a skip is indistinguishable from a pass.

Found while landing #13896: pytesseract was installed from requirements.txt, the tesseract binary was not, ocr_availability() returned false, and every end-to-end OCR test skipped. The visible symptom was an unrelated-looking assertion, assert (1, 2, 3, 4) == (3, 4).

That is #13885's exact shape reproduced in CI: the binding present, the binary absent, and a green run that proved nothing.

The general gap

autobot-slm-backend/ansible/roles/backend/tasks/main.yml installs these on hosts, each because some feature does not work without it:

xvfb, x11-utils, x11-apps        GUI / VNC automation
ffmpeg, libsndfile1, portaudio   audio / voice / Whisper
espeak-ng, espeak-ng-data        TTS (#10566: "no speech voices installed" without it)
tesseract-ocr, libtesseract-dev  OCR / CAPTCHA (#206)
python3-tk                       matplotlib GUI backend
postgresql-client                pg_dump for pre-migration backup

#13896 adds tesseract-ocr to the runner. The others are unaudited — for each, either its tests are skipping in CI today, or nothing tests it at all. Both are worth knowing, and neither is currently visible.

Why a skip is worse than a failure here

Every one of these features is guarded by an optional import that degrades to a message. That is correct at runtime and blinding in a test suite: the gate is always closed, the tests never run, and the suite is green. #13885 survived months on exactly that.

Scope

Audit each system package above: does a test need it, and does that test currently run or skip in CI?
Install the ones with real test coverage on the runner (cheap; tesseract-ocr added in feat(document): rasterize-then-OCR fallback for PDFs with no text layer #13896 as the first)
For each capability whose tests are skipif-gated on a system dependency, add the counterpart guard feat(document): rasterize-then-OCR fallback for PDFs with no text layer #13896 introduces — assert the provisioning when CI is set, so an always-closed gate fails loudly instead of passing quietly
Where a capability genuinely cannot be exercised in CI, record that explicitly rather than leaving a permanently-skipping test that reads as coverage

Note

The pattern worth copying is the guard, not the apt line: test_ci_actually_has_the_ocr_toolchain in autobot-backend/tests/test_ocr_fallback_13896.py. It turns "these tests skipped" from an invisible non-event into a failing assertion.

fix(deps): pytesseract binding never installed — every OCR surface is dormant #13885 — the binding/binary split that started this
feat(document): rasterize-then-OCR fallback for PDFs with no text layer #13896 — adds the tesseract install and the provisioning guard
ci: the backend Python suite gates nothing — python-suite is not a required check #14353 — making the Python suite gate anything at all

fix(ci): close the paths-filter gap that let the new guards skip (#14550

fix(media): stop ffmpeg rejecting every real audio extraction (#14550)

fix(ci): stop apt flags from being reported as installed packages (#1…

fix(ci): bound the apt-get install so a stuck dpkg lock cannot hang a…

fix(ci): a composite-action step key that broke every shard, not just…

fix(ci): give the ffmpeg install enough headroom for 12 independent c…

fix(ci): retry the ffmpeg apt install instead of guessing a bigger ti…

fix(ci): guard requirements-ci drift from production, mirror pytesser…

fix(ci): install ffmpeg in CI and guard ansible-vs-CI system packages (…

fix(ci): close the paths-filter gap that let the new guards skip (#14550

fix(media): stop ffmpeg rejecting every real audio extraction (#14550)

fix(ci): stop apt flags from being reported as installed packages (#1…

fix(ci): bound the apt-get install so a stuck dpkg lock cannot hang a…

fix(ci): a composite-action step key that broke every shard, not just…

fix(ci): give the ffmpeg install enough headroom for 12 independent c…

fix(ci): retry the ffmpeg apt install instead of guessing a bigger ti…

fix(ci): stop the guard-reach checker truncating on a mid-list comment (

fix(ci): guard requirements-ci drift and ansible-vs-CI system packages (

Delivered by PR #14578, squash-merged as 1d8352bcd and verified in origin/Dev_new_gui.

Enabling ffmpeg in CI immediately exposed a completely broken production method — which is this issue's thesis demonstrated rather than argued.

FFmpegService.extract_audio() passed -allowed_extensions, an option valid only for the dash/hls demuxers, so ffmpeg rejected the entire command for any plain file. Verified in base: those three occurrences are now explanatory comments, not the flag. It stayed invisible because the one test covering it was skipif-gated on shutil.which("ffmpeg") and every CI runner lacked the binary.

Measured 0/1 gated binaries provisioned before, 1/1 after. Ansible installs 13 feature packages on backend hosts; a tree-wide sweep for shutil.which skip-gates found exactly one with a real binary-dependent test. The other twelve are recorded as out of scope rather than silently dropped — and pg_dump's test replaces the backup function outright, so it proves nothing about the real binary either way.

The canary is the durable part: test_ci_actually_has_the_ffmpeg_toolchain asserts under CI=true only, so a provisioning regression fails loudly instead of skipping quietly — the pattern #13896 established for OCR.

Getting there took four rounds of CI, each a distinct defect, worth recording because none was the obvious one:

tree-sitter-python does not declare tree-sitter as a runtime dependency — the mirror's own rationale had assumed it did, unverified.
A mock.patch block in faiss_ivfpq_builder_test.py closed before its assertions, so they computed against the unpatched value and checked a file never written.
timeout-minutes is not valid on a composite-action step, so the fix for a dpkg-lock hang broke the action's template parsing and killed every shard before a test ran.
Twelve ephemeral runners each rolling against their own apt-daily timer — resolved with a bounded retry whose set -e propagation was proven by harness, not reasoned about.

Filed along the way: #14632 (actionlint's trigger never covered composite actions) and a new required check_composite_action_step_keys.py covering all three, so that class cannot recur silently.

fix(guards): the git conversion broke non-git trees, and a git call e…

guard(repo): five guards walked into other checkouts of this reposito…

fix(ci): fetch the CI system packages once per run instead of once pe…

fix(ci): give the resumable apt download its own budget, and make a p…

fix(ci): fetch the CI system packages once per run instead of once pe…

fix(ai-ml): the dynamic model loads consult the pinned registry, and …

Navigation Menu

Repository navigation

ci: system packages ansible installs on hosts are absent from CI runners, so their tests skip silently #14550

Summary

The general gap

Why a skip is worse than a failure here

Scope

Note

Related

9 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

ci: system packages ansible installs on hosts are absent from CI runners, so their tests skip silently #14550

Description

Summary

The general gap

Why a skip is worse than a failure here

Scope

Note

Related

Activity

9 remaining items

mrveiss commented on Aug 21, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions