Repository navigation
deploy: PLAY 2 fails at 'Browser | Wait for service to be ready' — autobot-browser-worker has no unit file and :3000 is held by Grafana on loopback; last blocker after #12883 #12912
Description
Summary
With #12871 and #12883 fixed, PLAY 2 now runs nearly to completion — backend .env regen, migrations, AI-stack, frontend and NPU all succeed (ok=108 changed=32). It then fails on the last component:
TASK [[PLAY 2] Browser | Restart service] **************************************
changed: [00-SLM-Manager]
TASK [[PLAY 2] Browser | Wait for service to be ready] *************************
fatal: FAILED! => {"changed": false, "elapsed": 61,
"msg": "Timeout when waiting for <node-ip>:3000"}
PLAY RECAP
00-SLM-Manager : ok=108 changed=32 unreachable=0 failed=1 skipped=24
The wait can never succeed, for two independent reasons.
Cause 1 — there is no autobot-browser-worker unit
$ systemctl cat autobot-browser-worker
(no output — no unit file)
$ systemctl show autobot-browser-worker -p ActiveState,SubState,Result,NRestarts
ActiveState=inactive
SubState=dead
Result=success
NRestarts=0
$ journalctl -u autobot-browser-worker -n 12
-- No entries --The service has never started and has no unit on disk. [PLAY 2] Browser | Deploy autobot-browser-worker copies files and Restart service reports changed, but nothing is left listening. (The separate autobot-playwright.service is active — worth confirming which of the two the play intends to manage, because the restart reporting changed while the named unit does not exist suggests it is restarting something else.)
Cause 2 — port 3000 is Grafana, bound to loopback only
$ ss -ltnp | grep :3000
LISTEN 0 4096 127.0.0.1:3000 0.0.0.0:* users:(("grafana",pid=678,fd=19))
$ curl -o /dev/null -w '%{http_code}' http://127.0.0.1:3000/health -> 301 (redirects to /grafana/health)
$ curl -o /dev/null -w '%{http_code}' http://<node-ip>:3000/health -> 000 (nothing listening)So even if the browser worker started, port 3000 is already taken by Grafana. And the play waits on the node's LAN address, while Grafana binds 127.0.0.1 only — so the probe fails regardless of which service is meant to answer.
Impact
- PLAY 2 fails at the final step, so
self_update_incompletestays true and the update never reports success — despite 108 tasks succeeding. - Browser automation has no running worker on this host.
- This is now the last blocker in the deploy: Update-All STILL skips Play 2 — #12567 detach activates but the transient scope is killed by the Play-1 SLM restart (incomplete #12425) #12596 → CRITICAL regression(slm): self-update fails in ~1s every run after #12775 — caller's finally unlinks the temp inventory/extra-vars out from under the now-detached playbook; API still returns success:true and status reports has_update:false #12803 → deploy: self-update PLAY 2 aborts on AnsibleUndefinedVariable 'backend_chromadb_auth_token' — playbook-level template task can't see roles/backend defaults, so PLAY 3 never runs and #12777's unit fix stays undeployable #12871 → CRITICAL(deploy): PLAY 2 .env regen overwrites the working autobot_app DB password with a dead one — two divergent credential stores; backend is one restart away from losing PostgreSQL #12883 chain.
Proposed fix
- Establish which unit owns browser automation —
autobot-browser-workerorautobot-playwright— and make the play deploy, restart and probe that one. ARestart servicetask reportingchangedfor a unit that does not exist is a silent no-op worth fixing on its own. - Resolve the port collision: move the browser worker off 3000, or move Grafana. Two services configured for the same port on one host cannot both be right.
- Probe the address the service actually binds. If it is loopback-only by design, waiting on the LAN address will always time out.
- Consider making the wait non-fatal (or
ignore_errorswith a warning) so one unready optional worker does not fail an otherwise complete update — the other 108 tasks succeeded.
Related
- deploy: Update-All STILL skips Play 2 — #12567 detach activates but the transient scope is killed by the Play-1 SLM restart (incomplete #12425) #12596 → CRITICAL regression(slm): self-update fails in ~1s every run after #12775 — caller's finally unlinks the temp inventory/extra-vars out from under the now-detached playbook; API still returns success:true and status reports has_update:false #12803 → deploy: self-update PLAY 2 aborts on AnsibleUndefinedVariable 'backend_chromadb_auth_token' — playbook-level template task can't see roles/backend defaults, so PLAY 3 never runs and #12777's unit fix stays undeployable #12871 → CRITICAL(deploy): PLAY 2 .env regen overwrites the working autobot_app DB password with a dead one — two divergent credential stores; backend is one restart away from losing PostgreSQL #12883 — the preceding blockers in this same chain, all now fixed
- bug(tts): synthesis 404s — deployed worker lacks /tts/synthesize/stream because autobot-tts-worker has NO resolve path AND NO drift visibility (silent skew on every backend update); /tts/clone-voice is a dead call that exists nowhere #12886 — the TTS worker gap found alongside this
- Umbrella: every service must be SLM-managed & updatable — postgres has no playbook; 'update all' skips co-located AI-stack roles #11820 — every service must be SLM-managed and updatable
Verification
sudo tail -40 /var/log/autobot/self-update-ansible.log | grep -A4 "Browser | Wait"
systemctl cat autobot-browser-worker ; systemctl is-active autobot-playwright
sudo ss -ltnp | grep :3000
Activity
mrveiss commentedon Jul 29, 2026
This is blocked by #12959 as well as by its own two causes — worth recording before anyone writes the fix.
Checked the role that owns the browser component:
There is no
autobot-browser-worker.service.j2. So the missing unit file is not deployment drift — the template does not exist in the repo either, which matches this issue's finding on the host.The second layer is the problem for whoever fixes it. Per #12959,
update-all-nodes.ymlnever appliesroles/browser(it appears in PR #12960's report as NEVER applied). So adding the unit template here produces a fix that is correct in the repo and cannot reach a host through code-sync / self-update — the same trap that left #12886 closed while TTS still 404s, and #12907 merged while both credential defects stay live.Practical consequence: this issue needs both halves to be verifiable —
autobot-browser-worker.service.j2(and resolve the:3000conflict with Grafana on loopback), androles/browseris actually applied.Fixing only (1) will look done, pass review, close — and PLAY 2 will fail at the identical task on the next run, which is exactly the outcome this issue already reports as "unchanged".
Confirmed on the host just now: no
autobot-browser-workerunit is present insystemctl list-unit-files.fix(deploy): report role-owned changes that never reached the host (#…
fix(deploy): report role-owned changes that never reached the host (#…
fix(deploy): flush the TTS restart immediately so a later failure can…
fix(deploy): restart the TTS worker explicitly, not via notify handler (
fix(deploy): wait on the port the browser worker binds, and deliver t…
github-actions commentedon Jul 31, 2026
PR #13150 (merged to
Dev_new_gui) references this issue with a close keyword.If this issue is fully resolved, close it manually. If work remains, no action is needed.
mrveiss commentedon Aug 1, 2026
Audit note — repo fix verified, live acceptance still outstanding
Repo-side, PR #13150 (
e489b51dc) addresses causes 1–3 and this was verified on the current tip:autobot-playwrightrather than the nonexistentautobot-browser-worker.playwright.service.j2confirmsWorkingDirectory=/opt/autobot/autobot-browser-worker+ExecStart=.../playwright-server.js, so that is the correct unit for the deployed files.port: "{{ playwright_port | default(9001) }}", androles/browser/defaults/main.yml:20confirmsplaywright_port: 9001.Why this stays open: PR #13150's own body states that acceptance is host-side and unproven until it merges, and commits to reporting the outcome either way. That report was never posted — the last activity here is the merge note. The issue cannot be discharged by static analysis.
One residual repo gap found during the audit, worth recording:
update-all-nodes.ymlrendersroles/backend(env_only,unit_only) androles/tts-worker(code_only), but never rendersroles/browser'splaywright.service.j2. So the play restarts a unit it does not deliver. If a host'sautobot-playwrightunit is stale or absent, this play cannot repair it — the same class of defect as #12777, and inside the scope of still-open #12959 ("every fix landing in an Ansible role is inert on hosts").mrveiss commentedon Sep 12, 2026
Closing — fully delivered; never auto-closed because Dev_new_gui wasn't GitHub's configured default branch at merge time.
Verified against
origin/main:update-all-nodes.yml:1464-1471waits on{{ playwright_port | default(9001) }}, not hardcoded 3000.roles/browser/defaults/main.yml:20playwright_port: 9001.update-all-nodes.yml:1449restartsautobot-playwright, not the nonexistentautobot-browser-worker.tests/test_updater_browser_port_and_backend_unit.py.Delivered by merged PR #13150.