Repository navigation
fix(ansible): unblock co-located SLM Manager provisioning (#9933 #9934) - #9935
Merged
Merged
Conversation
Two failures surfaced on the co-located SLM Manager once the fresh-VM hostname/systemd bugs (#9914/#9915, PR #9916) were cleared. Part of umbrella #9919 (deployment-install); concrete root causes for the 4a->4c chain tracked in #9281. #9933 nginx `duplicate upstream "autobot_backend"`: the frontend role (Phase 4b) enables a standalone autobot-frontend site before the co-location flag is set, so both autobot-frontend and autobot-slm declare the upstream and `nginx -t` fails in Phase 4c. Remove the conflicting sites-enabled/autobot-frontend symlink in the Phase 4c co-location play — where co-location is definitively known — before testing the config; include that removal in the nginx test/reload trigger conditions. #9934 backend systemd unit had no LimitNOFILE, so the multi-worker backend (DB/Redis/ChromaDB pools + HTTP clients) exhausted the default fd ceiling on the busy co-located host: OSError [Errno 24] Too many open files -> socket.accept() out of system resource -> the #3687 health gate never went green. Add LimitNOFILE (default 65536, overridable), matching every other AutoBot unit. Verified: ansible-playbook --syntax-check provision-fleet-roles.yml (exit 0); confirmed only autobot-frontend + autobot-slm declare the duplicate upstream on the failing host. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
✅ SSOT Configuration Compliance: Passing🎉 No hardcoded values detected that have SSOT config equivalents! |
2 tasks
This was referenced Jun 11, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thinking Path
After PR #9916 cleared the fresh-VM hostname/systemd bugs (#9914/#9915), I re-checked the wizard run: the native VM
node1721616826is now fully green (failed=0, ok=177), and the co-located SLM Manager got 200 tasks further but hit two deeper failures. Both are pre-existing co-location bugs in umbrella #9919 (deployment-install) and are the concrete root causes of the 4a→4c chain in #9281.What Changed
#9933 — nginx
duplicate upstream "autobot_backend"(Phase 4c)The frontend role (Phase 4b) enables a standalone
autobot-frontendsite before the co-location flag is set (the flag is resolved later, in Phase 4c), so bothsites-enabled/autobot-frontend:6andsites-enabled/autobot-slm:6declareupstream autobot_backend.include sites-enabled/*loads both →nginx -temerg. Fix: in the Phase 4c co-location play (provision-fleet-roles.yml), remove the conflictingsites-enabled/autobot-frontendsymlink when_is_slm_frontend_colocatedis true — beforenginx -t— and add that removal to the test/reload trigger conditions.#9934 — backend unhealthy: FD exhaustion
autobot-backend.service.j2had noLimitNOFILE— the only AutoBot unit missing it. The multi-worker backend (DB/Redis/ChromaDB pools + HTTP clients) hit the default fd ceiling on the busy co-located host:OSError: [Errno 24] Too many open files→socket.accept() out of system resource→ the#3687health gate never went green (120 retries / ~10 min). Fix:LimitNOFILE={{ backend_nofile_limit | default(65536) }}, matching the repo convention (slm-agent/frontend/ollama/browser/coordinator = 65535/65536).Verification
python3 yaml.safe_load_allon the playbook → OKansible-playbook --syntax-check playbooks/provision-fleet-roles.yml→ exit 0autobot-frontend+autobot-slmdeclare the duplicateupstream autobot_backend(autobot-backend.conf does not)LimitNOFILE=65536Model Used
Claude Opus 4.8 (1M context)
Part of umbrella #9919. Related: #9281.
Closes #9933
Closes #9934