Backend systemd unit missing LimitNOFILE → FD exhaustion (Errno 24) fails health gate on co-located/loaded hosts #9934

Summary

On the co-located SLM Manager the backend never passes Backend | Wait for backend to be healthy (#3687) (120 retries / ~10 min) and provisioning fails. The backend responds but reports unhealthy; backend-error.log spams:

OSError: [Errno 24] Too many open files
ERROR:asyncio:socket.accept() out of system resource

Root cause — missing LimitNOFILE

roles/backend/templates/autobot-backend.service.j2 has no LimitNOFILE directive, so the service inherits a low default. The backend is the heaviest AutoBot process (multiple uvicorn workers + DB/Redis/ChromaDB pools + HTTP clients), and on a busy co-located host it exhausts file descriptors → socket.accept() fails → health endpoint degraded.

Every other AutoBot unit already sets it (slm-agent, frontend, ollama, browser, coordinator, agent, loki, promtail = 65535/65536). The backend is the only one missing it. A lightly-loaded fresh VM passes the same check; the loaded co-located manager does not.

Fix

Add LimitNOFILE={{ backend_nofile_limit | default(65536) }} to the [Service] block, matching the repo convention (and the configurable pattern used by autobot-mcp-bridge@.service.j2).

Acceptance

• Backend unit sets LimitNOFILE (default 65536, overridable)
• Co-located SLM Manager backend passes the bug(backend): provisioning deploys startup-crashing backend — no smoke test after deploy #3687 health gate
• No Errno 24 Too many open files in backend-error.log under normal load

Note

If FD exhaustion persists after raising the limit, a descriptor leak should be investigated separately — but the absence of any limit is the immediate, certain gap.