Repository navigation
feat(slm): per-service health monitoring in SLM dashboard #1019
Description
Problem
SLM currently tracks nodes but has no visibility into individual services running on each node. Services can fail silently without any logs or alerts in the dashboard.
Current State
- SLM shows nodes:
03-AI-Stackonline, but doesn't show thatautobot-ai-stackservice (ChromaDB) is running while the AI Stack API (port 8080) is dead - No per-service health status in the dashboard
- Services fail silently (e.g.,
autobot-ai-stack.servicecrash-looped on Feb 18 with "No such file or directory" — nobody knew)
Requirements
Backend (SLM)
- SLM agent reports per-service status (running/stopped/failed/degraded) in heartbeat
- SLM backend stores service status per node
- API endpoint:
GET /api/nodes/{node_id}/servicesreturns service list with status - Service status includes: name, status, port, last_started, error_message (if failed)
Frontend (SLM)
- Node detail view shows service cards with status indicators
- Fleet overview shows service health counts per node
- Failed/stopped services show red indicators with last error
Logging
- All services must log to
/var/log/autobot/<service>.log - Failed services must have error output captured
- SLM agent reports last 5 lines of error log for failed services
Expected Service Inventory
| Node | Services | Ports |
|---|---|---|
| .19 SLM | autobot-slm-backend, postgresql, nginx, node_exporter | 443, 5432, 9100 |
| .20 Backend | autobot-backend, ollama, vnc | 8443, 11434, 6080 |
| .21 Frontend | nginx, node_exporter | 443, 9100 |
| .22 NPU | autobot-npu-worker, node_exporter | 8081, 9100 |
| .23 Redis | redis-server, node_exporter | 6379, 9100 |
| .24 AI Stack | autobot-ai-stack-api (8080), chromadb (8000), node_exporter | 8080, 8000, 9100 |
| .25 Browser | autobot-playwright, node_exporter | 3000, 9100 |
Acceptance Criteria
- SLM agent heartbeat includes service status array
- SLM dashboard shows per-service status on node detail
- Failed services show error context (not just "stopped")
- No service can fail silently without appearing in dashboard
Discovered During
Systematic debugging of .24 AI Stack VM — found services running/failing with no SLM visibility.
Activity
feat(slm): capture error context for failed services (#1019)
feat(slm): add service health counts to fleet overview + error contex…
mrveiss commentedon Feb 20, 2026
✅ Implemented per-service health monitoring in SLM dashboard
Changes (2 commits: ed1f6ff, 9f986c5):
Agent (health_collector.py):
_get_error_context()— captures last 5 journalctl lines for failed serviceserror_messagein heartbeatextra_data.discovered_servicesBackend (reconciler.py, schemas.py, nodes.py):
error_messagein Serviceextra_dataJSON fieldservice_summarytoNodeResponseschema (running/stopped/failed/total counts)list_nodesendpoint now queries service counts per node via grouped count queryServiceResponseschema includesextra_datafield for frontend accessFrontend (NodeCard.vue, NodeServicesPanel.vue, slm.ts):
SLMNodeandNodeServiceTypeScript types updated with new fieldsVerification:
extra_dataJSON column)fix(ansible): sync slm_agent role files with canonical agent code (#1623