Skip to content

feat: Role-based deployment architecture — standardized naming, scoped node deployment, manifest-driven lifecycle #926

Description

@mrveiss

Background

The AutoBot fleet currently has no consistent deployment architecture:

  • Nodes have wrong role directories (.21 has autobot-slm-frontend/ which belongs on .19)
  • .19 SLM server has the entire monorepo dumped into /opt/autobot/
  • .20 backend has a dual layout — old flat .py files coexist with autobot-user-backend/ subdir
  • Legacy directories remain on nodes (npu-worker/, browser-service/)
  • Repo folder names (autobot-user-frontend), Ansible role names (frontend), SLM role names (browser-service), and systemd service names (autobot-playwright) are all different
  • infrastructure/ doesn't follow the autobot-* convention
  • autobot-slm-agent code is buried inside autobot-slm-backend/slm/agent/ instead of being its own role
  • No roles for autobot-ai-stack, autobot-database, autobot-monitoring, autobot-slm-database as top-level code dirs
  • No pre-flight conflict checking before role assignment

Target Architecture

Rule: every role is a top-level directory

Each autobot-<role>/ contains code only. Each autobot-infrastructure/autobot-<role>/ contains the Ansible role, systemd templates, nginx configs, vault, and manifest.

Full role list

Role Node Notes
autobot-backend .20 FastAPI + Celery worker
autobot-frontend .21 nginx + Vue3 static build
autobot-ollama .20 Local LLM runtime (coexists with backend, no port conflict)
autobot-slm-backend .19 SLM fleet management API
autobot-slm-frontend .19 SLM admin dashboard
autobot-slm-database .19 PostgreSQL for SLM
autobot-monitoring .19 Prometheus + Grafana
autobot-npu-worker .22 Intel NPU inference
autobot-browser-worker .25 Playwright automation
autobot-ai-stack .24 ChromaDB + embeddings
autobot-database .23 Redis Stack + PostgreSQL
autobot-slm-agent all nodes SLM agent — deployed to every node
autobot-shared all nodes Shared Python utilities
autobot-infrastructure deploy-time only Renamed from infrastructure/

Node assignments (all 9 nodes)

Node Roles
.19 SLM slm-backend + slm-frontend + slm-database + monitoring + slm-agent
.20 Backend backend (+ celery) + ollama + slm-agent
.21 Frontend frontend + slm-agent
.22 NPU npu-worker + slm-agent
.23 Database database + slm-agent
.24 AI Stack ai-stack + slm-agent
.25 Browser browser-worker + slm-agent
.26 Reserved slm-agent (ready for assignment)
.27 Reserved slm-agent (ready for assignment)

Each node gets: autobot-<its-roles>/ + autobot-shared/ + autobot-infrastructure/autobot-<its-roles>/ only.


Design

Role Manifest (manifest.yml)

Every role has autobot-infrastructure/autobot-<role>/manifest.yml — the single source of truth SLM reads for deployment, configuration, health checking, and conflict validation.

role: autobot-frontend
description: "User-facing chat interface served via nginx over HTTPS"
target_node: frontend

deploy:
  source: autobot-frontend/
  destination: /opt/autobot/autobot-frontend/
  shared: true
  infrastructure: true

system_dependencies:
  - nginx
  - nodejs        # v20 via NodeSource

services:
  - name: autobot-frontend-build
    type: oneshot
    start_order: 1
  - name: autobot-frontend
    type: systemd
    system_service: nginx
    start_order: 2

ports:
  - port: 443
    protocol: https
    public: true
  - port: 80
    protocol: http
    public: true

health:
  endpoint: https://localhost/api/health
  interval: 30s
  timeout: 5s
  expected_status: 200

secrets:
  own:
    - tls_cert
    - tls_key
  shared:
    - backend_host
    - backend_port
    - service_auth_token

conflicts_with: []
warns_with: []
compatible_with:
  - autobot-slm-agent

depends_on:
  - autobot-backend

tls:
  cert: /etc/autobot/certs/autobot-frontend.crt
  key:  /etc/autobot/certs/autobot-frontend.key
  ca:   /etc/autobot/certs/ca.crt
  expiry_warning_days: 30
  auto_rotate: true
  rotate_days_before: 14
  reload_command: "systemctl reload nginx"

system_updates:
  policy: full
  reboot_strategy: immediate
  pre_update_action: none

docs: autobot-infrastructure/autobot-frontend/README.md

Coexistence validation

SLM pre-flight check before assigning a role to a node:

  1. Port scan — no two roles on same node can bind the same port
  2. Conflict check — hard block if conflicts_with match
  3. Warn check — operator warning if warns_with match
  4. Resource check — RAM minimum, GPU requirements vs node specs

Known hard conflicts:

  • autobot-frontend + autobot-slm-frontend → port 443
  • autobot-monitoring + autobot-browser-worker → port 3000
  • autobot-database + autobot-slm-database → port 5432

Secrets model (Hybrid A+B)

  • Per-role vault (autobot-infrastructure/autobot-<role>/vault.yml) — role-specific secrets (TLS key, role API key)
  • Shared vault (autobot-infrastructure/shared/vault.yml) — cross-role secrets (Redis password, CA cert, service auth tokens)
  • Systemd loads /etc/autobot/autobot-<role>.env (mode 640, owned by root:<role-user>)
  • Each role runs as its own Linux service account (autobot-backend, autobot-frontend, etc.)

Deployment flow (6 phases)

SLM triggers ansible-playbook directly when a role is assigned or updated:

Phase 1 — CLEAN      Remove legacy dirs, remove wrong-node roles
Phase 2 — DEPLOY     rsync role code + shared + infra from SLM cache → node
Phase 3 — SYSTEM DEPS  apt install from manifest.system_dependencies
Phase 4 — SECRETS    Render .env.j2 templates → /etc/autobot/<role>.env (640)
Phase 5 — SERVICES   Install systemd units, start in manifest.start_order
Phase 6 — VERIFY     Poll health endpoint, report UP_TO_DATE to SLM

Updates skip phases 1 and 3 (clean + system deps). Full provisioning on first assignment only.

Code sync flow

Git repo is on dev machine. SLM server .19 is the distribution hub.

git push
  → post-commit hook detects changed role dirs
  → rsync changed roles only → /opt/autobot/cache/<role>/ on .19
  → POST /api/code-source/notify (changed_roles + commit hash)
  → SLM marks affected nodes OUTDATED
  → Operator triggers update (or auto)
  → Ansible deploys from cache → node (scoped to that role's dirs only)
  → Node reports UP_TO_DATE with commit hash

Per-role version tracking in SLM DB — each role on each node tracks its own commit independently.

Manual upload also supported (tarball via SLM GUI) for air-gapped scenarios.

System update flow (apt)

Independent from code updates. Per-role manifest declares update policy:

  • .21/.25/.26/.27 (stateless) → full, immediate reboot
  • .19/.20/.22/.24 (stateful) → security, scheduled reboot
  • .23 (database) → manual only

SLM daily cert + package check alerts when:

  • Node has pending security updates
  • Node has pending full updates
  • Reboot required flag detected

Security model

Per-role service accounts — each role runs as autobot-<role> Linux user. A compromised role process cannot read another role's secrets.

TLS everywhere — internal CA on SLM server. Per-node certs signed at provision time. Auto-rotation 14 days before expiry via systemctl reload (zero downtime).

SSH key rotation with autobot_admin safety net:

Phase K0 — Enable autobot_admin + verify SSH access on all nodes
Phase K1 — Generate new autobot keypair
Phase K2 — Add new key (both old + new valid — overlap window)
           If any node fails: autobot_admin fixes it live
Phase K3 — Remove old key after all nodes verified
Phase K4 — Final verification: new key works, old key rejected
Phase K5 — Disable autobot_admin (lock account)
Phase K6 — Audit log: rotation time, operator, duration admin was active

autobot_admin key is never rotated at the same time as autobot key. Its own rotation uses break-glass as the safety net.

CA rotation — rolling with transition bundle (old + new CA both trusted during window, then old removed). Zero downtime.

Firewall — generated automatically from manifests. If role A depends_on role B, Ansible adds the corresponding UFW allow rule on B's node for A's IP.

Documentation structure

docs/
  architecture/
    ROLE_ARCHITECTURE.md     ← master doc
    COEXISTENCE_MATRIX.md
    NETWORK_TOPOLOGY.md
    UPDATE_FLOWS.md
  runbooks/
    DEPLOY_NEW_NODE.md
    ASSIGN_ROLE.md
    CODE_UPDATE.md
    SYSTEM_UPDATE.md
    ROTATE_CERTS.md
    ROTATE_SSH_KEYS.md
    EMERGENCY_RECOVERY.md

autobot-infrastructure/autobot-<role>/
  manifest.yml               ← authoritative spec (IS the documentation)
  README.md                  ← deploy notes, gotchas, local dev setup

Implementation Phases

Phase 1 — Repo restructuring

  • Rename autobot-user-backend/ → autobot-backend/
  • Rename autobot-user-frontend/ → autobot-frontend/
  • Rename infrastructure/ → autobot-infrastructure/
  • Rename autobot-infrastructure/autobot-user-backend/ → autobot-infrastructure/autobot-backend/
  • Rename autobot-infrastructure/autobot-user-frontend/ → autobot-infrastructure/autobot-frontend/
  • Rename autobot-infrastructure/autobot-db-stack/ → autobot-infrastructure/autobot-database/
  • Extract autobot-slm-backend/slm/agent/ → autobot-slm-agent/ (top-level)
  • Create stub dirs: autobot-slm-database/, autobot-monitoring/, autobot-ai-stack/, autobot-database/
  • Update all imports, path references, CLAUDE.md, sync scripts

Phase 2 — Role manifests

  • Define manifest schema (Pydantic model in SLM backend)
  • Write manifest.yml for all 11 roles + slm-agent + shared
  • Write README.md for all 11 roles using template
  • Write docs/architecture/ROLE_ARCHITECTURE.md
  • Write docs/architecture/COEXISTENCE_MATRIX.md

Phase 3 — SLM manifest integration

  • SLM reads manifest at role assignment → pre-flight port/conflict/resource check
  • SLM reads manifest for health endpoint polling
  • SLM reads manifest for service start/stop ordering
  • SLM reads manifest for cert expiry monitoring
  • SLM reads manifest for system update policy per node
  • Per-role version tracking in DB (node_code_versions table)

Phase 4 — Ansible refactor (scoped deployment)

  • Each Ansible role deploys ONLY that role's dirs (no full monorepo)
  • Add Phase 1 CLEAN task to each role (removes legacy + wrong-node dirs)
  • Per-role service accounts created at provision time
  • Per-role /etc/autobot/<role>.env with correct ownership (640)
  • Firewall rules generated from manifest depends_on + ports
  • UFW loopback-only enforcement for loopback_only: true ports
  • system-update.yml playbook with policy-aware apt phases
  • rotate-certs.yml playbook with zero-downtime reload
  • rotate-ssh-keys.yml playbook with autobot_admin safety net

Phase 5 — Code sync refactor

  • Post-commit hook detects changed role dirs (git diff)
  • Hook rsyncs only changed roles to /opt/autobot/cache/ on SLM
  • POST /api/code-source/notify accepts changed_roles[] + commit hash
  • SLM marks only affected nodes OUTDATED (per-role)
  • Ansible update-node.yml deploys from cache (scoped, skips clean + system deps)
  • Manual upload endpoint (POST /api/code-source/upload-package) for air-gapped use
  • node_code_versions DB table — per-role per-node commit tracking

Phase 6 — Node cleanup

  • Remove autobot-slm-frontend/ from .21
  • Remove full monorepo from .19 — leave only SLM-role dirs
  • Remove old flat .py files from .20 — keep only autobot-backend/
  • Remove legacy npu-worker/ from .22
  • Remove legacy browser-service/ from .25
  • Remove legacy ai-stack/ from .24 — replace with autobot-ai-stack/
  • Remove loose config/, data/, logs/, scripts/ from node roots
  • Verify all nodes match target state

Phase 7 — Security hardening

  • Per-role Linux service accounts on all nodes
  • Internal CA + per-node cert generation in Ansible
  • Cert expiry monitoring in SLM dashboard
  • autobot_admin safety net baked into rotate-ssh-keys.yml
  • Break-glass key split storage documented in EMERGENCY_RECOVERY.md
  • docs/runbooks/ROTATE_SSH_KEYS.md
  • docs/runbooks/ROTATE_CERTS.md

Phase 8 — Documentation

  • docs/architecture/ROLE_ARCHITECTURE.md
  • docs/architecture/COEXISTENCE_MATRIX.md
  • docs/architecture/NETWORK_TOPOLOGY.md
  • docs/architecture/UPDATE_FLOWS.md
  • All 7 runbooks in docs/runbooks/
  • manifest.yml + README.md for all 11 roles
  • Update CLAUDE.md with new dir names and sync paths

Acceptance Criteria

  • ls /opt/autobot/ on each node shows only that node's assigned roles + shared + infrastructure
  • No legacy dirs (npu-worker/, browser-service/, ai-stack/, flat .py files) on any node
  • SLM blocks assignment of autobot-frontend + autobot-slm-frontend to same node (port conflict)
  • git push → only changed role dirs sync to SLM cache → only affected nodes marked OUTDATED
  • Ansible update-node.yml deploys only the role's dirs to the node (nothing else)
  • systemctl status autobot-<role> uses consistent naming on every node
  • Per-role .env files exist at /etc/autobot/autobot-<role>.env with mode 640
  • rotate-ssh-keys.yml enables autobot_admin, rotates, verifies, disables — all in one playbook
  • All 11 manifests exist and pass SLM schema validation
  • All 7 runbooks written and reviewed
  • vue-tsc --build and ruff check pass after Phase 1 renames

Labels

  • enhancement
  • infrastructure
  • ansible
  • architecture

Size estimate

Large epic — implement in phases. Each phase is independently mergeable.

Activity

  1. mrveiss commented on Feb 18, 2026

    @mrveiss
    OwnerAuthor

    Addition: Migration Page (all roles) + Backup/Restore Integration

    Current state

    Migration tab (/orchestration/migration) exists in OrchestrationView.vue tab 4 but only has one endpoint:

    • POST /api/orchestration/services/{service_name}/migrate — moves a service between nodes

    Backup/Restore is fully implemented but separate:

    • /backups route → BackupsView.vue (559 lines)
    • /api/stateful/backups* + /api/stateful/replications* (38 endpoints)
    • Covers Redis, PostgreSQL, ChromaDB, SQLite

    Neither is connected to the new role architecture.


    Migration page — what it needs to show

    The migration tab must become the control center for the role-based architecture transition and ongoing role management.

    /orchestration/migration
    
    ┌─────────────────────────────────────────────────────────────────┐
    │  Role Migration Status                          [ Migrate All ] │
    ├──────────────────┬──────┬───────────┬───────────┬──────────────┤
    │ Role             │ Node │ Current   │ Target    │ Action       │
    ├──────────────────┼──────┼───────────┼───────────┼──────────────┤
    │ autobot-backend  │ .20  │ WRONG_NAME│ READY     │ [Rename]     │
    │ autobot-frontend │ .21  │ WRONG_NAME│ READY     │ [Rename]     │
    │ autobot-frontend │ .21  │ EXTRA_DIR │ CLEAN     │ [Clean]      │
    │  slm-frontend    │      │ (on .21)  │           │              │
    │ autobot-ollama   │ .20  │ MISSING   │ DEPLOY    │ [Deploy]     │
    │ autobot-database │ .23  │ MISSING   │ DEPLOY    │ [Deploy]     │
    │ autobot-ai-stack │ .24  │ WRONG_NAME│ READY     │ [Rename]     │
    │ autobot-slm-agent│ all  │ PARTIAL   │ DEPLOY    │ [Deploy All] │
    │ ...              │      │           │           │              │
    └──────────────────┴──────┴───────────┴───────────┴──────────────┘
    
    Status badges:
      UP_TO_DATE   — deployed, correct name, correct node
      OUTDATED     — deployed but behind HEAD commit
      WRONG_NAME   — deployed but old name (autobot-user-frontend → autobot-frontend)
      WRONG_NODE   — deployed on incorrect node (slm-frontend on .21)
      MISSING      — role not yet deployed to target node
      LEGACY_PRESENT — old dir exists alongside new (npu-worker/ + autobot-npu-worker/)
      EXTRA_DIR    — dir present that should not be on this node
    

    Each row shows:

    • Role name
    • Assigned node
    • Current state (detected by Ansible fact-gathering from /opt/autobot/ on each node)
    • Target state (from manifest.yml target_node + deploy.destination)
    • Action button (context-sensitive to current state)

    Bulk actions:

    • [ Migrate All ] — runs all pending actions in dependency order
    • [ Backup All ] — snapshots all nodes before migration
    • [ Dry Run ] — shows what Ansible would change without executing

    Backup/Restore — integrated into migration flow

    Backup is already implemented. It needs two additions:

    1. Pre-migration auto-backup

    Any destructive migration action (CLEAN, RENAME, WRONG_NODE removal) automatically triggers a backup before executing:

    Operator clicks [Clean] on .21 (remove autobot-slm-frontend/)
    
      Step 1 — AUTO BACKUP
        Trigger: POST /api/stateful/backups
        Scope: .21 node, autobot-frontend role data
        Wait for completion (or skip if no data to back up for this role)
    
      Step 2 — EXECUTE CLEAN
        Ansible: remove /opt/autobot/autobot-slm-frontend/ from .21
    
      Step 3 — VERIFY
        Confirm dir is gone, health endpoint still passing
    
      If Step 2 or 3 fails:
        [Restore] button appears in migration row
        Restore from backup taken in Step 1
    

    2. Role-aware backup manifest

    Extend the backup model to understand roles:

    # What each role backs up before migration
    autobot-backend:   [config, /etc/autobot/autobot-backend.env, /opt/autobot/autobot-backend/data/]
    autobot-frontend:  [config, /etc/autobot/autobot-frontend.env]   # stateless, no data
    autobot-database:  [redis-rdb, postgresql-dump]                   # heaviest backup
    autobot-ollama:    [config, model-list]                           # models re-downloaded, not backed up
    autobot-slm-backend: [config, postgresql-dump]
    autobot-monitoring: [config, prometheus-data, grafana-dashboards]

    Added to manifest.yml:

    backup:
      pre_migration: true          # always backup before any migration action
      scope:
        - type: config             # /etc/autobot/<role>.env
        - type: data               # role-specific data dirs
          path: data/
        - type: database           # triggers stateful.py backup for this service
          service: postgresql
      exclude_from_backup:
        - node_modules/
        - dist/
        - __pycache__/

    3. Migration page backup panel

    Below the role status table, a backup history panel:

    Recent Migration Backups
      2026-02-18 14:32  .21 pre-clean     autobot-frontend   2.1 MB   ✅  [Restore]
      2026-02-18 14:28  .20 pre-rename    autobot-backend    45 MB    ✅  [Restore]
      2026-02-18 09:00  .23 manual        autobot-database   1.2 GB   ✅  [Restore]
    

    Restore action:

    • Stops the role's services
    • Restores files + database from backup
    • Restarts services
    • Marks node as pre-migration state in SLM

    New implementation phases for Issue #926

    Phase 9 — Migration page: all roles

    • Backend: GET /api/orchestration/migration/status — scans all nodes, compares /opt/autobot/ contents against manifests, returns per-role status (UP_TO_DATE/WRONG_NAME/MISSING/etc.)
    • Backend: POST /api/orchestration/migration/plan — returns ordered list of actions needed (dry run)
    • Backend: POST /api/orchestration/migration/execute — runs migration actions with pre-backup, WebSocket progress
    • Frontend: Migration tab shows full role × node status matrix
    • Frontend: Per-role action buttons (Rename, Clean, Deploy, Migrate to correct node)
    • Frontend: Bulk [ Migrate All ] with dependency ordering
    • Frontend: Real-time progress via existing WebSocket

    Phase 10 — Backup/Restore integration

    • Add backup section to all 11 role manifests (scope, pre_migration flag)
    • Backend: Pre-migration auto-backup hook in migration execute endpoint
    • Backend: Backup linked to migration action (backup_id stored with migration event)
    • Backend: POST /api/orchestration/migration/{action_id}/restore — restore from pre-migration backup
    • Frontend: [Restore] button appears on failed/completed migration rows
    • Frontend: Migration backup history panel below status table
    • Extend BackupsView.vue to show migration-triggered backups separately from manual backups

    Updated acceptance criteria

    • /orchestration/migration shows all 11 roles + slm-agent across all 9 nodes with correct status
    • Any CLEAN or WRONG_NODE action auto-triggers a backup before executing
    • [Restore] button available for any migration action that has a linked backup
    • Dry run shows exactly what Ansible would change without executing
    • Bulk [ Migrate All ] completes in dependency order with real-time WebSocket progress
  2. mrveiss commented on Feb 18, 2026

    @mrveiss
    OwnerAuthor

    ✅ Phase 1 Complete — commit 00ae80e1

    Directory renames completed:

    • autobot-user-backend/ → autobot-backend/
    • autobot-user-frontend/ → autobot-frontend/
    • infrastructure/ → autobot-infrastructure/
    • autobot-infrastructure/autobot-user-backend/ → autobot-infrastructure/autobot-backend/
    • autobot-infrastructure/autobot-user-frontend/ → autobot-infrastructure/autobot-frontend/
    • autobot-infrastructure/autobot-db-stack/ → autobot-infrastructure/autobot-database/

    New directories extracted/created:

    • autobot-slm-agent/ (extracted from autobot-slm-backend/slm/agent/)
    • autobot-slm-database/ (stub)
    • autobot-monitoring/ (stub)
    • autobot-ai-stack/ (stub)
    • autobot-database/ (stub)

    Config files updated: .pre-commit-config.yaml, .flake8, .bandit, .git/hooks/pre-commit.legacy, .git/hooks/post-checkout, Ansible roles + playbooks, role_registry.py, service_orchestrator.py, path_constants.py, CLAUDE.md, README.md

    All pre-commit hooks pass. Pushed to Dev_new_gui.

    Next: Phase 2 — Role manifests (manifest.yml for all 11 roles)

  3. mrveiss commented on Feb 18, 2026

    @mrveiss
    OwnerAuthor

    ✅ Phase 2 Complete — Role Manifests, Schema & Architecture Docs

    Commit: ed6f133

    Delivered

    Pydantic RoleManifest schema (autobot-slm-backend/models/manifest.py):

    • Full Pydantic v2 model with field validators and enums
    • @field_validator("role") enforces autobot- prefix
    • Helper methods: get_service_account(), get_env_file(), port_numbers(), hard_conflicts(), metadata()
    • Supporting models: ManifestDeploy, ManifestService, ManifestPort, ManifestHealth, ManifestSecrets, ManifestTLS, ManifestSystemUpdates, ManifestCoexistence
    • Enums: UpdatePolicy, RebootStrategy, ServiceType, ProtocolType

    manifest.yml for all 13 roles (single source of truth):

    Role Target Key Ports
    autobot-backend .20 8443 HTTPS, 5555 loopback
    autobot-frontend .21 443 public
    autobot-slm-backend .19 loopback 8000 + nginx 443
    autobot-slm-frontend .19 (via slm-backend nginx)
    autobot-slm-database .19 5432 loopback
    autobot-monitoring .19 9090/3000 loopback
    autobot-npu-worker .22 8081
    autobot-browser-worker .25 3000 (service=autobot-playwright)
    autobot-ai-stack .24 8080
    autobot-database .23 6379/8001/5432 (manual updates)
    autobot-ollama .20 11434 loopback
    autobot-slm-agent all 9100 node_exporter
    autobot-shared all (library, no services)

    README.md for all 13 roles — deploy commands, health checks, secrets, gotchas, Ansible role reference

    Architecture docs:

    • docs/architecture/ROLE_ARCHITECTURE.md — role catalogue, node assignments, manifest schema reference, 6-phase deployment flow, code sync flow, security model
    • docs/architecture/COEXISTENCE_MATRIX.md — 13×13 compatibility matrix, conflict/warning explanations, port conflict reference

    Conflicts Identified

    • ❌ autobot-frontend ↔ autobot-slm-frontend (both bind port 443 via nginx)
    • ⚠️ autobot-monitoring ↔ autobot-browser-worker (Grafana and browser-worker both default to port 3000)
  4. mrveiss commented on Feb 18, 2026

    @mrveiss
    OwnerAuthor

    Phase 3 Complete ✅

    Commit: 2737ac53

    Changes

    New: autobot-slm-backend/services/manifest_loader.py

    • ManifestLoader singleton with 5-min TTL cache
    • Loads RoleManifest Pydantic objects from autobot-infrastructure/<role>/manifest.yml
    • Convenience accessors: get_health_endpoint, get_port_numbers, get_hard_conflicts, get_soft_warnings, get_update_policy, get_service_order, get_tls_auto_rotate, node_update_policy

    Modified: autobot-slm-backend/api/nodes.py

    • _run_preflight() helper — port overlap + hard coexistence conflict detection before role assignment
    • assign_role_to_node now calls preflight; returns HTTP 409 with reasons on hard conflicts
    • GET /{node_id}/roles/preflight?role=<name> — query preflight without modifying state
    • GET /{node_id}/update-policy — effective system update policy (most restrictive across all roles)
    • GET /{node_id}/service-order — ordered service list for all assigned roles

    Modified: autobot-slm-backend/services/reconciler.py

    • Manifest health polling every 60s (_poll_manifest_health)
    • Cert expiry daily check (_check_cert_expiry) with per-role rotate_days_before threshold

    Modified: autobot-slm-backend/models/schemas.py

    • 5 new schemas: PreflightConflict, PreflightResult, ServiceOrderEntry, NodeServiceOrderResponse, UpdatePolicyResponse

    Modified: autobot-slm-backend/main.py

    • init_manifest_loader() called in lifespan startup

    Phase 3 checklist

    • ✅ SLM reads manifest at role assignment → pre-flight port/conflict check
    • ✅ SLM reads manifest for health endpoint polling
    • ✅ SLM reads manifest for service start/stop ordering
    • ✅ SLM reads manifest for cert expiry monitoring
    • ✅ SLM reads manifest for system update policy per node
    • ✅ Per-role version tracking in DB (existing NodeRole table has current_version + last_synced_at — no new table needed)

    Ready for Phase 4.

  5. mrveiss commented on Feb 18, 2026

    @mrveiss
    OwnerAuthor

    Phase 4 Complete — Scoped Ansible Deployment ✅

    Commits

    • 1e3b2032 — Phase 4 core: clean task files + defaults for all 7 roles
    • 122173f5 — Phase 4 complete: remaining main.yml updates + 3 new playbooks

    Changes

    Clean task files (one per role)
    Each role now begins with import_tasks: clean.yml (tagged clean) that removes legacy/wrong-node directories from /opt/autobot/. This enforces the invariant that a node only contains its own role's code.

    Per-role Linux service accounts

    • autobot-npu (system user, no shell/home) on NPU Worker
    • autobot-ai on AI Stack
    • autobot-browser on Browser Worker
    • autobot-backend account created (service still runs as autobot — full migration in Phase 7)
    • autobot-frontend account created (same deferral)

    Per-role scoped env files
    /etc/autobot/autobot-<role>.env (mode 640, owner root, group ) deployed by each role's Ansible tasks. Service now reads role-scoped env in addition to legacy .env.

    Fixed stale install paths

    • npu_install_dir: /opt/autobot/npu-worker → /opt/autobot/autobot-npu-worker
    • ai_install_dir: /opt/autobot/ai-stack → /opt/autobot/autobot-ai-stack

    New playbooks (all pass --syntax-check)

    • playbooks/system-update.yml — policy-aware apt updates (full/security/manual) in dependency order (SLM → Backend → DB → AI → NPU → Browser → Frontend) with reboot support
    • playbooks/rotate-certs.yml — zero-downtime TLS cert rotation with nginx graceful reload + uvicorn restart + fingerprint verification
    • playbooks/rotate-ssh-keys.yml — 7-phase key rotation (K0-K6) with temp safety-net key to prevent lockouts

    Phase 4 Checklist

    • Per-role clean task files (remove wrong-node legacy dirs)
    • Per-role Linux service accounts with /etc/autobot/autobot-\<role\>.env (mode 640)
    • Firewall rules from manifest ports already handled by existing UFW tasks in each role
    • system-update.yml playbook with policy-aware apt phases
    • rotate-certs.yml playbook with zero-downtime reload
    • rotate-ssh-keys.yml playbook with autobot_admin safety net (K0-K6 phases)
  6. mrveiss commented on Feb 18, 2026

    @mrveiss
    OwnerAuthor

    ✅ Phase 5 Complete — Per-Role Code Sync Refactor

    Commit: 304854f6

    Changes

    autobot-slm-backend/models/database.py

    • Added NodeCodeVersion model with (node_id, role_name) unique constraint for per-role per-node commit tracking

    autobot-slm-backend/api/code_source.py (+306 lines)

    • CodeNotification extended with changed_roles: List[str] = []
    • _update_code_sync_state delegates to _mark_roles_outdated (scoped) or _mark_all_nodes_outdated (backwards compat)
    • _mark_roles_outdated — marks only nodes with the changed roles assigned as OUTDATED + upserts NodeCodeVersion rows
    • _upsert_node_code_version — insert/update per-role per-node version entry
    • GET /api/code-source/node-code-versions — list per-role version status (filterable by node_id, role_name)
    • POST /api/code-source/upload-package — air-gapped multipart .tar.gz upload, extracts to /opt/autobot/cache/<role>/, marks nodes OUTDATED

    scripts/hooks/slm-post-commit (rewritten)

    • Detects which autobot-* role dirs changed via git diff HEAD~1 HEAD --name-only
    • Parallel rsync of each changed role to /opt/autobot/cache/<role>/ on SLM
    • wait for all rsyncs to complete before notifying (cache ready before SLM triggers deploy)
    • changed_roles JSON array in notify payload
    • Falls back to empty changed_roles (all nodes OUTDATED) if no role dirs changed

    autobot-slm-backend/ansible/playbooks/update-node.yml (new, 262 lines)

    • Cache-based scoped deploy — skips Phase 1 (clean) and Phase 3 (system deps)
    • Play 1: Authenticate with SLM API, resolve role list from variable or cache dirs
    • Play 2: synchronize from /opt/autobot/cache/<role>/ → /opt/autobot/<role>/ on target node; rebuild frontend dist if needed; restart affected services
    • Play 3: Mark node UP_TO_DATE via POST /api/code-sync/nodes/{node_id}/mark-synced

    Phase 5 checklist

    • Post-commit hook detects changed role dirs (git diff)
    • Hook rsyncs only changed roles to /opt/autobot/cache/ on SLM
    • POST /api/code-source/notify accepts changed_roles[] + commit hash
    • SLM marks only affected nodes OUTDATED (per-role)
    • Ansible update-node.yml deploys from cache (scoped, skips clean + system deps)
    • Manual upload endpoint (POST /api/code-source/upload-package) for air-gapped use
    • node_code_versions DB table — per-role per-node commit tracking
  7. added a commit that references this issue on Feb 18, 2026
  8. 12 remaining items

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions