Repository navigation
feat: Role-based deployment architecture — standardized naming, scoped node deployment, manifest-driven lifecycle #926
Description
Activity
Addition: Migration Page (all roles) + Backup/Restore Integration
Current state
Migration tab (
/orchestration/migration) exists in OrchestrationView.vue tab 4 but only has one endpoint:POST /api/orchestration/services/{service_name}/migrate— moves a service between nodes
Backup/Restore is fully implemented but separate:
/backupsroute →BackupsView.vue(559 lines)/api/stateful/backups*+/api/stateful/replications*(38 endpoints)- Covers Redis, PostgreSQL, ChromaDB, SQLite
Neither is connected to the new role architecture.
Migration page — what it needs to show
The migration tab must become the control center for the role-based architecture transition and ongoing role management.
/orchestration/migration ┌─────────────────────────────────────────────────────────────────┐ │ Role Migration Status [ Migrate All ] │ ├──────────────────┬──────┬───────────┬───────────┬──────────────┤ │ Role │ Node │ Current │ Target │ Action │ ├──────────────────┼──────┼───────────┼───────────┼──────────────┤ │ autobot-backend │ .20 │ WRONG_NAME│ READY │ [Rename] │ │ autobot-frontend │ .21 │ WRONG_NAME│ READY │ [Rename] │ │ autobot-frontend │ .21 │ EXTRA_DIR │ CLEAN │ [Clean] │ │ slm-frontend │ │ (on .21) │ │ │ │ autobot-ollama │ .20 │ MISSING │ DEPLOY │ [Deploy] │ │ autobot-database │ .23 │ MISSING │ DEPLOY │ [Deploy] │ │ autobot-ai-stack │ .24 │ WRONG_NAME│ READY │ [Rename] │ │ autobot-slm-agent│ all │ PARTIAL │ DEPLOY │ [Deploy All] │ │ ... │ │ │ │ │ └──────────────────┴──────┴───────────┴───────────┴──────────────┘ Status badges: UP_TO_DATE — deployed, correct name, correct node OUTDATED — deployed but behind HEAD commit WRONG_NAME — deployed but old name (autobot-user-frontend → autobot-frontend) WRONG_NODE — deployed on incorrect node (slm-frontend on .21) MISSING — role not yet deployed to target node LEGACY_PRESENT — old dir exists alongside new (npu-worker/ + autobot-npu-worker/) EXTRA_DIR — dir present that should not be on this nodeEach row shows:
- Role name
- Assigned node
- Current state (detected by Ansible fact-gathering from
/opt/autobot/on each node) - Target state (from manifest.yml
target_node+deploy.destination) - Action button (context-sensitive to current state)
Bulk actions:
[ Migrate All ]— runs all pending actions in dependency order[ Backup All ]— snapshots all nodes before migration[ Dry Run ]— shows what Ansible would change without executing
Backup/Restore — integrated into migration flow
Backup is already implemented. It needs two additions:
1. Pre-migration auto-backup
Any destructive migration action (CLEAN, RENAME, WRONG_NODE removal) automatically triggers a backup before executing:
Operator clicks [Clean] on .21 (remove autobot-slm-frontend/) Step 1 — AUTO BACKUP Trigger: POST /api/stateful/backups Scope: .21 node, autobot-frontend role data Wait for completion (or skip if no data to back up for this role) Step 2 — EXECUTE CLEAN Ansible: remove /opt/autobot/autobot-slm-frontend/ from .21 Step 3 — VERIFY Confirm dir is gone, health endpoint still passing If Step 2 or 3 fails: [Restore] button appears in migration row Restore from backup taken in Step 12. Role-aware backup manifest
Extend the backup model to understand roles:
# What each role backs up before migration autobot-backend: [config, /etc/autobot/autobot-backend.env, /opt/autobot/autobot-backend/data/] autobot-frontend: [config, /etc/autobot/autobot-frontend.env] # stateless, no data autobot-database: [redis-rdb, postgresql-dump] # heaviest backup autobot-ollama: [config, model-list] # models re-downloaded, not backed up autobot-slm-backend: [config, postgresql-dump] autobot-monitoring: [config, prometheus-data, grafana-dashboards]
Added to manifest.yml:
backup: pre_migration: true # always backup before any migration action scope: - type: config # /etc/autobot/<role>.env - type: data # role-specific data dirs path: data/ - type: database # triggers stateful.py backup for this service service: postgresql exclude_from_backup: - node_modules/ - dist/ - __pycache__/
3. Migration page backup panel
Below the role status table, a backup history panel:
Recent Migration Backups 2026-02-18 14:32 .21 pre-clean autobot-frontend 2.1 MB ✅ [Restore] 2026-02-18 14:28 .20 pre-rename autobot-backend 45 MB ✅ [Restore] 2026-02-18 09:00 .23 manual autobot-database 1.2 GB ✅ [Restore]Restore action:
- Stops the role's services
- Restores files + database from backup
- Restarts services
- Marks node as pre-migration state in SLM
New implementation phases for Issue #926
Phase 9 — Migration page: all roles
- Backend:
GET /api/orchestration/migration/status— scans all nodes, compares/opt/autobot/contents against manifests, returns per-role status (UP_TO_DATE/WRONG_NAME/MISSING/etc.) - Backend:
POST /api/orchestration/migration/plan— returns ordered list of actions needed (dry run) - Backend:
POST /api/orchestration/migration/execute— runs migration actions with pre-backup, WebSocket progress - Frontend: Migration tab shows full role × node status matrix
- Frontend: Per-role action buttons (Rename, Clean, Deploy, Migrate to correct node)
- Frontend: Bulk
[ Migrate All ]with dependency ordering - Frontend: Real-time progress via existing WebSocket
Phase 10 — Backup/Restore integration
- Add
backupsection to all 11 role manifests (scope, pre_migration flag) - Backend: Pre-migration auto-backup hook in migration execute endpoint
- Backend: Backup linked to migration action (backup_id stored with migration event)
- Backend:
POST /api/orchestration/migration/{action_id}/restore— restore from pre-migration backup - Frontend: [Restore] button appears on failed/completed migration rows
- Frontend: Migration backup history panel below status table
- Extend
BackupsView.vueto show migration-triggered backups separately from manual backups
Updated acceptance criteria
-
/orchestration/migrationshows all 11 roles + slm-agent across all 9 nodes with correct status - Any CLEAN or WRONG_NODE action auto-triggers a backup before executing
- [Restore] button available for any migration action that has a linked backup
- Dry run shows exactly what Ansible would change without executing
- Bulk
[ Migrate All ]completes in dependency order with real-time WebSocket progress
- added a commit that references this issue
on Feb 18, 2026 ✅ Phase 1 Complete — commit
00ae80e1Directory renames completed:
autobot-user-backend/→autobot-backend/autobot-user-frontend/→autobot-frontend/infrastructure/→autobot-infrastructure/autobot-infrastructure/autobot-user-backend/→autobot-infrastructure/autobot-backend/autobot-infrastructure/autobot-user-frontend/→autobot-infrastructure/autobot-frontend/autobot-infrastructure/autobot-db-stack/→autobot-infrastructure/autobot-database/
New directories extracted/created:
autobot-slm-agent/(extracted fromautobot-slm-backend/slm/agent/)autobot-slm-database/(stub)autobot-monitoring/(stub)autobot-ai-stack/(stub)autobot-database/(stub)
Config files updated:
.pre-commit-config.yaml,.flake8,.bandit,.git/hooks/pre-commit.legacy,.git/hooks/post-checkout, Ansible roles + playbooks,role_registry.py,service_orchestrator.py,path_constants.py,CLAUDE.md,README.mdAll pre-commit hooks pass. Pushed to
Dev_new_gui.Next: Phase 2 — Role manifests (
manifest.ymlfor all 11 roles)- added a commit that references this issue
on Feb 18, 2026 ✅ Phase 2 Complete — Role Manifests, Schema & Architecture Docs
Commit: ed6f133
Delivered
Pydantic RoleManifest schema (
autobot-slm-backend/models/manifest.py):- Full Pydantic v2 model with field validators and enums
@field_validator("role")enforcesautobot-prefix- Helper methods:
get_service_account(),get_env_file(),port_numbers(),hard_conflicts(),metadata() - Supporting models:
ManifestDeploy,ManifestService,ManifestPort,ManifestHealth,ManifestSecrets,ManifestTLS,ManifestSystemUpdates,ManifestCoexistence - Enums:
UpdatePolicy,RebootStrategy,ServiceType,ProtocolType
manifest.yml for all 13 roles (single source of truth):
Role Target Key Ports autobot-backend .20 8443 HTTPS, 5555 loopback autobot-frontend .21 443 public autobot-slm-backend .19 loopback 8000 + nginx 443 autobot-slm-frontend .19 (via slm-backend nginx) autobot-slm-database .19 5432 loopback autobot-monitoring .19 9090/3000 loopback autobot-npu-worker .22 8081 autobot-browser-worker .25 3000 (service=autobot-playwright) autobot-ai-stack .24 8080 autobot-database .23 6379/8001/5432 (manual updates) autobot-ollama .20 11434 loopback autobot-slm-agent all 9100 node_exporter autobot-shared all (library, no services) README.md for all 13 roles — deploy commands, health checks, secrets, gotchas, Ansible role reference
Architecture docs:
docs/architecture/ROLE_ARCHITECTURE.md— role catalogue, node assignments, manifest schema reference, 6-phase deployment flow, code sync flow, security modeldocs/architecture/COEXISTENCE_MATRIX.md— 13×13 compatibility matrix, conflict/warning explanations, port conflict reference
Conflicts Identified
- ❌
autobot-frontend↔autobot-slm-frontend(both bind port 443 via nginx) ⚠️ autobot-monitoring↔autobot-browser-worker(Grafana and browser-worker both default to port 3000)
- added a commit that references this issue
on Feb 18, 2026 Phase 3 Complete ✅
Commit:
2737ac53Changes
New:
autobot-slm-backend/services/manifest_loader.pyManifestLoadersingleton with 5-min TTL cache- Loads
RoleManifestPydantic objects fromautobot-infrastructure/<role>/manifest.yml - Convenience accessors:
get_health_endpoint,get_port_numbers,get_hard_conflicts,get_soft_warnings,get_update_policy,get_service_order,get_tls_auto_rotate,node_update_policy
Modified:
autobot-slm-backend/api/nodes.py_run_preflight()helper — port overlap + hard coexistence conflict detection before role assignmentassign_role_to_nodenow calls preflight; returns HTTP 409 with reasons on hard conflictsGET /{node_id}/roles/preflight?role=<name>— query preflight without modifying stateGET /{node_id}/update-policy— effective system update policy (most restrictive across all roles)GET /{node_id}/service-order— ordered service list for all assigned roles
Modified:
autobot-slm-backend/services/reconciler.py- Manifest health polling every 60s (
_poll_manifest_health) - Cert expiry daily check (
_check_cert_expiry) with per-role rotate_days_before threshold
Modified:
autobot-slm-backend/models/schemas.py- 5 new schemas:
PreflightConflict,PreflightResult,ServiceOrderEntry,NodeServiceOrderResponse,UpdatePolicyResponse
Modified:
autobot-slm-backend/main.pyinit_manifest_loader()called in lifespan startup
Phase 3 checklist
- ✅ SLM reads manifest at role assignment → pre-flight port/conflict check
- ✅ SLM reads manifest for health endpoint polling
- ✅ SLM reads manifest for service start/stop ordering
- ✅ SLM reads manifest for cert expiry monitoring
- ✅ SLM reads manifest for system update policy per node
- ✅ Per-role version tracking in DB (existing
NodeRoletable hascurrent_version+last_synced_at— no new table needed)
Ready for Phase 4.
Phase 4 Complete — Scoped Ansible Deployment ✅
Commits
1e3b2032— Phase 4 core: clean task files + defaults for all 7 roles122173f5— Phase 4 complete: remaining main.yml updates + 3 new playbooks
Changes
Clean task files (one per role)
Each role now begins withimport_tasks: clean.yml(taggedclean) that removes legacy/wrong-node directories from/opt/autobot/. This enforces the invariant that a node only contains its own role's code.Per-role Linux service accounts
autobot-npu(system user, no shell/home) on NPU Workerautobot-aion AI Stackautobot-browseron Browser Workerautobot-backendaccount created (service still runs asautobot— full migration in Phase 7)autobot-frontendaccount created (same deferral)
Per-role scoped env files
/etc/autobot/autobot-<role>.env(mode 640, owner root, group ) deployed by each role's Ansible tasks. Service now reads role-scoped env in addition to legacy.env.Fixed stale install paths
npu_install_dir:/opt/autobot/npu-worker→/opt/autobot/autobot-npu-workerai_install_dir:/opt/autobot/ai-stack→/opt/autobot/autobot-ai-stack
New playbooks (all pass
--syntax-check)playbooks/system-update.yml— policy-aware apt updates (full/security/manual) in dependency order (SLM → Backend → DB → AI → NPU → Browser → Frontend) with reboot supportplaybooks/rotate-certs.yml— zero-downtime TLS cert rotation with nginx graceful reload + uvicorn restart + fingerprint verificationplaybooks/rotate-ssh-keys.yml— 7-phase key rotation (K0-K6) with temp safety-net key to prevent lockouts
Phase 4 Checklist
- Per-role clean task files (remove wrong-node legacy dirs)
- Per-role Linux service accounts with
/etc/autobot/autobot-\<role\>.env(mode 640) - Firewall rules from manifest ports already handled by existing UFW tasks in each role
-
system-update.ymlplaybook with policy-aware apt phases -
rotate-certs.ymlplaybook with zero-downtime reload -
rotate-ssh-keys.ymlplaybook with autobot_admin safety net (K0-K6 phases)
✅ Phase 5 Complete — Per-Role Code Sync Refactor
Commit:
304854f6Changes
autobot-slm-backend/models/database.py- Added
NodeCodeVersionmodel with(node_id, role_name)unique constraint for per-role per-node commit tracking
autobot-slm-backend/api/code_source.py(+306 lines)CodeNotificationextended withchanged_roles: List[str] = []_update_code_sync_statedelegates to_mark_roles_outdated(scoped) or_mark_all_nodes_outdated(backwards compat)_mark_roles_outdated— marks only nodes with the changed roles assigned as OUTDATED + upsertsNodeCodeVersionrows_upsert_node_code_version— insert/update per-role per-node version entryGET /api/code-source/node-code-versions— list per-role version status (filterable by node_id, role_name)POST /api/code-source/upload-package— air-gapped multipart .tar.gz upload, extracts to/opt/autobot/cache/<role>/, marks nodes OUTDATED
scripts/hooks/slm-post-commit(rewritten)- Detects which
autobot-*role dirs changed viagit diff HEAD~1 HEAD --name-only - Parallel rsync of each changed role to
/opt/autobot/cache/<role>/on SLM waitfor all rsyncs to complete before notifying (cache ready before SLM triggers deploy)changed_rolesJSON array in notify payload- Falls back to empty
changed_roles(all nodes OUTDATED) if no role dirs changed
autobot-slm-backend/ansible/playbooks/update-node.yml(new, 262 lines)- Cache-based scoped deploy — skips Phase 1 (clean) and Phase 3 (system deps)
- Play 1: Authenticate with SLM API, resolve role list from variable or cache dirs
- Play 2:
synchronizefrom/opt/autobot/cache/<role>/→/opt/autobot/<role>/on target node; rebuild frontend dist if needed; restart affected services - Play 3: Mark node UP_TO_DATE via
POST /api/code-sync/nodes/{node_id}/mark-synced
Phase 5 checklist
- Post-commit hook detects changed role dirs (git diff)
- Hook rsyncs only changed roles to
/opt/autobot/cache/on SLM -
POST /api/code-source/notifyacceptschanged_roles[]+ commit hash - SLM marks only affected nodes OUTDATED (per-role)
- Ansible
update-node.ymldeploys from cache (scoped, skips clean + system deps) - Manual upload endpoint (
POST /api/code-source/upload-package) for air-gapped use -
node_code_versionsDB table — per-role per-node commit tracking
- Added
- added a commit that references this issue
on Feb 18, 2026 12 remaining items
- added a commit that references this issue
on Feb 18, 2026 - added a commit that references this issue
on Mar 9, 2026 - added 6 commits that reference this issue
on Apr 7, 2026 - added a commit that references this issue
on Aug 12, 2026 - added 4 commits that reference this issue
on Sep 7, 2026
Background
The AutoBot fleet currently has no consistent deployment architecture:
.21hasautobot-slm-frontend/which belongs on.19).19SLM server has the entire monorepo dumped into/opt/autobot/.20backend has a dual layout — old flat.pyfiles coexist withautobot-user-backend/subdirnpu-worker/,browser-service/)autobot-user-frontend), Ansible role names (frontend), SLM role names (browser-service), and systemd service names (autobot-playwright) are all differentinfrastructure/doesn't follow theautobot-*conventionautobot-slm-agentcode is buried insideautobot-slm-backend/slm/agent/instead of being its own roleautobot-ai-stack,autobot-database,autobot-monitoring,autobot-slm-databaseas top-level code dirsTarget Architecture
Rule: every role is a top-level directory
Each
autobot-<role>/contains code only. Eachautobot-infrastructure/autobot-<role>/contains the Ansible role, systemd templates, nginx configs, vault, and manifest.Full role list
autobot-backend.20autobot-frontend.21autobot-ollama.20autobot-slm-backend.19autobot-slm-frontend.19autobot-slm-database.19autobot-monitoring.19autobot-npu-worker.22autobot-browser-worker.25autobot-ai-stack.24autobot-database.23autobot-slm-agentautobot-sharedautobot-infrastructureinfrastructure/Node assignments (all 9 nodes)
.19SLM.20Backend.21Frontend.22NPU.23Database.24AI Stack.25Browser.26Reserved.27ReservedEach node gets:
autobot-<its-roles>/+autobot-shared/+autobot-infrastructure/autobot-<its-roles>/only.Design
Role Manifest (
manifest.yml)Every role has
autobot-infrastructure/autobot-<role>/manifest.yml— the single source of truth SLM reads for deployment, configuration, health checking, and conflict validation.Coexistence validation
SLM pre-flight check before assigning a role to a node:
conflicts_withmatchwarns_withmatchKnown hard conflicts:
autobot-frontend+autobot-slm-frontend→ port 443autobot-monitoring+autobot-browser-worker→ port 3000autobot-database+autobot-slm-database→ port 5432Secrets model (Hybrid A+B)
autobot-infrastructure/autobot-<role>/vault.yml) — role-specific secrets (TLS key, role API key)autobot-infrastructure/shared/vault.yml) — cross-role secrets (Redis password, CA cert, service auth tokens)/etc/autobot/autobot-<role>.env(mode 640, owned byroot:<role-user>)autobot-backend,autobot-frontend, etc.)Deployment flow (6 phases)
SLM triggers
ansible-playbookdirectly when a role is assigned or updated:Updates skip phases 1 and 3 (clean + system deps). Full provisioning on first assignment only.
Code sync flow
Git repo is on dev machine. SLM server
.19is the distribution hub.Per-role version tracking in SLM DB — each role on each node tracks its own commit independently.
Manual upload also supported (tarball via SLM GUI) for air-gapped scenarios.
System update flow (apt)
Independent from code updates. Per-role manifest declares update policy:
.21/.25/.26/.27(stateless) →full,immediatereboot.19/.20/.22/.24(stateful) →security,scheduledreboot.23(database) →manualonlySLM daily cert + package check alerts when:
Security model
Per-role service accounts — each role runs as
autobot-<role>Linux user. A compromised role process cannot read another role's secrets.TLS everywhere — internal CA on SLM server. Per-node certs signed at provision time. Auto-rotation 14 days before expiry via
systemctl reload(zero downtime).SSH key rotation with autobot_admin safety net:
autobot_adminkey is never rotated at the same time asautobotkey. Its own rotation uses break-glass as the safety net.CA rotation — rolling with transition bundle (old + new CA both trusted during window, then old removed). Zero downtime.
Firewall — generated automatically from manifests. If role A
depends_onrole B, Ansible adds the corresponding UFW allow rule on B's node for A's IP.Documentation structure
Implementation Phases
Phase 1 — Repo restructuring
autobot-user-backend/→autobot-backend/autobot-user-frontend/→autobot-frontend/infrastructure/→autobot-infrastructure/autobot-infrastructure/autobot-user-backend/→autobot-infrastructure/autobot-backend/autobot-infrastructure/autobot-user-frontend/→autobot-infrastructure/autobot-frontend/autobot-infrastructure/autobot-db-stack/→autobot-infrastructure/autobot-database/autobot-slm-backend/slm/agent/→autobot-slm-agent/(top-level)autobot-slm-database/,autobot-monitoring/,autobot-ai-stack/,autobot-database/Phase 2 — Role manifests
manifest.ymlfor all 11 roles + slm-agent + sharedREADME.mdfor all 11 roles using templatedocs/architecture/ROLE_ARCHITECTURE.mddocs/architecture/COEXISTENCE_MATRIX.mdPhase 3 — SLM manifest integration
node_code_versionstable)Phase 4 — Ansible refactor (scoped deployment)
/etc/autobot/<role>.envwith correct ownership (640)depends_on+portsloopback_only: trueportssystem-update.ymlplaybook with policy-aware apt phasesrotate-certs.ymlplaybook with zero-downtime reloadrotate-ssh-keys.ymlplaybook with autobot_admin safety netPhase 5 — Code sync refactor
/opt/autobot/cache/on SLMPOST /api/code-source/notifyacceptschanged_roles[]+ commit hashupdate-node.ymldeploys from cache (scoped, skips clean + system deps)POST /api/code-source/upload-package) for air-gapped usenode_code_versionsDB table — per-role per-node commit trackingPhase 6 — Node cleanup
autobot-slm-frontend/from.21.19— leave only SLM-role dirs.pyfiles from.20— keep onlyautobot-backend/npu-worker/from.22browser-service/from.25ai-stack/from.24— replace withautobot-ai-stack/config/,data/,logs/,scripts/from node rootsPhase 7 — Security hardening
rotate-ssh-keys.ymlEMERGENCY_RECOVERY.mddocs/runbooks/ROTATE_SSH_KEYS.mddocs/runbooks/ROTATE_CERTS.mdPhase 8 — Documentation
docs/architecture/ROLE_ARCHITECTURE.mddocs/architecture/COEXISTENCE_MATRIX.mddocs/architecture/NETWORK_TOPOLOGY.mddocs/architecture/UPDATE_FLOWS.mddocs/runbooks/manifest.yml+README.mdfor all 11 rolesAcceptance Criteria
ls /opt/autobot/on each node shows only that node's assigned roles + shared + infrastructurenpu-worker/,browser-service/,ai-stack/, flat.pyfiles) on any nodeautobot-frontend+autobot-slm-frontendto same node (port conflict)git push→ only changed role dirs sync to SLM cache → only affected nodes marked OUTDATEDupdate-node.ymldeploys only the role's dirs to the node (nothing else)systemctl status autobot-<role>uses consistent naming on every node.envfiles exist at/etc/autobot/autobot-<role>.envwith mode 640rotate-ssh-keys.ymlenables autobot_admin, rotates, verifies, disables — all in one playbookvue-tsc --buildandruff checkpass after Phase 1 renamesLabels
enhancementinfrastructureansiblearchitectureSize estimate
Large epic — implement in phases. Each phase is independently mergeable.