Repository navigation
fix(ansible): Add Redis password and ensure systemd services auto-start #724
Copy link
Copy link
Closed
Labels
Description
Issue: VM Services Not Auto-Starting After Reboot
Problem
After VM reboot, Redis and NPU worker services are not running, requiring manual intervention.
Symptoms Observed (2026-01-14)
ERROR: Failed to get async Redis client for database 'main': Error 111 connecting to 172.16.168.23:6379. Connection refused.
ERROR: HTTP request failed: Cannot connect to host 172.16.168.22:8081 ssl:default [Connect call failed]
After manual start:
ERROR: AUTH <password> called without any password configured for the default user
Root Causes
1. Redis Password Not Configured in Ansible
- Template:
ansible/roles/redis/templates/redis-stack.conf.j2has NOrequirepassdirective - Vault:
ansible/inventory/group_vars/vault.ymlhas Redis password commented out - Database config:
ansible/inventory/group_vars/database.ymlhasauth_enabled: false - Backend expects:
REDIS_PASSWORD=V+6qyK5ppiEbmeQ2i/UK/G/p2TGtEZ0ustf/wDh1gag=(from .env)
2. Systemd Services Not Deployed
- Redis VM (172.16.168.23):
redis-stack-server.service could not be found - NPU Worker VM (172.16.168.22):
npu-worker.service could not be found
The Ansible roles DEFINE the services but they're not deployed to the VMs.
Required Fixes
Phase 1: Redis Password Configuration
1. Update Vault (ansible/inventory/group_vars/vault.yml)
vault_redis_password: "V+6qyK5ppiEbmeQ2i/UK/G/p2TGtEZ0ustf/wDh1gag="2. Update Database Config (ansible/inventory/group_vars/database.yml)
security:
redis_security:
auth_enabled: true
password: "{{ vault_redis_password }}"3. Update Redis Template (ansible/roles/redis/templates/redis-stack.conf.j2)
# Authentication
{% if redis_password is defined and redis_password %}
requirepass {{ redis_password }}
{% endif %}
4. Update Redis Role Variables (ansible/roles/redis/vars/main.yml)
redis_password: "{{ vault_redis_password }}"Phase 2: Ensure Systemd Services Are Deployed
1. Run Ansible Playbook for Redis
ansible-playbook -i ansible/inventory/production.yml ansible/playbooks/deploy-database.yml --tags systemd2. Run Ansible Playbook for NPU Worker
ansible-playbook -i ansible/inventory/production.yml ansible/site.yml --limit npu-worker --tags systemdPhase 3: Verify Auto-Start
Checklist:
- Redis service enabled:
systemctl is-enabled redis-stack-server - NPU worker service enabled:
systemctl is-enabled autobot-npu-worker - Services start on boot after VM reboot
- Redis accepts password authentication
- Backend connects without circuit breaker trips
Files to Modify
| File | Change |
|---|---|
ansible/inventory/group_vars/vault.yml |
Add vault_redis_password |
ansible/inventory/group_vars/database.yml |
Enable auth, reference vault password |
ansible/roles/redis/vars/main.yml |
Add redis_password variable |
ansible/roles/redis/templates/redis-stack.conf.j2 |
Add requirepass directive |
ansible/roles/redis/tasks/main.yml |
Verify systemd enable task |
ansible/roles/npu_worker/tasks/main.yml |
Add state: started to systemd task |
Temporary Fix Applied
Manual commands were run to restore services:
# Redis
ssh autobot@172.16.168.23 "sudo redis-stack-server --daemonize yes"
ssh autobot@172.16.168.23 "redis-cli CONFIG SET requirepass 'V+6qyK5ppiEbmeQ2i/UK/G/p2TGtEZ0ustf/wDh1gag='"
# NPU Worker
ssh autobot@172.16.168.22 "cd /home/autobot/autobot-npu-worker && source venv/bin/activate && nohup python npu_worker_main.py > logs/npu.log 2>&1 &"These fixes are NOT persistent across reboots.
Labels
buginfrastructureansiblepriority:high
Reactions are currently unavailable
Activity
mrveiss commentedon Jan 19, 2026
Implementation Progress
Phase 1: Redis Password Configuration ✅
Files Modified:
ansible/inventory/group_vars/vault.yml- Addedvault_redis_passwordwith the correct passwordansible/inventory/group_vars/database.yml- Enabled auth and referenced vault password:ansible/roles/redis/templates/redis-stack.conf.j2- Addedrequirepassdirective:ansible/roles/redis/vars/main.yml- Addedredis_password: "{{ vault_redis_password }}"ansible/roles/redis/tasks/main.yml- Updated Redis connectivity test to use password authenticationPhase 2: Systemd Services Auto-Start ✅
ansible/roles/npu_worker/tasks/main.yml- Addedstate: startedto systemd task (Redis task already had this)Checklist
state: startedNext Steps
Run Ansible playbooks to deploy changes:
mrveiss commentedon Jan 19, 2026
Implementation Complete ✅
Commit:
971c20ae- fix(ansible): Add Redis password and ensure systemd services auto-start (#724)Summary of Changes
ansible/inventory/group_vars/database.ymlauth_enabled: trueand added password referenceansible/roles/redis/templates/redis-stack.conf.j2requirepassdirectiveansible/roles/redis/vars/main.ymlredis_passwordvariableansible/roles/redis/tasks/main.ymlansible/roles/npu_worker/tasks/main.ymlstate: startedto systemd taskNote on vault.yml
The
vault.ymlfile is correctly gitignored for security reasons. The Redis password has been configured locally but should be:ansible-vaultbefore deployment, ORDeployment Instructions
Verification After Deployment
mrveiss commentedon Jan 19, 2026
Correction: Ansible playbooks should be run from the lifecycle manager node, not locally.
Deployment Instructions (from lifecycle manager)
Alternatively, sync the ansible directory to the lifecycle manager first:
fix(ansible): Add Redis password and ensure systemd services auto-sta…