Skip to content

fix(ansible): Add Redis password and ensure systemd services auto-start #724

Closed

Description

@mrveiss

Issue: VM Services Not Auto-Starting After Reboot

Problem

After VM reboot, Redis and NPU worker services are not running, requiring manual intervention.

Symptoms Observed (2026-01-14)

ERROR: Failed to get async Redis client for database 'main': Error 111 connecting to 172.16.168.23:6379. Connection refused.
ERROR: HTTP request failed: Cannot connect to host 172.16.168.22:8081 ssl:default [Connect call failed]

After manual start:

ERROR: AUTH <password> called without any password configured for the default user

Root Causes

1. Redis Password Not Configured in Ansible

  • Template: ansible/roles/redis/templates/redis-stack.conf.j2 has NO requirepass directive
  • Vault: ansible/inventory/group_vars/vault.yml has Redis password commented out
  • Database config: ansible/inventory/group_vars/database.yml has auth_enabled: false
  • Backend expects: REDIS_PASSWORD=V+6qyK5ppiEbmeQ2i/UK/G/p2TGtEZ0ustf/wDh1gag= (from .env)

2. Systemd Services Not Deployed

  • Redis VM (172.16.168.23): redis-stack-server.service could not be found
  • NPU Worker VM (172.16.168.22): npu-worker.service could not be found

The Ansible roles DEFINE the services but they're not deployed to the VMs.

Required Fixes

Phase 1: Redis Password Configuration

1. Update Vault (ansible/inventory/group_vars/vault.yml)

vault_redis_password: "V+6qyK5ppiEbmeQ2i/UK/G/p2TGtEZ0ustf/wDh1gag="

2. Update Database Config (ansible/inventory/group_vars/database.yml)

security:
  redis_security:
    auth_enabled: true
    password: "{{ vault_redis_password }}"

3. Update Redis Template (ansible/roles/redis/templates/redis-stack.conf.j2)

# Authentication
{% if redis_password is defined and redis_password %}
requirepass {{ redis_password }}
{% endif %}

4. Update Redis Role Variables (ansible/roles/redis/vars/main.yml)

redis_password: "{{ vault_redis_password }}"

Phase 2: Ensure Systemd Services Are Deployed

1. Run Ansible Playbook for Redis

ansible-playbook -i ansible/inventory/production.yml ansible/playbooks/deploy-database.yml --tags systemd

2. Run Ansible Playbook for NPU Worker

ansible-playbook -i ansible/inventory/production.yml ansible/site.yml --limit npu-worker --tags systemd

Phase 3: Verify Auto-Start

Checklist:

  • Redis service enabled: systemctl is-enabled redis-stack-server
  • NPU worker service enabled: systemctl is-enabled autobot-npu-worker
  • Services start on boot after VM reboot
  • Redis accepts password authentication
  • Backend connects without circuit breaker trips

Files to Modify

File Change
ansible/inventory/group_vars/vault.yml Add vault_redis_password
ansible/inventory/group_vars/database.yml Enable auth, reference vault password
ansible/roles/redis/vars/main.yml Add redis_password variable
ansible/roles/redis/templates/redis-stack.conf.j2 Add requirepass directive
ansible/roles/redis/tasks/main.yml Verify systemd enable task
ansible/roles/npu_worker/tasks/main.yml Add state: started to systemd task

Temporary Fix Applied

Manual commands were run to restore services:

# Redis
ssh autobot@172.16.168.23 "sudo redis-stack-server --daemonize yes"
ssh autobot@172.16.168.23 "redis-cli CONFIG SET requirepass 'V+6qyK5ppiEbmeQ2i/UK/G/p2TGtEZ0ustf/wDh1gag='"

# NPU Worker  
ssh autobot@172.16.168.22 "cd /home/autobot/autobot-npu-worker && source venv/bin/activate && nohup python npu_worker_main.py > logs/npu.log 2>&1 &"

These fixes are NOT persistent across reboots.

Labels

  • bug
  • infrastructure
  • ansible
  • priority:high

Activity

  1. mrveiss commented on Jan 19, 2026

    @mrveiss
    OwnerAuthor

    Implementation Progress

    Phase 1: Redis Password Configuration ✅

    Files Modified:

    1. ansible/inventory/group_vars/vault.yml - Added vault_redis_password with the correct password
    2. ansible/inventory/group_vars/database.yml - Enabled auth and referenced vault password:
      redis_security:
        auth_enabled: true
        password: "{{ vault_redis_password }}"
    3. ansible/roles/redis/templates/redis-stack.conf.j2 - Added requirepass directive:
      {% if redis_password is defined and redis_password %}
      requirepass {{ redis_password }}
      {% endif %}
      
    4. ansible/roles/redis/vars/main.yml - Added redis_password: "{{ vault_redis_password }}"
    5. ansible/roles/redis/tasks/main.yml - Updated Redis connectivity test to use password authentication

    Phase 2: Systemd Services Auto-Start ✅

    1. ansible/roles/npu_worker/tasks/main.yml - Added state: started to systemd task (Redis task already had this)

    Checklist

    • vault.yml updated with Redis password
    • database.yml auth enabled
    • Redis template has requirepass directive
    • Redis role variables reference vault password
    • Redis systemd task verified (already correct)
    • NPU worker systemd task updated with state: started
    • Deploy and verify on VMs

    Next Steps

    Run Ansible playbooks to deploy changes:

    # Deploy Redis configuration
    ansible-playbook -i ansible/inventory/production.yml ansible/playbooks/deploy-database.yml --tags redis
    
    # Deploy NPU worker configuration
    ansible-playbook -i ansible/inventory/production.yml ansible/site.yml --limit npu-worker --tags npu_worker
  2. mrveiss commented on Jan 19, 2026

    @mrveiss
    OwnerAuthor

    Implementation Complete ✅

    Commit: 971c20ae - fix(ansible): Add Redis password and ensure systemd services auto-start (#724)

    Summary of Changes

    File Change
    ansible/inventory/group_vars/database.yml Enabled auth_enabled: true and added password reference
    ansible/roles/redis/templates/redis-stack.conf.j2 Added requirepass directive
    ansible/roles/redis/vars/main.yml Added redis_password variable
    ansible/roles/redis/tasks/main.yml Updated connectivity test with password auth
    ansible/roles/npu_worker/tasks/main.yml Added state: started to systemd task

    Note on vault.yml

    The vault.yml file is correctly gitignored for security reasons. The Redis password has been configured locally but should be:

    1. Encrypted with ansible-vault before deployment, OR
    2. Managed via a secure secrets management system

    Deployment Instructions

    # Deploy Redis configuration
    ansible-playbook -i ansible/inventory/production.yml ansible/playbooks/deploy-database.yml --tags redis
    
    # Deploy NPU worker configuration
    ansible-playbook -i ansible/inventory/production.yml ansible/site.yml --limit npu-worker --tags npu_worker

    Verification After Deployment

    # Check Redis service
    ssh autobot@172.16.168.23 "systemctl is-enabled redis-stack-server && systemctl status redis-stack-server"
    
    # Check NPU worker service
    ssh autobot@172.16.168.22 "systemctl is-enabled autobot-npu-worker && systemctl status autobot-npu-worker"
    
    # Test Redis authentication
    ssh autobot@172.16.168.23 "redis-cli -a '\$REDIS_PASSWORD' ping"
  3. mrveiss commented on Jan 19, 2026

    @mrveiss
    OwnerAuthor

    Correction: Ansible playbooks should be run from the lifecycle manager node, not locally.

    Deployment Instructions (from lifecycle manager)

    # SSH to lifecycle manager first
    ssh autobot@lifecycle-manager
    
    # Then run the playbooks
    ansible-playbook -i ansible/inventory/production.yml ansible/playbooks/deploy-database.yml --tags redis
    ansible-playbook -i ansible/inventory/production.yml ansible/site.yml --limit npu-worker --tags npu_worker

    Alternatively, sync the ansible directory to the lifecycle manager first:

    ./scripts/utilities/sync-to-vm.sh lifecycle-manager ansible/ /home/autobot/autobot/ansible/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions