feat(security): Migrate services to mTLS authentication (PKI-based)

Feature: mTLS Service Authentication

Summary

Migrate from password-based service authentication to mutual TLS (mTLS) using the existing PKI infrastructure. This provides stronger security through certificate-based authentication.

Current State

Existing PKI Infrastructure ✅

AutoBot already has a comprehensive PKI system in src/pki/:

PKIManager - Main orchestrator for certificate lifecycle
CertificateGenerator - Creates CA and service certificates
CertificateDistributor - Distributes certs to VMs via SSH
ServiceConfigurator - Updates service configs for TLS

Redis TLS Support ✅

src/utils/redis_management/config.py already supports:

@dataclass
class RedisConfig:
    ssl: bool = False
    ssl_ca_certs: Optional[str] = None
    ssl_certfile: Optional[str] = None
    ssl_keyfile: Optional[str] = None
    ssl_cert_reqs: str = "required"

Auto-configuration when AUTOBOT_REDIS_TLS_ENABLED=true is set.

Connection Manager ✅

src/utils/redis_management/connection_manager.py already creates SSL contexts with client certificates:

ssl_context = ssl.create_default_context(ssl.Purpose.SERVER_AUTH)
ssl_context.load_verify_locations(config.ssl_ca_certs)
ssl_context.load_cert_chain(config.ssl_certfile, config.ssl_keyfile)
pool_params["ssl"] = ssl_context

What's Missing

1. Environment Configuration

Update .env to enable TLS:

# TLS Mode
AUTOBOT_TLS_MODE=required

# Redis TLS
AUTOBOT_REDIS_TLS_ENABLED=true
AUTOBOT_REDIS_TLS_PORT=6380

2. Redis Server TLS Configuration

The ServiceConfigurator creates this config but it needs tls-auth-clients yes for mTLS:

# /etc/redis-stack.conf
tls-port 6380
port 0  # Disable non-TLS port
tls-cert-file /etc/autobot/certs/server-cert.pem
tls-key-file /etc/autobot/certs/server-key.pem
tls-ca-cert-file /etc/autobot/certs/ca-cert.pem
tls-auth-clients yes  # Require client certificates (mTLS)

3. Ansible Integration

Update Redis role to deploy TLS configuration:

Generate certificates during enroll-node.yml
Configure Redis with TLS in deploy-database.yml
Distribute client certs to all VMs that connect to Redis

4. Per-Service Certificates

Each VM needs its own client certificate:

VM	Certificate	Common Name
main-host	`certs/main-host/`	autobot-main-host
frontend	`certs/frontend/`	autobot-frontend
npu-worker	`certs/npu-worker/`	autobot-npu-worker
ai-stack	`certs/ai-stack/`	autobot-ai-stack
browser	`certs/browser/`	autobot-browser

Implementation Plan

Phase 1: Certificate Generation & Distribution

Run PKI setup: python -c "from src.pki import setup_pki; import asyncio; asyncio.run(setup_pki(force=True))"
Verify certificates generated in certs/ directory
Verify certificates distributed to /etc/autobot/certs/ on each VM

Phase 2: Redis mTLS Configuration

Update src/pki/configurator.py - set tls-auth-clients yes
Update Redis config template to disable plain port (port 0)
Run configurator to update Redis VM
Restart Redis with TLS: systemctl restart redis-stack-server
Verify TLS connection: redis-cli --tls --cert ... --key ... --cacert ... ping

Phase 3: Backend mTLS Client Configuration

Set AUTOBOT_REDIS_TLS_ENABLED=true in .env
Verify RedisConfig.__post_init__ auto-loads cert paths
Restart backend and verify connection via circuit breaker logs
Remove REDIS_PASSWORD from .env (no longer needed)

Phase 4: Ansible Automation

Update ansible/roles/redis/templates/redis-stack.conf.j2 for TLS
Add certificate distribution tasks to Redis role
Update ansible/playbooks/enroll-node.yml to generate per-node certs
Add TLS verification to ansible/playbooks/health-check.yml

Phase 5: Other Services (Future)

FastAPI backend HTTPS
NPU worker inter-service auth
Celery broker TLS
WebSocket TLS

Security Benefits

Aspect	Password Auth	mTLS Auth
Credential exposure	Password in .env, logs, memory	Private key never transmitted
Authentication strength	Single shared secret	Per-service cryptographic identity
Revocation	Must rotate everywhere	Revoke single certificate
Audit trail	IP-based only	Certificate CN identifies client
Forward secrecy	No	Yes (with proper cipher suites)

Rollback Plan

If issues arise:

Set AUTOBOT_REDIS_TLS_ENABLED=false in .env
Re-enable Redis password: redis-cli CONFIG SET requirepass <password>
Restart backend services
Services fall back to password authentication

Related Issues

Closes fix(ansible): Add Redis password and ensure systemd services auto-start #724 (Redis password and systemd services)
Supersedes password-based Redis auth

Files to Modify

File	Change
`.env`	Add `AUTOBOT_REDIS_TLS_ENABLED=true`
`src/pki/configurator.py`	Set `tls-auth-clients yes`, `port 0`
`ansible/roles/redis/templates/redis-stack.conf.j2`	Add TLS config
`ansible/roles/redis/tasks/main.yml`	Add cert distribution
`ansible/playbooks/enroll-node.yml`	Generate per-node certs

Testing Checklist

Redis accepts TLS connections only
Backend connects with client certificate
Non-TLS connections rejected
Certificate expiration monitoring works
Health checks pass
Circuit breaker doesn't trip on TLS handshake

Labels

enhancement
security
infrastructure

fix(slm): Fix agent code quality issues (#726)

Implementation Plan - Refined

Design document created: docs/plans/2026-01-29-mtls-service-authentication-design.md

Phase 0: Cleanup (Pre-requisite)

Remove port 8000 references from main-host firewall rules
Remove port 8090 references from ansible/inventory/group_vars/aiml.yml:363
Remove port range 8000-8099 from ansible/inventory/group_vars/all.yml:118
Audit/deprecate legacy main.py in project root
Archive docker/ directory (not used - services run natively)

Phase 1: Certificate Generation & Distribution

Run PKIManager.setup(force=True)
Verify certificates in certs/ directory
Distribute certs to /etc/autobot/certs/ on all VMs
Verify permissions (600 for keys, 644 for certs)

Phase 2: Redis Dual-Auth Configuration

Backup Redis config on VM3
Add TLS config (keep password active): port 6379 + tls-port 6380
Open firewall for port 6380
Restart Redis Stack
Test TLS connection: redis-cli --tls -p 6380 ... ping
Verify password auth still works on 6379

Phase 3: Backend mTLS Client (Main Host)

Set AUTOBOT_REDIS_TLS_ENABLED=true in .env
Restart backend service
Verify TLS connection in logs
Monitor circuit breaker for 24 hours

Phase 4: Celery mTLS Client (Main Host)

Update backend/celery_app.py with SSL context
Change broker URL to rediss://
Restart Celery worker
Submit and verify test task

Phase 5: AI Stack mTLS Client (VM4)

Update AI Stack Redis connection to rediss://
Add client certificate configuration
Restart autobot-ai-stack service
Verify health endpoint

Phase 6: Disable Password Auth (MANUAL GATE)

Pre-requisites:

Zero connections on port 6379 for 24 hours
All services healthy via TLS
Operator explicit approval

Actions:

Update Redis: port 0, tls-auth-clients yes, remove requirepass
Restart Redis Stack
Verify port 6379 is closed
Full health check all services

Phase 7: HTTPS for HTTP APIs

Add SSL to Backend uvicorn.run() (port 8443)
Add SSL to NPU Worker (VM2, port 8081)
Add SSL to AI Stack (VM4, port 8080)
Update frontend VITE_HTTP_PROTOCOL=https

Phase 8: Ansible Automation

Update ansible/roles/redis/templates/redis-stack.conf.j2 with TLS
Add cert distribution to Redis role
Update firewall rules for port 6380
Update ansible/playbooks/health-check.yml for TLS

Per-VM Summary

VM	IP	Service	Redis Client	Server Cert
VM0 SLM	172.16.168.19	SLM Backend	-	nginx
Main Host	172.16.168.20	Backend + Celery	Yes	`main-host/`
VM1	172.16.168.21	Frontend	-	`frontend/`
VM2	172.16.168.22	NPU Worker	-	`npu-worker/`
VM3	172.16.168.23	Redis Stack	Server	`redis/`
VM4	172.16.168.24	AI Stack	Yes	`ai-stack/`
VM5	172.16.168.25	Browser	-	`browser/`

Rollback Script

Emergency rollback script will be at /usr/local/bin/autobot-tls-rollback.sh

Phase 0: Cleanup Targets Identified

Detailed cleanup document: docs/plans/2026-01-29-port-cleanup-targets.md

Files to Modify

File	Line	Change
`ansible/inventory/group_vars/all.yml`	118	Remove `port: "8000:8099"` range
`ansible/inventory/group_vars/aiml.yml`	361-366	Remove `port: "8090:8099"` block
`main.py` (root)	-	Rename to `main.py.deprecated`
`docker/`	-	Add `DEPRECATED.md` notice

Test/Script References to Update

These reference the legacy main.py and should point to backend/main.py:

tests/screenshots/test_security_endpoints.py:26
scripts/utilities/settings_diagnostic_utility.py:164
scripts/utilities/verify_backend_config.py:115
scripts/utilities/fix_settings_loading.py:160

Verification After Cleanup

# On main-host - should return NO results
grep -r "8000" /etc/autobot/
grep -r "8090" /etc/autobot/
netstat -tlnp | grep -E ":8000|:8090"

Note on SLM References

References to port 8000 for SLM Server (172.16.168.19) are CORRECT and should NOT be changed:

ansible/roles/slm_agent/defaults/main.yml
slm-server/config.py
scripts/slm-agent-standalone.py

SLM runs on a separate machine (VM0), not main-host.

Phase 0: Cleanup Complete

Commit: 4728a935

Completed Tasks

Deprecate root main.py → renamed to main.py.deprecated
Create stub main.py with deprecation warning
Add docker/DEPRECATED.md explaining services run natively
Add TODO markers to ansible inventory files for SLM firewall cleanup
Remove hardcoded IPs from design docs (replaced with ${AUTOBOT_*_HOST} variables)

Deferred Tasks

Script references to legacy main.py need to be updated in a separate PR after converting scripts to use proper logging (pre-commit hook enforces logging standards).

Files affected:

scripts/utilities/fix_settings_loading.py
scripts/utilities/settings_diagnostic_utility.py
scripts/utilities/verify_backend_config.py
tests/screenshots/test_security_endpoints.py

Firewall Port Cleanup

Firewall rules for ports 8000-8099 and 8090-8099 should be cleaned up via SLM config management, not direct ansible file edits. TODO markers added to track this.

Ready for Phase 1

Phase 0 cleanup is complete. Ready to proceed with Phase 1: Certificate Generation & Distribution when authorized.

Implementation Complete ✅

All phases of mTLS migration have been implemented and committed to Dev_new_gui branch.

Commits

4728a935 - Phase 0: Port cleanup, main.py deprecation, docker DEPRECATED.md
4d2e2654 - Phase 2-3: Migration script, backend TLS support
04b92647 - Phase 4: Celery mTLS, Windows NPU worker TLS
2e3b5ea8 - Phase 5-6: Verification, password auth removal

Files Modified/Created

File	Purpose
`scripts/security/mtls-migrate.py`	Migration orchestration tool
`backend/main.py`	TLS configuration for uvicorn
`backend/celery_app.py`	Redis TLS for Celery
`resources/windows-npu-worker/app/utils/redis_client.py`	TLS for NPU worker
`main.py`	Deprecation stub
`docker/DEPRECATED.md`	Docker deprecation notice
`docs/plans/2026-01-29-mtls-service-authentication-design.md`	Design document
`docs/plans/2026-01-29-port-cleanup-targets.md`	Cleanup targets

Migration Workflow

# Step 1: Enable Redis TLS (dual-auth mode)
python scripts/security/mtls-migrate.py --phase redis-dual-auth

# Step 2: Enable TLS in .env
AUTOBOT_REDIS_TLS_ENABLED=true

# Step 3: Restart services and verify
systemctl restart autobot-backend
python scripts/security/mtls-migrate.py --phase verify

# Step 4: After 24h validation with 0 plain connections
python scripts/security/mtls-migrate.py --phase disable-password