Implement Redis pub/sub for queue change notifications

[lbedner]

Objective

Set up Redis pub/sub infrastructure to publish events when worker queues change, enabling real-time dashboard updates.

Tasks

• Set up Redis pub/sub channels for worker events
• Publish events when jobs are enqueued
• Publish events when jobs complete or fail
• Configure Redis keyspace notifications for arq:* keys
• Create event aggregator to batch updates
• Implement subscription manager in health service

Event Types to Publish

• Job enqueued: queue name, job id, timestamp
• Job started: queue name, job id, worker id
• Job completed: queue name, job id, result, duration
• Job failed: queue name, job id, error, retry info
• Worker connected/disconnected
• Queue metrics changed

Redis Configuration

# Enable keyspace notifications
CONFIG SET notify-keyspace-events KEA
# K = Keyspace events
# E = Keyevent events  
# A = All commands

Channel Structure

• worker:events:{queue_name} - Queue-specific events
• worker:events:all - All worker events
• health:updates - Aggregated health updates

Implementation Notes

• Consider using Redis Streams instead of Pub/Sub for persistence
• Batch events to reduce network overhead
• Include timestamp and event type in all messages
• Handle Redis connection failures gracefully