feat(llc): Heartbeat scheduler — Redis sorted set, cron dispatch, restart-safe #8225

Parent

Phase 3 (#8207) · Umbrella (#8204)

Depends On

#8218 — Agent API keys. #8213 — Work item atomic checkout.

Context

The heartbeat scheduler is the engine that makes agents autonomous. It fires adapter invocations on cron without human intervention. It must survive process restarts without losing scheduled runs.

Acceptance Criteria

• llc_heartbeat_runs table:
  • id UUID PK, company_id FK, agent_id FK, invocation_source ENUM(scheduler, manual, callback), status ENUM(queued, running, succeeded, failed, cancelled, timed_out), started_at, finished_at, error TEXT nullable, external_run_id TEXT nullable, context_snapshot JSONB nullable, created_at
• agent_org_nodes extended with: heartbeat_cron TEXT nullable (cron expression), heartbeat_enabled BOOLEAN DEFAULT FALSE, last_heartbeat_at TIMESTAMPTZ nullable, adapter_type TEXT, adapter_config JSONB, context_mode ENUM(thin, fat) DEFAULT thin
• HeartbeatScheduler in llc/scheduler/heartbeat_scheduler.py:
  • On startup: loads all heartbeat_enabled = true agents from DB, computes next fire time per cron, writes to Redis sorted set llc:heartbeat:schedule with score = next fire timestamp (epoch seconds)
  • Polling loop: every 5s, ZRANGEBYSCORE llc:heartbeat:schedule 0 {now} to find due agents
  • For each due agent: creates heartbeat_run record, dispatches adapter (async), updates sorted set with next fire time
  • On process restart: re-reads DB, re-populates sorted set (idempotent)
• Manual trigger: POST /api/llc/agents/{id}/heartbeat/trigger — creates queued run, dispatches immediately
• GET /api/llc/agents/{id}/runs — list heartbeat runs, paginated, filterable by status
• GET /api/llc/agents/{id}/runs/{run_id} — run detail with context snapshot
• Scheduler is started in autobot-backend/initialization/background_tasks.py
• python -m pytest autobot-backend/llc/ -k heartbeat_scheduler passes