discovery(scheduler): ConnectorScheduler is per-worker — schedules lost on worker restart, status reports inconsistent #6556

Summary

autobot-backend/knowledge/connectors/scheduler.py is an in-process asyncio singleton (ConnectorScheduler with get_connector_scheduler()). The backend runs 4 uvicorn workers in production (backend_workers: 4 in autobot-slm-backend/ansible/roles/backend/defaults/main.yml:26). The four workers do NOT share scheduler state.

This produces three correctness bugs:

1. Worker-affinity: POST /knowledge_base/connectors lands on ONE of the 4 workers. That worker calls scheduler.start(connector_id, schedule). The other 3 workers don't know about the schedule.
2. Restart loss: When a worker dies/restarts (e.g. worker_max_tasks_per_child rotation, OOM, deploy), all asyncio tasks it owned are lost. The connector silently stops syncing until someone re-saves it.
3. Status inconsistency: GET /knowledge_base/connectors/{id} calls scheduler.is_running(connector_id) — answer depends on which worker handles the request. Worker that owns the schedule says scheduled: true; the other 3 say scheduled: false.

How discovered

While investigating #6507, I noticed ConnectorScheduler uses _tasks: Dict[str, asyncio.Task] in-process state with no Redis backing. Cross-referenced with backend deployment which uses 4 uvicorn workers.

Reproduce

1. Create a connector with schedule_cron: "*/5".
2. Send 4 GET /knowledge_base/connectors/{id} requests in quick succession.
3. Observe: at least one response has scheduled: false (the workers without the asyncio task).

After backend restart:
4. Check journalctl — connector sync events stop firing.
5. Edit the connector via API → schedule resumes (lands on one worker).

Architectural options

This needs a design call before implementation. Three patterns:

Option A — Redis leader election + Redis-persisted schedules

• Workers compete for a Redis lock; one becomes scheduler-leader.
• Schedule definitions stored in Redis; leader runs them.
• On leader death, lock expires and next worker takes over.
• ~150-200 LOC + tests. No new dependencies.
• Risk: leader-failover delay (lock TTL = max sync stall).

Option B — Migrate to celery-redbeat

• Redis-backed Beat scheduler that supports dynamic schedule mutation at runtime.
• API endpoints write RedBeatSchedulerEntry records to Redis; one Beat process owns firing.
• New dependency (celery-redbeat).
• Depends on discovery(deploy): Celery Beat configured but not deployed — periodic schedules don't fire in prod #6555 (Beat must be deployed first).
• Cleaner architecturally — unifies with refactor(async): Phase 1 of #6495 — unify task queues onto Celery (3 systems → 1) #6505 plan.

Option C — Pin connector scheduler to a single worker

• Add a "scheduler worker" mode (AUTOBOT_BACKEND_ROLE=scheduler) on one of the 4 workers.
• Other workers no-op scheduler.start/stop and forward to the scheduler worker via Redis pubsub.
• Hacky but minimal change.

Acceptance criteria

• Schedules survive worker restart (any worker dying does not lose schedules).
• GET /knowledge_base/connectors/{id} returns consistent scheduled status regardless of which worker handles it.
• Schedule firing is single-flight (one connector with */5 fires once every 5 minutes total, not 4×).
• Existing API contract unchanged (no breaking changes to API consumers).
• Integration test exercises: create connector → restart all workers → schedule still fires.
• Document chosen pattern in docs/architecture/async-work.md.

Severity

P1 — silent correctness failure. Connectors appear to work (UI shows them as "scheduled") but periodic syncs may not run after any backend restart.

Related

• refactor(async): Phase 3 of #6495 — promote Celery Beat as canonical periodic scheduler (4 → 1+1) #6507 (Phase 3 of refactor(async): unify task-queue + progress-tracking + periodic-scheduler stack (10 parallel systems) #6495) — original scheduler unification
• discovery(deploy): Celery Beat configured but not deployed — periodic schedules don't fire in prod #6555 — Celery Beat not deployed (prerequisite for Option B)
• refactor(async): Phase 1 of #6495 — unify task queues onto Celery (3 systems → 1) #6505 (Phase 1 of refactor(async): unify task-queue + progress-tracking + periodic-scheduler stack (10 parallel systems) #6495) — Celery worker / Beat consolidation (Option B fits cleanly here)

Out of scope

This issue is for filing the bug + architectural options. Do not auto-implement until the design call is made — Options A/B/C have meaningfully different operational characteristics.