feat(coordination): SharedRuntimeBag — Redis-backed cross-worker constraint envelope (generalize #4502 A2A pattern) #6630

Problem

AutoBot runs 4 uvicorn workers in production (backend_workers: 4 per ansible config). Several subsystems hold state in module-level singletons that are silently per-worker, not shared:

• autobot-backend/services/connectors/scheduler.py:69-148 ConnectorScheduler — schedules CRUD'd via API land on one worker; other 3 don't see them. (Active design call in discovery(scheduler): ConnectorScheduler is per-worker — schedules lost on worker restart, status reports inconsistent #6556.)
• autobot-backend/live_event_manager.py:1-144 LiveEventManager — in-memory channel router; WebSocket subscribers only see events from their connected worker.
• General agent runtime state — budget counters, abort flags, parent-goal markers — none of it shared cross-worker today.

This is a recurring class of bug, not an isolated one (memory references at least #6556, similar pattern in #6479 persist-assignment-state).

The A2A task store (autobot-backend/a2a/task_manager.py, Issue #4502) already proves the right pattern: Redis-backed with TTL + pub/sub for change notification. We just don't extend it.

Proposed Solution

Generalize #4502's pattern into a reusable SharedRuntimeBag primitive. Any subsystem that needs cross-worker state uses it instead of a module-level singleton.

API:

class SharedRuntimeBag(Generic[T]):
    """Redis-backed dict with TTL, optimistic locking, and pub/sub change notifications."""

    def __init__(self, namespace: str, value_type: type[T], default_ttl_s: int = 3600): ...

    async def get(self, key: str) -> T | None: ...
    async def set(self, key: str, value: T, ttl_s: int | None = None) -> None: ...
    async def update(self, key: str, mutator: Callable[[T], T], retries: int = 3) -> T: ...  # CAS
    async def delete(self, key: str) -> None: ...
    async def keys(self, pattern: str = "*") -> list[str]: ...
    async def subscribe_changes(self) -> AsyncIterator[ChangeEvent]: ...  # pub/sub

Storage: Redis hash per namespace, msgpack/json serialization, key pattern runtime_bag:{namespace}:{key}, change channel runtime_bag:{namespace}:changes.

Migration targets (in priority order):

1. AgentBudgetTracker — required for feat(cost): budget policy with hard-stop auto-pause to prevent runaway agent spend #6470 budget hard-stop (any worker hitting threshold should pause the agent globally)
2. AgentAbortFlags — cancel(agent_id) on worker A propagates to in-flight task on worker B
3. AgentParentGoalState — needed for feat(orchestration): goal ancestry as schema on Task model so prompts always carry the why #6469 goal ancestry to work cross-worker
4. ConnectorScheduler state migration (resolves discovery(scheduler): ConnectorScheduler is per-worker — schedules lost on worker restart, status reports inconsistent #6556's per-worker problem)
5. LiveEventManager subscriber registry (so cross-worker WebSocket fanout works without extra hops)

Acceptance Criteria

• autobot_shared/coordination/shared_runtime_bag.py implementing the API above
• Optimistic locking via Redis WATCH/MULTI/EXEC with retry-on-conflict
• Pub/sub change notifications wired to subscribe_changes async iterator
• TTL handling and expiry events
• Unit tests covering: get/set, CAS conflict + retry, TTL expiry, pub/sub fanout
• Integration test: 4 simulated workers; state set on one is visible on the others within 50ms
• Migration of AgentBudgetTracker as the canonical first consumer (links to feat(cost): budget policy with hard-stop auto-pause to prevent runaway agent spend #6470)
• Documentation: docs/architecture/shared-runtime-bag.md with examples and migration cookbook
• Pre-commit lint rule: warn on new module-level singletons in agent/runtime paths (recommend SharedRuntimeBag)

Files Affected

• New: autobot_shared/coordination/shared_runtime_bag.py
• New: autobot_shared/coordination/__init__.py
• New: docs/architecture/shared-runtime-bag.md
• New: autobot-backend/tests/coordination/test_shared_runtime_bag.py
• Migration consumer: autobot-backend/services/budget/agent_budget_tracker.py (or wherever feat(cost): budget policy with hard-stop auto-pause to prevent runaway agent spend #6470 lands)

Why This Matters

Per-worker singleton is the #1 recurring architectural bug class in AutoBot. Memory has at least 5 cross-references (#6556, #6479, ConnectorScheduler, LiveEventManager, in-process schedule registries). Fix the pattern, not each instance. Existing #4502 already proves the design works at scale for A2A.

This also unblocks proper governance: budget hard-stop, cross-worker abort, goal ancestry — none of which work cleanly without shared state.

Related

• Pattern source: bug(a2a): task manager uses in-process memory store — 404 flapping across uvicorn workers #4502 (A2A Redis-backed task store)
• Direct consumer: feat(cost): budget policy with hard-stop auto-pause to prevent runaway agent spend #6470 (budget hard-stop), feat(heartbeat): add status field to AgentRuntimeState (active/paused/terminated) — precondition for #6470 #6476 (AgentRuntimeState.status)
• Resolves recurring class: discovery(scheduler): ConnectorScheduler is per-worker — schedules lost on worker restart, status reports inconsistent #6556 (ConnectorScheduler per-worker), refactor(orchestration): persist task-assignment metadata to Redis — distributed_management dicts lose state on restart #6479 (persist assignment state)
• Adjacent: feat(orchestration): goal ancestry as schema on Task model so prompts always carry the why #6469 (goal ancestry needs cross-worker visibility)