Repository navigation
umbrella(resilience): five platform failure classes we have no defence against #15773
Description
Problem
A comparative failure-mode review of large open-source agent-orchestration platforms surfaced
six recurring classes of production failure. Each was checked against this codebase; the six
below are gaps we are demonstrably exposed to. None is hypothetical — each names the file that
lacks the defence.
Two classes were checked and found already covered, and are recorded here so they are not
re-audited:
- Agent/session lifecycle leaks (stuck "running", never-closed workspaces) — covered by the TTL +
reaper inautobot-backend/services/agent_terminal/session_manager.py:24,466. - Runaway token cost / agent loops — covered by
autobot-backend/llm_shared/token_budget.py,
the provider circuit breaker, andautobot_shared/repetition_guard.py.
Tasks
Wave 1 — independent, no blockers:
- Task 1: resilience(logging): the canonical logger has no flood guard — a dependency outage can fill the disk #15774 — log-flood guard in the canonical logger (
autobot_shared/logging_manager.py) - Task 2: bug(api): DB integrity errors surface as raw 500s outside llc/ — no shared 409/422 mapping #15775 — map DB integrity/constraint errors to 409/422 across the API surface
- Task 3: tech-debt(migrations): 33 destructive migrations with no expand-contract guard — rolling updates can drop live columns #15776 — expand-contract guard for destructive migrations (
drop_column) - Task 4: bug(sandbox): recursive delete has no unsaved-work guard — agent-callable rmtree is unrecoverable #15777 — guard recursive delete against unsaved work (
autobot-backend/api/sandbox_files.py) - Task 5: feat(api): no idempotency keys on creation endpoints — a retried agent POST duplicates the resource #15778 — idempotency keys on agent-driven creation endpoints
- Task 6: bug(scoping): a null-scope resource is denied to every principal including admin — un-repairable in-app #15779 — null-scope resource is denied to every principal incl. admin (
autobot_shared/scoping/visibility.py:80,85)
Acceptance Criteria
- Every task above is closed with code evidence from merged
Dev_new_gui - Each child states its own criteria; this umbrella closes only when all six are closed
Notes
Related existing issues (same classes, narrower scope): #15772 (read-then-insert returns 500
instead of 409), #13803 (unbounded warn on failure path).
Activity
mrveiss commentedon Sep 5, 2026
Second verification pass on the three classes left open at filing time:
is_visible()(autobot_shared/scoping/visibility.py:80,85) and its single entry pointcan_access()(autobot-backend/services/resource_visibility.py:30-46) have no admin bypass, so a null-owner + null-company + no-grant resource is denied to every principal and cannot be repaired through the product.X-Internal-API-Keyis used only by node-proxy and the SLM service, mapping to the syntheticslm-backendprincipal (autobot-backend/api/envelope_secrets.py:86). No issue filed.autobot_shared/retry_mechanism.pyuses a canonical retryable exception set with non-retryable precedence (refactor(celery): derive CELERY_TRANSIENT_ERRORS from autobot_shared.retry_mechanism canonical exception sets #11689), not a single-status-code gate. No issue filed.mrveiss commentedon Oct 3, 2026
What would have to be true to close this — and a correction to my own triage
Found by a sweep for umbrellas whose native children are all closed (10 of 138 parents, across 1,733 open issues). I first put this in "scope outruns its children". That was wrong: this umbrella's criteria are about its children, and all six are closed.
Its two non-task criteria read:
Every task above is closed with code evidence from merged Dev_new_guiEach child states its own criteria; this umbrella closes only when all six are closedThe second is already met — #15774, #15775, #15776, #15777, #15778 and #15779 are all closed (6/6 native children).
The one thing missing
Per-child code evidence against merged
main, gathered once and posted here — six short entries, each naming the file/symbol that satisfies that child.That is a bounded artifact, not open-ended work. One caveat for whoever does it: the criterion says "merged
Dev_new_gui", which is the superseded base. Evidence must come frommain(git show origin/main:<path>), and the criterion's wording wants updating in the same pass so the next reader does not go looking at a branch that is no longer the base.Nothing ticked — all six task boxes are still unticked despite the children being closed, and they should be ticked in one pass with the evidence rather than tidied now.
mrveiss commentedon Oct 3, 2026
Closed — both umbrella criteria met
AC2 — "this umbrella closes only when all six are closed." All six children are
closed/completed: #15774, #15775, #15776, #15777, #15778, #15779.AC1 — "every task above is closed with code evidence." Each child's closure carries
it. #15779 is worth naming because it nearly tripped me: a scan for
file.py:NNNpatterns returned zero for it, and the comment in fact reads "Closed by #16927,
merged as
28a5896d2. Each AC was verified against the merged branch code" followed byper-AC detail. The scan was correct and answered a narrower question than the criterion
asks — evidence is not required to take one shape.
One wording correction rather than a silent tick. AC1 says "from merged
Dev_new_gui". That base is superseded; the repo's PR base ismain, and the children'sevidence is against the branches merged into it. The criterion's intent — closed with
code evidence, not closed to tidy — is met. The named branch is stale, and I am recording
that here rather than editing history, so a reader comparing the criterion to the
evidence is not left reconciling a branch that no longer exists.
Scope check before closing. This umbrella's own body adds no acceptance criteria
beyond the six tasks and those two conditions, so "all children closed" genuinely is the
whole of it. That is not true of every umbrella — several carry criteria describing an
outcome no child was ever going to satisfy, which is why they stay open with their gaps
named rather than closing on a complete child list.
Triage by the research session; children and evidence verified here.