Repository navigation
bug(lifespan): shutdown handler joins its own thread, and its except swallows every later cleanup error #13585
Description
Observed in the 2026-08-04 log sweep: initialization.lifespan - ERROR - Error during shutdown: cannot join current thread (x4). Runs on every restart.
Cause
autobot-backend/initialization/lifespan.py:2406
except Exception as e:
logger.error("Error during shutdown: %s", e)Two problems:
- The join itself — something in the shutdown path calls
.join()on the thread it is currently running in, which always raisesRuntimeError: cannot join current thread. - The blast radius — that single
exceptwraps the whole cleanup body. The first failure aborts every subsequent step, and the only trace is oneERRORline with no traceback. Today the first failure is the cosmetic self-join, which means any real cleanup below it has been silently skipped on every restart for as long as this has been firing.
The second point is why this is not cosmetic: the log line understates what was lost, and there is no way to tell from the log which cleanup steps ran.
Acceptance criteria
- Identify and fix the self-join (do not join the current thread)
- Cleanup steps fail independently — one failing step does not skip the rest
- The handler logs
exc_info=Trueso the failing step is identifiable from logs alone - Test asserts a raising cleanup step does not prevent later steps from running
Activity
mrveiss commentedon Aug 23, 2026
Fixed in PR #14898, which takes this together with the four other items of the same defect class under umbrella #13852: a failed or impossible operation reporting success or health. Every fix is mutation-checked (the fixed line broken in a scratch copy, the test confirmed red, then discarded); the mutation table is in the PR body.
style(lint): satisfy isort on the files this branch touches (#14843, #…
fix(lifespan): assert the shutdown guard by driving it, not by grepin…
fix(observability): a failed or impossible operation must never repor…
mrveiss commentedon Aug 23, 2026
Delivered and merged as
42ac95b53(PR #14898).a1721728c.code-reviewerpass. Verdict: ready to merge. It traced every one of the five fixes for a new silent failure one layer up — a health aggregator that cannot be reached, a counter whose backend is down, a monitor whose probe binary is absent — and found none.Verified in base after merge: zero
|| echofabricating sentinels remain inaccess_control_monitor.sh, and_KNOWN_FABRICATINGis genuinely emptied rather than the guard being weakened to accommodate the change.How each failure is now observable:
/statusalways healthysuccess=FalseandUNKNOWNnaming the exception type; an empty probe registry counts as failureload_catalogreturned True either waydegradedwhile built-ins are in useexceptswallowed every later cleanup errorautobot_audit_write_failures_total{action,error_type}, the same counter the RBAC middleware usesUNKNOWN/{}/0|0|0#12969 was deliberately not closed. Its acceptance is host evidence (
Loaded error catalog ... from <path>, no fallback warning) that a checkout cannot produce, and the second candidate path fixes a layout inferred from a log rather than observed. It stays open.A severity claim in this PR was overstated and has been corrected — see the correction on #14880. The backticks executing as shell commands produced two spurious stderr lines per run, not corrupted output; the reviewer reproduced the exact structure and measured the captured value as correct. The genuinely serious bug found alongside it stands: the monitor's pre-flight imported
backend.services.feature_flags, a module that has never existed here, somain()exited 1 before the dashboard ran on every invocation.Reviewer notes carried forward, neither blocking:
cleanup_services(app)'s newlist[str]return is not consumed by its only caller (the failure signal is fully visible in the log either way), and/api/system/healthitself still reportsokon zero probes — clamped at the consumer instead, a disclosed scope boundary.