Repository navigation
fix(backend): enable faulthandler so the SIGABRT crash-loop is diagnosable (#12777) - #12815
Merged
Merged
Conversation
…sable (#12777) autobot-backend aborts with SIGABRT every ~25-55 min and produces ZERO diagnostics, so the cause cannot currently be identified at all. #12777 ruled out every ordinary explanation from cgroup counters — memory.events all-zero with a 3.4 GiB peak against a 12 GiB cap, pids.peak 65 against a 512 limit — so this is a native abort, and a native abort is exactly what the current setup cannot record. Two things independently destroy the forensic trail: 1. faulthandler was never enabled (no PYTHONFAULTHANDLER, no -X faulthandler), so a fatal signal inside a C extension unwinds without printing a Python stack. backend-error.log just stops mid-stream and resumes at the next boot. 2. Core capture is broken on the deployment platform: core_pattern pipes to /wsl-capture-crash, which does not exist. systemd still reports code=dumped while the core is discarded. Only (1) is fixable in this repo, and it is the prerequisite for (3) in the issue — you cannot pin the aborting extension without a stack. Adds Environment="PYTHONFAULTHANDLER=1" to the canonical unit template (the unit is authoritative per #11670), so the next abort writes a native + Python traceback to StandardError, which the same unit already appends to backend-error.log. Deliberately NOT included: - core_pattern is host/WSL configuration, not repo state, and the issue itself asks for it to be fixed or explicitly opted out of with a recorded reason — an owner call, not a drive-by. - Pinning or rebuilding the suspect extension (uvloop / chromadb / torch-onnx against Python 3.14) stays a HYPOTHESIS until a stack exists. The single uvloop "Socket operation on non-socket" error preceded only ONE of the two observed aborts, so it does not explain both and must not be treated as the cause yet. Tests: 3 render tests following the test_pg_hba_multiuser.py precedent (#10636), pinning that faulthandler stays enabled, that stderr still has somewhere to land (the two only work as a pair), and that Restart=always survives. Deleting the env line fails the first — mutation-verified. Full slm-backend suite: 976 passed, 1 skipped. NOTE: this is a systemd unit change, so it only takes effect after the backend role is redeployed — it will not reach the box via a code-only sync. Closes #12777
Contributor
✅ SSOT Configuration Compliance: Passing🎉 No hardcoded values detected that have SSOT config equivalents! |
mrveiss
added a commit
that referenced
this pull request
Jul 27, 2026
…us (#12818) (#12819) My #12802 fix stopped auto-update CRASHING but did not make it work. On base head f6639e4 the job went green while reporting: Dev_new_gui was updated — checking open PRs for staleness... All PR branches are up to date (or none could be listed). Three PRs (#12806, #12815, #12817) were BEHIND at that exact moment and each still needed a manual PUT /pulls/{n}/update-branch. Cause: mergeStateStatus is computed LAZILY by GitHub. Immediately after a push — exactly when this workflow fires — it returns UNKNOWN for the very PRs that just went stale, so `select(.mergeStateStatus == "BEHIND")` matched nothing and the script took the "all up to date" branch. The failure mode moved from loud-and- broken to quiet-and-broken, which is worse in one respect: a passing required- adjacent check now asserted work that never happened. Fix: ask for the comparison directly. GET /repos/{repo}/compare/{base}...{head} is evaluated on demand and is accurate at push time, so no retry loop is needed. Verified by running the new detection against the live repo while those three PRs were behind: PR #12817 behind_by=2 (issue-12778) PR #12815 behind_by=1 (issue-12777) PR #12806 behind_by=1 (issue-12780) i.e. it finds precisely the PRs the old filter reported as up to date. $HEAD_REF is now interpolated into an API PATH, so it is validated the same way $PR already was: rejected unless it matches ^[A-Za-z0-9._/-]+$, and explicitly rejected if it contains a '..' segment (which would traverse to a different endpoint) or begins with '-'. Everything else from #12802 is retained: the REST update-branch call (verified on 8 PRs during the 2026-07-27 drain), the `|| true` guards, the numeric $PR validation, and the reachable trailing exit 0. Closes #12818
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thinking Path
autobot-backendaborts with SIGABRT every ~25–55 minutes and produces zero diagnostics. The issue already ruled out every ordinary cause from cgroup counters —memory.eventsall-zero with a 3.4 GiB peak against a 12 GiB cap,pids.peak65 against a 512 limit — so this is a native abort. And a native abort is precisely what the current setup cannot record.Two things independently destroy the forensic trail:
PYTHONFAULTHANDLER, no-X faulthandler. A fatal signal inside a C extension unwinds without printing a Python stack, sobackend-error.logstops mid-stream and resumes at the next boot.core_patternpipes to/wsl-capture-crash, which does not exist. systemd reportscode=dumpedwhile the core is discarded.So the first fix is not a fix for the crash — it is making the crash observable. Without a stack, step 3 of the issue ("pin the aborting extension") is unactionable.
What Changed
One line on the canonical unit template (authoritative per #11670):
The next abort now writes a native + Python traceback to
StandardError, which the same unit already appends tobackend-error.log.Deliberately not included
core_patternis host/WSL configuration, not repo state. The issue itself asks for it to be either fixed or explicitly opted out of with a recorded reason — that's an owner call, not a drive-by.OSError: [Errno 88] Socket operation on non-socketappears once in the whole log and preceded only one of the two observed aborts — it cannot explain both, and acting on it now would be guessing.Verification
Three render tests, following the
test_pg_hba_multiuser.pyprecedent (#10636): faulthandler stays enabled; stderr still has somewhere to land (the two only work as a pair — enabling faulthandler while stderr goes nowhere leaves the crash just as invisible); andRestart=alwayssurvives, since the loop is only survivable because systemd restarts the unit.Mutation-verified — deleting the env line fails the first test:
The test registers Ansible's
dirname/basenamefilters so the real template renders outside Ansible, rather than testing a copy.Deployment note
This is a systemd unit change, so it only takes effect after the backend role is redeployed — it will not reach the box via a code-only sync. The crash loop stays undiagnosable until that happens.
Model Used
Claude Opus 5 (1M context)
Closes #12777