Skip to content

fix(backend): enable faulthandler so the SIGABRT crash-loop is diagnosable (#12777) - #12815

Merged
mrveiss merged 5 commits into
Dev_new_guifrom
issue-12777
Jul 27, 2026
Merged

mrveiss merged 5 commits into
Dev_new_guifrom
issue-12777

Conversation

@mrveiss

@mrveiss mrveiss commented Jul 27, 2026

Copy link
Copy Markdown
Owner

Thinking Path

autobot-backend aborts with SIGABRT every ~25–55 minutes and produces zero diagnostics. The issue already ruled out every ordinary cause from cgroup counters — memory.events all-zero with a 3.4 GiB peak against a 12 GiB cap, pids.peak 65 against a 512 limit — so this is a native abort. And a native abort is precisely what the current setup cannot record.

Two things independently destroy the forensic trail:

  1. faulthandler was never enabled — no PYTHONFAULTHANDLER, no -X faulthandler. A fatal signal inside a C extension unwinds without printing a Python stack, so backend-error.log stops mid-stream and resumes at the next boot.
  2. Core capture is broken on the platform — core_pattern pipes to /wsl-capture-crash, which does not exist. systemd reports code=dumped while the core is discarded.

So the first fix is not a fix for the crash — it is making the crash observable. Without a stack, step 3 of the issue ("pin the aborting extension") is unactionable.

What Changed

One line on the canonical unit template (authoritative per #11670):

Environment="PYTHONFAULTHANDLER=1"

The next abort now writes a native + Python traceback to StandardError, which the same unit already appends to backend-error.log.

Deliberately not included

  • core_pattern is host/WSL configuration, not repo state. The issue itself asks for it to be either fixed or explicitly opted out of with a recorded reason — that's an owner call, not a drive-by.
  • Pinning or rebuilding the suspect extension (uvloop / chromadb / torch-onnx against a very new Python 3.14) stays a hypothesis until a stack exists. Worth being precise: the single uvloop OSError: [Errno 88] Socket operation on non-socket appears once in the whole log and preceded only one of the two observed aborts — it cannot explain both, and acting on it now would be guessing.

Verification

$ python3 -m pytest tests/test_backend_service_faulthandler_12777.py -q
3 passed

$ python3 -m pytest tests/ -q
976 passed, 1 skipped

Three render tests, following the test_pg_hba_multiuser.py precedent (#10636): faulthandler stays enabled; stderr still has somewhere to land (the two only work as a pair — enabling faulthandler while stderr goes nowhere leaves the crash just as invisible); and Restart=always survives, since the loop is only survivable because systemd restarts the unit.

Mutation-verified — deleting the env line fails the first test:

env line removed  → test_faulthandler_is_enabled FAILED
env line restored → 3 passed

The test registers Ansible's dirname/basename filters so the real template renders outside Ansible, rather than testing a copy.

Deployment note

This is a systemd unit change, so it only takes effect after the backend role is redeployed — it will not reach the box via a code-only sync. The crash loop stays undiagnosable until that happens.

Model Used

Claude Opus 5 (1M context)

Closes #12777

…sable (#12777)

autobot-backend aborts with SIGABRT every ~25-55 min and produces ZERO
diagnostics, so the cause cannot currently be identified at all. #12777 ruled
out every ordinary explanation from cgroup counters — memory.events all-zero
with a 3.4 GiB peak against a 12 GiB cap, pids.peak 65 against a 512 limit — so
this is a native abort, and a native abort is exactly what the current setup
cannot record.

Two things independently destroy the forensic trail:

  1. faulthandler was never enabled (no PYTHONFAULTHANDLER, no -X faulthandler),
     so a fatal signal inside a C extension unwinds without printing a Python
     stack. backend-error.log just stops mid-stream and resumes at the next
     boot.
  2. Core capture is broken on the deployment platform: core_pattern pipes to
     /wsl-capture-crash, which does not exist. systemd still reports
     code=dumped while the core is discarded.

Only (1) is fixable in this repo, and it is the prerequisite for (3) in the
issue — you cannot pin the aborting extension without a stack. Adds
Environment="PYTHONFAULTHANDLER=1" to the canonical unit template (the unit is
authoritative per #11670), so the next abort writes a native + Python traceback
to StandardError, which the same unit already appends to backend-error.log.

Deliberately NOT included:
  - core_pattern is host/WSL configuration, not repo state, and the issue itself
    asks for it to be fixed or explicitly opted out of with a recorded reason —
    an owner call, not a drive-by.
  - Pinning or rebuilding the suspect extension (uvloop / chromadb / torch-onnx
    against Python 3.14) stays a HYPOTHESIS until a stack exists. The single
    uvloop "Socket operation on non-socket" error preceded only ONE of the two
    observed aborts, so it does not explain both and must not be treated as the
    cause yet.

Tests: 3 render tests following the test_pg_hba_multiuser.py precedent (#10636),
pinning that faulthandler stays enabled, that stderr still has somewhere to land
(the two only work as a pair), and that Restart=always survives. Deleting the
env line fails the first — mutation-verified.

Full slm-backend suite: 976 passed, 1 skipped.

NOTE: this is a systemd unit change, so it only takes effect after the backend
role is redeployed — it will not reach the box via a code-only sync.

Closes #12777
@github-actions

Copy link
Copy Markdown
Contributor

✅ SSOT Configuration Compliance: Passing

🎉 No hardcoded values detected that have SSOT config equivalents!

mrveiss added a commit that referenced this pull request Jul 27, 2026
…us (#12818) (#12819)

My #12802 fix stopped auto-update CRASHING but did not make it work. On base
head f6639e4 the job went green while reporting:

  Dev_new_gui was updated — checking open PRs for staleness...
  All PR branches are up to date (or none could be listed).

Three PRs (#12806, #12815, #12817) were BEHIND at that exact moment and each
still needed a manual PUT /pulls/{n}/update-branch.

Cause: mergeStateStatus is computed LAZILY by GitHub. Immediately after a push —
exactly when this workflow fires — it returns UNKNOWN for the very PRs that just
went stale, so `select(.mergeStateStatus == "BEHIND")` matched nothing and the
script took the "all up to date" branch. The failure mode moved from loud-and-
broken to quiet-and-broken, which is worse in one respect: a passing required-
adjacent check now asserted work that never happened.

Fix: ask for the comparison directly. GET /repos/{repo}/compare/{base}...{head}
is evaluated on demand and is accurate at push time, so no retry loop is needed.

Verified by running the new detection against the live repo while those three
PRs were behind:

  PR #12817  behind_by=2  (issue-12778)
  PR #12815  behind_by=1  (issue-12777)
  PR #12806  behind_by=1  (issue-12780)

i.e. it finds precisely the PRs the old filter reported as up to date.

$HEAD_REF is now interpolated into an API PATH, so it is validated the same way
$PR already was: rejected unless it matches ^[A-Za-z0-9._/-]+$, and explicitly
rejected if it contains a '..' segment (which would traverse to a different
endpoint) or begins with '-'.

Everything else from #12802 is retained: the REST update-branch call (verified
on 8 PRs during the 2026-07-27 drain), the `|| true` guards, the numeric $PR
validation, and the reachable trailing exit 0.

Closes #12818
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant