Skip to content

fix: only adopt the real gateway after a crash (#194) - #200

Open
vkpeter wants to merge 1 commit into
techartdev:mainfrom
vkpeter:fix/gateway-adoption-194
Open

vkpeter wants to merge 1 commit into
techartdev:mainfrom
vkpeter:fix/gateway-adoption-194

Conversation

@vkpeter

@vkpeter vkpeter commented Oct 19, 2026

Copy link
Copy Markdown

Fixes #194.

Problem

After the gateway exits (OOM kill, crash), the supervisor loop calls find_gateway_daemon_pid to look for a self-restarted daemon. Tier 3 returns the first process with openclaw anywhere in its command line, and tier 2 (pgrep -f openclaw-gateway) also matches any shell containing that string. The loop then polls that PID and does not relaunch the gateway until it exits.

Seen on a Raspberry Pi 4 (add-on 0.5.95, OpenClaw 2026.10.4) on 19 Oct 2026: the gateway was OOM-killed twice, and both times the supervisor logged OpenClaw runtime active (PID …); monitoring. for an unrelated process: a scheduled shell job (cd /config/.openclaw/workspace/..., runs up to ~100 min), and once a monitoring loop that had openclaw-gateway in its command line. Nothing listened on the gateway port; the ingress watchdog stayed green.

While the gateway was healthy, the old tier 3 would have adopted 4 processes on that system, including the gateway's own helper workers (.../openclaw/dist/process/spawn-broker/worker.js, .../infra/sqlite-readonly-location.worker.js), which can outlive a killed gateway.

Change

Implements points 1 and 2 of the fix proposed in #194:

  • Tier 2: pgrep -fx openclaw-gateway (exact process title), skipping known and rejected PIDs.
  • Tier 3: a new is_gateway_entrypoint_pid accepts only the title openclaw-gateway or argv [node] <path>/openclaw[.mjs|.js] gateway run [...]. Anything that merely mentions openclaw is ignored. The /proc cmdline is NUL-padded after process.title is set, so empty fields are skipped.
  • Port deadline: an adopted non-child must own GATEWAY_INTERNAL_PORT within GW_ADOPT_PORT_DEADLINE (300 s). If not, it is logged, added to GW_REJECTED_PIDS (never adopted again) and the supervisor relaunches. If the port turns out to be owned by a different PID, the loop switches to that PID. Remote mode is unchanged.
  • The tier 1 lookup moved into gateway_port_owner_pid so the loop can reuse it.

Why 300 s and not the 120 s suggested in the issue: on the Pi 4 above, a cold start of 2026.10.4 took ~260 s before the port was bound (startup phase: ready … total=258127ms). With 120 s a slow but real gateway would be dropped and relaunched.

Point 3 of #194 (pointing the add-on watchdog at the gateway) is left out to keep this change small.

Testing

  • bash -n openclaw_assistant/run.sh
  • Sourced the new functions inside the running add-on container (0.5.95) and checked them against live processes:
process is_gateway_entrypoint_pid
running gateway (title openclaw-gateway) yes
sh -c 'cd /config/.openclaw/workspace; sleep 40' no
sh -c 'sleep 40; echo openclaw gateway run' no
argv node /tmp/fk/openclaw gateway status no
argv node /tmp/fk/openclaw gateway run yes
  • find_gateway_daemon_pid with an unused port and GW_PID set to the live gateway returned nothing (the old tier 3 returned one of 7 candidates). With a fake gateway run process it returned that PID, and nothing once that PID was in GW_REJECTED_PIDS. With the real port, tier 1 returned the gateway PID.
  • The port-deadline branch in the supervisor loop is not exercised by an automated test; it reuses the same ss lookup as tier 1.

After the gateway exited, find_gateway_daemon_pid could return any process
with "openclaw" in its command line: gateway helper workers, agent CLI
children, scripts under /config/.openclaw. The supervisor then polled that
PID and did not relaunch the gateway until it exited, which for a
long-running script can take hours.

- Tier 2 matches the process title exactly (pgrep -fx openclaw-gateway).
- Tier 3 accepts only the gateway entrypoint ([node] .../openclaw gateway run).
- An adopted PID must own the gateway port within 300 s, otherwise it is
  dropped and never adopted again, and the supervisor relaunches.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: Supervisor adopts any process with "openclaw" in its cmdline as the restarted gateway, so a crashed gateway can stay down

1 participant