fix(ci): reap leaked Activity gate servers and wait for a quiet host

[inureyes]

Summary

The WebUI installed artifact job's Activity performance gate had two environmental faults on the GB10 runner (#1949). It leaked its server when the verifier was killed or its cleanup gave up, and the leaked server then failed every later run at the GPU precondition, which printed only a count (on 2026-10-05, five consecutive runs of PR #2117 after main 680f64e failed in scoring). It also scored while other work ran on the host and reported degradations inside its own noise. This PR closes both in the job itself.

Changes

• scripts/webui/activity_gate_host.py precheck replaces the bare nvidia-smi count as the first command of the gate body (after the CI flock and gpu-lock):
  • A process carries the job's identity when its executable is in $RUNNER_TEMP/mlxcel-webui-installed/, or its working directory or HOME is a verifier run directory ($RUNNER_TEMP/mlxcel-activity-performance-*). The verifier gives the server, node and the browser HOME=<run dir>/home; Playwright launches the browser detached, in a process group of its own, so HOME is what reaches it.
  • Stops a process of that identity an earlier run leaked: orphaned (PPID 1), or with its directory or executable reading (deleted) or its run directory gone. A live process of that identity, or a leak owned by another uid, fails closed instead.
  • Fails closed if any GPU compute process remains, before the quiet wait (so a conflict is reported at once) and again after it, naming each with pid, process name, memory, parent and uid. Executable and directory paths are printed only for the job's own processes, since logs and artifacts are public.
  • Waits up to 600 s until the mean runnable-task count from /proc/loadavg over 10 one-second samples is at most a quarter of the CPUs (5 on GB10), and fails closed with a host snapshot otherwise. Load averages and ps %cpu lag a fresh compile, so neither is used, and the window must fill before the gate can refuse.
  • Writes the observed host state (snapshots at start and end with load averages and compiler pids, quiet-wait samples, anything reaped, any refusal) to webui-activity-host-state.json, uploaded with the diagnostics.
• activity_gate_host.py reap, a new if: always() step after the gate: stops the process groups the verifier's pid file names and anything else of the job's identity, prints what it found, and fails the job if anything survives or the pid file is unreadable (the identity sweep still runs). A missing pid file is a clean no-op.
• Signalling safety: processes with a controlling terminal (a developer's shell on the runner) are never touched; a process matched only by directory or HOME is signalled alone, never with its group; signals go through a pidfd checked against the recorded start time, so a reused pid is never signalled; pid-file records carry the boot id.
• verify_activity_performance.py:
  • --pid-file records every owned process (Xvfb, openbox, the server, the node activity script) outside the run directory as it starts; Xvfb and openbox now run in the run directory.
  • --host-state embeds the precheck's state into the full evidence as host_state.gate_start, plus a host_state.after_activity load sample taken before the output is validated, so a noisy verdict keeps it. The summary does not carry it and summarize_activity_evidence.py is unchanged.
  • terminate_owned no longer raises out of its SIGKILL wait or on a PermissionError, and it stops group members a quickly exiting leader left behind instead of only recording process_group_empty: false.
• Step timeout 65 to 75 minutes for the quiet wait, job timeout 90 to 120 for two cold builds plus that step; docs in docs/webui-integration-matrix.md; the new tests join make verify-webui-helper-tests and the job's helper test step.

The server spawns no child processes (model workers are in-process threads), so its process group is the whole server. The verdict rule and thresholds are untouched (#1925).

Changes during review

The security review found that a directory-only match could SIGKILL a developer's shell and its whole group, that node and the browser had no reap handle, and that paths of unrelated GPU processes would reach public logs. All three are fixed above, together with pidfd signalling, tolerant pid-file parsing, the GPU check before the quiet wait, and the after-activity sample on the failure path from the implementation review.

Verification

• make verify-webui-helper-tests: all suites pass (18 host-helper tests, 21 verifier tests, 10 summary tests). The two terminate_owned tests fail on the previous code: one raises subprocess.TimeoutExpired from the SIGKILL wait, the other ends with the group still populated. actionlint 1.7.7 is clean.
• Runner, lablup-dgxspark21 (machine spark-101):
  • Run 37387059748 (first head): the precheck found the real leaked server from the earlier failed run, pid 1453937 with executable and cwd both (deleted) and PPID 1, stopped it with SIGTERM, and the gate then passed. This is the leak that failed PR fix(generate): sample each penalized step against the token just read #2117's five runs.
  • Run 37388347876 (final head 634444a): green. Host quiet at mean 1.4 runnable tasks (threshold 5), no GPU process, server_shutdown exit_code 0, forced false, process_group_empty true, full evidence carries host_state.gate_start and after_activity, and reap found nothing to do.
  • Run 37388422039, job 112027483381 (throwaway probe PR fix(ci): runner probe for #1949 evidence, do not merge #2120 on top of the final head, closed): under one busy loop per CPU the precheck refused with host did not become quiet within 60s: mean runnable tasks 23.8 over the last 10 samples, threshold 5; with the load removed it passed at 1.1. The verifier was then SIGKILLed mid-activity while the server held 4604 MiB, and the always() reap stopped Xvfb, openbox, the server, node and the browser (the last by HOME); the following check found nvidia-smi --query-compute-apps and pgrep -af mlxcel-server-webui both empty.
• GB10 simulation on spark-102 (real mlxcel-server serving qwen3-0.6b-4bit, under gpu-lock): a leaked server (PPID 1, deleted run directory, 4116 MiB) was named and stopped; a foreign GPU process was refused at once, named without its path, and left running; a busy host (another session's compile) was refused after the wait; a server left by a SIGKILLed verifier was reaped by pid file.

Closes #1949