Skip to content

Worker: kill the whole process group when an action times out or is cancelled - #2809

Merged
amankrx merged 6 commits into
TraceMachina:mainfrom
amankrx:pr/worker-kill-grace
Sep 29, 2026
Merged

amankrx merged 6 commits into
TraceMachina:mainfrom
amankrx:pr/worker-kill-grace

Conversation

@amankrx

@amankrx amankrx commented Sep 28, 2026 •

Copy link
Copy Markdown
Member

What and why

A timed-out or cancelled action was killed by its direct child only: a shell that had forked a compiler, or a test runner its workers, left the descendants running with their memory and CPU charged against the next action on the worker. The action is already its own process group leader, so the kill now goes to the whole group. It lives in the child wrapper's kill, so the timeout, cancel and drop-guard paths all get it; a namespaced action is unchanged, its stub already tears the sandbox down.

How was this verified?

running_actions_manager_test: an action whose child spawns a grandchild that writes its pid is cancelled, and the grandchild is gone afterwards; the test runs without namespaces so it holds on a plain worker too. Linux only, like the code it tests. On a test cluster, cancelling a build that had forked sleep children left no stray processes on the worker where it used to leave one per action.

Risk

Low. Only the non-namespaced kill changes, and only on Linux. A killpg failure other than ESRCH is logged as an error rather than dropped. An action that deliberately detaches a daemon into its own process group is unaffected, as before.

AI assistance

An agent drafted the change and I reviewed every line.


This change is Reviewable

@vercel

vercel Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Sep 29, 2026 4:27am UTC
nativelink-aidm Ready Ready Preview Sep 29, 2026 4:27am UTC

Request Review

(cherry picked from commit 53257f3fc37576ea2bb74bbe1e8a39a7df79004a)
(cherry picked from commit 45ee98fc54abe9cccbc6d2f416bbde2dd12bbc0c)
(cherry picked from commit 76312e6e521b265ca01e74430a8fbf3ee6bbacd7)
Comment thread nativelink-worker/src/running_actions_manager.rs Outdated
Comment thread nativelink-worker/src/running_actions_manager.rs
Comment thread nativelink-worker/src/running_actions_manager.rs Outdated
Comment thread nativelink-worker/src/running_actions_manager.rs Outdated

This branch was successfully deployed

2 active deployments
Preview – nativelink — 6e715467 Deployed Sep 29, 2026 by vercel[bot]
Preview – nativelink-aidm — 6e715467 Deployed Sep 29, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants