Skip to content

Persistent workers: a configurable pool with an idle sweeper, the action environment, namespaces, measured usage, and a wait at the cap - #2830

Merged
amankrx merged 8 commits into
TraceMachina:mainfrom
amankrx:pr/persistent-worker-pool
Sep 30, 2026
Merged

amankrx merged 8 commits into
TraceMachina:mainfrom
amankrx:pr/persistent-worker-pool

Conversation

@amankrx

@amankrx amankrx commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

What and why

The persistent worker pool had no configuration, no idle reaping, and served an action without its environment, outside the worker's namespaces and with no usage reported. persistent_workers on the worker config now sizes it: max_workers_per_key (default 4), idle_timeout_s (default 300, a sweeper retires a process idle past it), max_requests_per_worker (default 200), shutdown_grace_ms and acquire_timeout_s. The pool key carries the executable, its arguments, the environment and the protocol; the process starts with the worker's additional_environment (the sources that are the same for every action) and the action's own variables on top, as a one-shot action does, a process is spawned in the same namespaces as a one-shot action, its measured usage goes into the result, and an action at the cap waits acquire_timeout_s (default 30) for a returned process before it runs one-shot; that wait is raced against the scheduler's kill and bounded by the action's own timeout, which the dispatch then gets the remainder of. The key's Debug prints environment names, never values, since it is logged and quoted in errors; the CPU reported for a pooled action is the process's spend from the moment sampling began, and its peak memory is documented as a whole-process figure.

How was this verified?

Pool tests: the key tells environments apart and ignores their order, the sweeper retires a worker idle past the timeout, and an acquire at the cap waits for a returned worker. The running actions manager tests build the configuration with the new settings, the persistent workers page and the config reference are updated, and the snippet lint passes.

Risk

PersistentWorkersConfig::default() has the pool on, as the serde default and the docs say. Defaults match what the pool did before, except that the sweeper now retires a process idle for five minutes and an action at the cap waits up to thirty seconds before falling back. A pooled process stays outside soft memory enforcement, as the enforcement docs say.

AI assistance

An agent drafted the change and I reviewed every line.


This change is Reviewable

…ion environment, namespaces, measured usage, and a wait at the cap
@vercel

vercel Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Sep 30, 2026 3:02am UTC
nativelink-aidm Ready Ready Preview Sep 30, 2026 3:02am UTC

Request Review

Comment thread nativelink-worker/src/persistent_worker/pool.rs Outdated
Comment thread nativelink-worker/src/running_actions_manager.rs Outdated
Comment thread nativelink-worker/src/running_actions_manager.rs
Comment thread nativelink-worker/src/persistent_worker/live_worker.rs
Comment thread nativelink-config/src/cas_server.rs
Comment thread nativelink-config/src/cas_server.rs
… cap is raced against the kill and bounded by the action's time, CPU as a delta, one environment helper, the pool on by default
…r-pool

# Conflicts:
#	web/apps/docs/content/docs/reference/nativelink-config/main.mdx
corcillo
corcillo previously approved these changes Sep 30, 2026
@amankrx
amankrx merged commit 035dd63 into TraceMachina:main Sep 30, 2026
46 of 47 checks passed

This branch was successfully deployed

2 active deployments
Preview – nativelink — 0f7a801d Deployed Sep 30, 2026 by vercel[bot]
Preview – nativelink-aidm — 0f7a801d Deployed Sep 30, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants