Worker: build the cleanup mark's guard after the lock is released

[amankrx]

What and why

Three of four workers on the drydock bed stopped executing in the middle of a TensorFlow run while looking healthy: keepalives and readiness fine, the scheduler listing each at twelve of twelve, no timer-driven warning from any action. A native backtrace showed twelve of the eighteen runtime threads blocked in parking_lot::RawMutex::lock_slow, one of them inside perform_cleanup through CleanupGuard::drop, called from the orphan sweep. perform_cleanup took the cleaning_up_operations lock and built its guard with then_some, which constructs the argument whatever the boolean says, so when the operation was already marked the guard was dropped at once and its Drop took the same non-reentrant lock on the same thread. That thread never returns, and every thread that then comes to clean up or start an action queues behind it, while the keepalive task keeps the worker alive and unevictable. The orphan sweep marks every directory it visits every orphan_sweep_interval_s, and a directory whose action is mid-cleanup is already marked, so the double mark is routine; every freeze sat on the sweep's cadence from the worker's start. The guard is now built only after the lock is released. The mark's other two holders are the orphan sweep and a retry removing its earlier attempt's stale directory, both brief, so an action's own cleanup now waits for the mark instead of stepping aside: stepping aside while the sweep held it skipped that cleanup for good and left the action's entries and reservations behind. The sweep checks ownership before taking the mark and again under it, giving it back at once for a directory that became owned in between, and the retry path takes the mark for its removal so it cannot race the sweep on the same tree.

How was this verified?

running_actions_manager_test: with a mark held, a second mark and the sweep run on their own thread; the test waits ten seconds and fails instead of hanging. On the old code it deadlocks; on the new one the second mark is refused, the sweep leaves the directory to its cleanup, and removes it once the mark is released. Two more: an action's cleanup started while the mark is held does nothing until the mark is released and then removes the directory and its entry, and a retry started while the mark is held leaves the stale tree untouched until the release and then replaces it. On the bed four rebuilt workers ran through every sweep tick of two TensorFlow runs, and a planted two-hour-old directory on a live worker was removed on the next tick.

Risk

An action whose cleanup used to be skipped under a held mark now waits for it, for as long as a sweep entry or a stale removal takes. perform_cleanup, take_cleanup_mark and CleanupGuard become public for the tests.

AI assistance

An agent drafted the change and I reviewed every line.

---

This change is 

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
nativelink	Ready	Preview	Sep 30, 2026 4:33pm UTC
nativelink-aidm	Ready	Preview	Sep 30, 2026 4:33pm UTC