Skip to content

Matching pass: room that opens mid-pass goes to the oldest waiting action, one store read per queued action - #2806

Merged
amankrx merged 4 commits into
TraceMachina:mainfrom
amankrx:pr/matching-pass
Sep 28, 2026
Merged

amankrx merged 4 commits into
TraceMachina:mainfrom
amankrx:pr/matching-pass

Conversation

@amankrx

@amankrx amankrx commented Sep 28, 2026 •

Copy link
Copy Markdown
Member

What and why

A pass reads the queue one action at a time, about a millisecond each on a store-backed queue. When a worker freed up mid-pass, its room went to whatever action the pass was reading, newer than everything it had already passed over. Actions that find no room are now parked in listing order, highest priority first then oldest, and when the fleet's capacity changes the pass offers the room to them in that order before the one in hand, asking the fleet once per property shape. A keepalive reporting more free memory counts as room opening too. scheduler.matching.parked_dispatched counts the parked actions a worker was actually told to run.

The listing also serves each action from the record it already read instead of fetching it again, one store read per queued action instead of two.

How was this verified?

Gated-queue tests free a worker mid-pass and check the oldest waiting action gets the slot, check two slots go to the two oldest across interleaved shapes rather than draining one shape, and check a higher memory report mid-pass goes to the older vetoed action; each fails without its fix. Store tests cover the listing serving the loaded record and keepalives still merging into it. On a test cluster, a 1,000-action burst went from 669 dispatched out of order to 0, and the pass from 40 ms to 17 ms.

Risk

Medium: the matching pass is rewritten, though its outcome per action is the same. Parking is capped at 10,000 actions per pass. No config, wire or data changes.

AI assistance

An agent drafted the change and I reviewed every line.


This change is Reviewable

(cherry picked from commit cb4642683c8821b4582e4792fc13b337fa1d1346)
(cherry picked from commit f9ca73fa1be6d3ce6a00326f2b71b056a5c9e520)
@vercel

vercel Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Sep 28, 2026 9:39pm UTC
nativelink-aidm Ready Ready Preview Sep 28, 2026 9:39pm UTC

Request Review

Comment thread nativelink-scheduler/src/simple_scheduler.rs Outdated
Comment thread nativelink-scheduler/src/api_worker_scheduler.rs Outdated
Comment thread nativelink-scheduler/src/simple_scheduler.rs Outdated
Comment thread nativelink-scheduler/src/simple_scheduler.rs Outdated
… report is room opening, the parked metric counts confirmed sends
@amankrx
amankrx merged commit 2762510 into TraceMachina:main Sep 28, 2026
48 of 49 checks passed

This branch was successfully deployed

2 active deployments
Preview – nativelink — e52fc477 Deployed Sep 28, 2026 by vercel[bot]
Preview – nativelink-aidm — e52fc477 Deployed Sep 28, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants