Skip to content

scheduler: validate placement requirements at assignment (N3) - #46

Open
b7r6 wants to merge 1 commit into
mainfrom
fix/n3-placement-version
Open

b7r6 wants to merge 1 commit into
mainfrom
fix/n3-placement-version

Conversation

@b7r6

@b7r6 b7r6 commented Oct 7, 2026 •

Copy link
Copy Markdown
Owner

What and why

Lane A stability fix from the findings catalog (see nativelink-findings-2026-10-05.csv
on the meta/findings-catalog-2026-10-05 branch), rebased onto OSS main
@bf4b3aa855. This fork PR exists to validate the change on CI that is
representative of upstream PR lanes (public read-only cache, same endpoints)
before any upstream submission.

How was this verified?

The branch carries its own regression tests (fails-without where the fix
class allows it — see the branch's test commits). This PR's purpose is the
full representative CI run; green here rehearses the exact checks an
upstream PR would face.

Risk

Fork-internal validation PR; nothing lands anywhere by merging this. The
underlying fix's risk assessment happens at upstream-submission time.

AI assistance

Agent-assisted pipeline (Claude) for rebase and PR mechanics under my
direction; the fix itself carries verification in its tests.

@b7r6
b7r6 force-pushed the fix/n3-placement-version branch 3 times, most recently from fa9fe06 to af4578c Compare October 7, 2026 20:34
Matching computed typed resource requirements once and parked that
snapshot; assignment reloaded the current action but only guarded
against double-assignment. It never compared the reloaded action's
requirements against the ones the matcher used to pick the worker.

So a matcher could park a 4 GiB snapshot of X; a peer runs X, hits a
memory-limit kill, requeues X escalated to 8 GiB; the stale 4 GiB
snapshot then dispatched X to a 4 GiB-only worker. The DB said 8 GiB
but StartExecute and the reservation carried 4 GiB — the escalation
was silently bypassed.

Fix: assign_operation now carries the placement-relevant requirement
snapshot the worker was chosen for. The accept, inside the same CAS
that flips the action to Executing, rejects (Aborted) if the action's
current platform_properties no longer match that snapshot. A stale
proposal becomes a lost race; the action stays queued and is rematched
against its current requirements. Dispatch still sends the parked
payload, so rejecting before send keeps DB and worker in agreement.

Regression test assignment_rejects_a_stale_placement_after_escalation
reproduces the two-matcher interleaving: it FAILS without the guard
(the 4 GiB placement is silently accepted to Executing) and PASSES
with it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@b7r6
b7r6 force-pushed the fix/n3-placement-version branch from af4578c to 21a424d Compare October 8, 2026 16:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant