Two-Mac cluster serving: real JACCL worker, guided link setup, coordinator and provider fixes

[jonathan308]

Summary

Experimental two-Mac ("cluster") serving: one model split across two Macs as a two-stage layer pipeline over MLX's JACCL backend (RDMA over Thunderbolt 5). This PR carries the provider and runtime side of the foundation extracted from the research drafts (#1226, #1227), completes the parts that extraction left out, and adds what was needed to run it on real hardware. Ordinary single-Mac serving is unchanged; everything here is opt-in (darkbloom start --local --distributed, darkbloom cluster …).

Status: draft. As of 2026-10-12 the pair serves through the product path and through a locally built coordinator, including a public request with billing, on an M3 Ultra + M5 Ultra over RDMA, with SSH-free guided setup. See the round 3 update below and "Not shown yet".

Repository ownership (2026-10-09). #1408 moved all coordinator and backend code to Layr-Labs/darkbloom-platform. The coordinator half of this work is now a draft pull request there (number 6, branch feat/cluster-pairs), and coordinator/ on this branch is restored to master's content, byte for byte (763dea5b4). This pull request is provider-side only: provider-swift/, libs/, the shared wire-contract pages and the handoff files. Coordinator commits made before I had read that rule remain in the branch history; the tree no longer carries them, and none will be added.

Update 2026-10-12: integration round 3 (SSH-free guided setup, public serving, paged KV and MTP by default, supervision, observability)

Head d88ba3ee6, 227 signed commits on fa41f769d, provider and runtime only. Merged in order: the SSH-free guided setup (S07/S08 link session, S17 pairing, S09 pair serving slot, S16 encrypted data plane), the public end-to-end path (registration v3, stop/cancel receipts, and a memory-gate fix that counts speculative file pages, without which a 256 GB M5 Ultra refused every stage), per-model pair-serving grants (D4), the prefix-cache memory fix (F1), the request envelope (S19), always-on supervision (S18), cluster observability (S24), universal placement with the planner crediting MTP, paged KV in cluster stages by default, and the owner writer waking on enqueue. Three integration fixes. The optimization experiments (tensor parallel, SSD prefix tier, hand-off speedups, N-Mac, exact rows, decode engine) are not in this round.

Hardware: new test pair, M3 Ultra (256 GB) + M5 Ultra (256 GB) over Thunderbolt 5 RDMA, both cable ports isolated by link setup v2 (one macOS admin approval per Mac). All on bin/integration3 unless noted:

• Guided setup with no SSH configuration from fresh homes on both Macs: link check, pairing by code, plan over the paired link, hand-over, serve the 9B and the 27B, restart both members with nothing asked. Prompts seen: 0 firewall, 0 Local Network.
• Product path (darkbloom start --local --distributed + darkbloom cluster link-follower, no SSH in setups): 9B and 27B with phase split + MTP; prefix reuse (repeat first token 0.17 s 9B, 0.39 s 27B); client hang-up; clean stop with memory back on both Macs.
• Public request served by the pair through a locally built coordinator (darkbloom-platform PR 6 branch): encrypted, peer-verified data plane; billing rows carrying the cluster session and member pins; owner earnings; uptime intervals; leader failover and re-formation; follower drain.
• Faults: rank 1 SIGTERM mid-decode gives the client an error frame, the leader ends itself, nothing is left, memory is back and a restart serves; guard v2's surviving rank exits 75 in 30.2 s.
• Owner writer wake: 9B pipeline on the product path +23 to +27 % frames/s (inter-token gaps 23-26 ms → 18-19 ms), tokens identical.
• Every family across the new pair (universal-families head, bin/universal9): all pass the near-tie rule against both Macs' own single-Mac references at the cuts a fresh install picks, except GPT-OSS at cut 4 (diverges at its second token; under investigation, see below).
• Paged KV by default (bin/paged2): tokens identical to the earlier paged runs on all 16 requests; 27B paged decode within 3 % of contiguous, 9B within 10 %.

Tests at the head: cluster library 246 (only the known file-cache environment test and one memory-interference test); worker 58; all 32 check runners including StartupChecks; provider native suite 4,635 tests with only the 7 baseline failures. Private-detail scan of every added line: clean.

Not shown yet / known gaps (tracked):

• Regression found after integration (fix in progress): with paged KV now the default for Qwen-class cluster stages, MTP verify windows do not set the paged cache's rectangular-attention serialization (the single-Mac engine does), so each verify round re-reads the whole context; prompt chunks also re-read earlier pages. Tokens are unaffected; speed is: 9B 30K 8.5 s contiguous vs 15.3 s paged, 27B 30K 23.7 s vs 42.0 s. With contiguous storage the pair beats the M5 Ultra alone at long context (9B 30K +5-7 %, 27B 30K +9 %, A3B 16K +20 %). DARKBLOOM_CLUSTER_KV_BACKEND=contiguous is the interim switch.
• GPT-OSS cut-4 divergence on the new pair: root cause in progress; until fixed, the planner must not offer that setup.
• The release build does not yet package darkbloom-cluster-worker / darkbloom-cluster-plan into Darkbloom.app; members do not start themselves after a reboot until the S18 login agent ships in a Developer-ID-signed build (which is also what makes the Local Network approval one-time).
• Pair-serving grants are still a hand-written per-model table (being replaced by grants derived from qualification records); a few fixed values remain in the guided flow's setups.
• The leader-first onboarding flow (darkbloom cluster on one Mac, then on the other; 2 commands and 4 keys on this pair) is proven on a separate branch and is not in this round.

Update 2026-10-11: integration round 2 (no fixed lifetime, 32K context, MTP and prefix reuse on by default, SSH-free link channel, setup wizard, MLX guard v2)

Ten first-parent commits on 1cb482bf8 (six merges, the MLX pin move, three fixes; provider and runtime only):

• No fixed session lifetime or request cap: a loaded model stays resident until it is stopped; a renewable rank lease (renew 5 s, TTL 30 s) ends workers whose owner disappears; keep-warm between requests.
• Context to 32,768 tokens for the dense pair; the prefill attention transient priced in; time budgets scale with the prompt.
• MTP on by default (phase_split_mtp_v1, drafter and verifier on the decoding Mac) for models that ship a head.
• Prompt-prefix reuse on by default, composed with MTP; real cached_tokens in usage.
• SSH-free authenticated link control channel hosted in the provider; guided setup wizard on bare darkbloom cluster.
• MLX pins: Layr-Labs/mlx fdae3174 (JACCL guard v2: 30 s limit, teardown exit with memory release; revert of the NAX quantised-MoE simdgroup skip) and mlx-swift 64070c4 (mlx-c 489e965).
• Fixes found while integrating: placement plans for the reservation and ranks admit the largest context at load; Nemotron interval-free plans always carry the transient; a lease renewal arriving just after a worker's stop no longer fails rank 0 (found on hardware).

Tests: cluster library 211/212 (one environment-dependent test); worker package 71; 28 check runners + worker startup checks; provider native suite 4,557 tests with only master's five environment failures.

Hardware (M3 Ultra rank 0 + M5 Max rank 1 over Thunderbolt 5 RDMA, indicative figures): all nine families pass on this head (tokens equal to the single-Mac reference or first difference at an exact tie). MTP default: 9B decode 115-124 tok/s (acceptance 0.73-0.79), 27B 44.7-50 tok/s (pipeline without MTP 26). Prefix reuse with MTP: repeat first token 9B 2.35 → 0.093 s, 27B 8.60 → 0.166 s (5,120 tokens), 9B 20,480 tokens 10.5 → 0.070 s. 9B 20,480-token request with MTP passes. Faults: a rank killed mid-decode releases memory on both Macs with no SIGKILL; guard v2 dead peer exits 75 in ~30 s.

Known limits: the 27B 20,480-token request was refused by the M5 Max's per-request memory gate with ~43 GB of desktop applications resident there (the rank had admitted a 32K context at load); a refused request currently ends the pair session (being fixed); on a prefix-cache hit MTP acceptance drops; not yet run on hardware: provider-served pair sessions and coordinator-driven serving.

Update 2026-10-10: every model family across both Macs, placement, clean hang-up, link setup v2, no orphaned memory

This branch now carries ten merged work branches (128 signed commits on 807ac8ea9, 345 files, provider and runtime only; no coordinator code):

• Model families on the pair: Qwen3.5 35B A3B and Qwen3.6 35B A3B (routed-expert adapter, per-adapter protocol), Ternary Bonsai 2 27B (Prism Hadamard adapter), GPT-OSS 20B, Nemotron 3.5 Lightning (Nemotron-H adapter; moves libs/mlx-swift-lm to 9e901a0, the stage seam in draft Layr-Labs/mlx-swift-lm 301), MiMo V2.6 Flash (one resident catalog), Gemma 4 26B — in addition to the registered 9B and 27B.
• Placement: device profiles, a planner that places stages from each Mac's detected memory and speed, darkbloom cluster plan.
• Installed path: a client hang-up ends the request with a clean stop and the session keeps serving; a lost rank gives the client an error frame (about 120 ms on hardware).
• Link setup v2: the cluster port moves out of the Thunderbolt Bridge onto its own service with a fixed address and no route, under one approval (built and dry-run; not applied on the test pair).
• No orphaned memory: one forced-exit path releases MLX's wired limit and caches first, with a dead-man timer; an orphan-wired guard at worker start; kill grace sized from planned bytes.

Tests: cluster library 196/197 (one environment-dependent sampler test); worker package 66/66; 27 check runners; provider native suite 4,558 tests with only master's five environment failures.

Hardware smoke on this head's products (M3 Ultra rank 0 + M5 Max rank 1 over Thunderbolt 5 RDMA; one cold request each, indicative; pass = exact, tokens equal with logits differing, or first difference at a near tie):

Family	Cut	Prompt / outputs	Verdict vs single-Mac reference	First token	Decode
Qwen3.5 9B	4	8,192 / 128	near tie at 47 (exact tie)	2.66 s	65.7 tok/s
Qwen3.8 27B	16	1,024 / 128	tokens equal	1.41 s	27.5 tok/s
Qwen3.5 35B A3B	12	4,096 / 64	near tie at 10 (exact tie)	1.01 s	71.1 tok/s
Qwen3.6 35B A3B	12	4,096 / 64	tokens equal	1.01 s	69.9 tok/s
Ternary Bonsai 2 27B	24	8,192 / 128	tokens equal	11.85 s	31.9 tok/s
Nemotron 3.5 Lightning	25	34 / 64	tokens equal	0.17 s	81.0 tok/s
GPT-OSS 20B	8	8,192 / 128	tokens equal	1.87 s	72.8 tok/s
Gemma 4 26B QAT 4-bit	18	31 / 64	tokens equal	0.14 s	75.8 tok/s
MiMo V2.6 Flash (~156 GB)	38, chosen by the planner	27 / 37	near tie at 28 (exact tie)	1.33 s	27.7 tok/s

Every run: both ranks exited cleanly, no worker process left, wired memory back on both Macs. Fault on this build (9B, rank 1 SIGTERM mid-decode): the signalled rank released and exited in 0.07 s, the survivor ended itself through the release path, wired memory back on both Macs within 1.1 s, no SIGKILL.

Not in this head yet: long prompts to 32K (proven separately on the 9B and 27B at 5,120 to 30,720 tokens; to be merged), the MLX pin move to the guarded JACCL head with the NAX MoE simdgroup-skip revert (draft Layr-Labs/mlx 33 fdae3174, mlx-swift 54 64070c4), MTP on the cluster path, coordinator-driven serving.

Note: draft d-inference PR 1426 is stacked on this branch; it needs a plain merge of the new head.

Linked issue

None.

Before and after

Before — the extracted branch could not start a collective, could not finish a distributed start, and gave the operator no way to see why.

flowchart LR
    classDef bad fill:#fde2e1,stroke:#b42318,color:#3b0d0a
    classDef ok fill:#e3f1e5,stroke:#1e7a34,color:#0b2e14
    classDef neutral fill:#eef1f5,stroke:#5b6676,color:#1a2230

    OP["Operator"]:::neutral --> START["start --local --distributed"]:::neutral
    START --> REG["Register as cluster member"]:::ok
    REG --> ACK["Coordinator never acknowledges"]:::bad
    ACK --> KILL["Leader tears down after 10 s"]:::bad
    START --> WRK["Launch rank worker"]:::neutral
    WRK --> NOSRC["Worker has no source in the tree"]:::bad
    WRK --> STUB["macOS 14 target compiles the JACCL stub"]:::bad
    OP --> LINK["Thunderbolt port without its own IPv4 address"]:::neutral
    LINK --> SILENT["Nothing reports it; JACCL refuses at init"]:::bad

Loading

After — a 26.2 worker package builds real JACCL, the coordinator (its side is in darkbloom-platform) acknowledges and guards members, and the link is inspected before anything is launched.

flowchart LR
    classDef ok fill:#e3f1e5,stroke:#1e7a34,color:#0b2e14
    classDef neutral fill:#eef1f5,stroke:#5b6676,color:#1a2230
    classDef pending fill:#fff3d6,stroke:#9a6700,color:#3d2a00

    OP["Operator"]:::neutral --> CL["darkbloom cluster link"]:::ok
    CL --> VERDICT["Per-port verdict and one sentence of guidance"]:::ok
    OP --> STAGE["darkbloom-cluster-stage-check"]:::ok
    STAGE --> LOADED["Each Mac loads and releases its stage of the real model"]:::ok
    OP --> START["start --local --distributed"]:::neutral
    START --> REG["Register as cluster member"]:::ok
    REG --> ACK["cluster_member_accepted"]:::ok
    START --> WRK["darkbloom-cluster-worker, macOS 26.2 build"]:::ok
    WRK --> JACCL["Real JACCL group"]:::ok
    JACCL --> XPORT["Collectives over RDMA: exact on two Macs"]:::ok
    XPORT --> RUN["Two-rank generation of the real model"]:::ok
    RUN --> SERVE["Served by start --distributed and the coordinator"]:::pending
    OP --> SETUP["darkbloom cluster: detect, one approval, ready"]:::ok

Loading

Green is exercised by a test or a run recorded below; amber is built but not yet run on two Macs.

What changed

Native worker package (libs/darkbloom-cluster-worker, new). The provider and the shared cluster modules build for macOS 14, where the pinned mlx-swift compiles the JACCL stub. A sibling package with a macOS 26.2 minimum builds the real backend with no submodule pin change. build-native-worker.sh refuses a binary that holds the stub or whose Mach-O minimum is not exactly 26.2. Products:

• darkbloom-cluster-worker — the rank worker the installed plan launches (from Preserve shared distributed runtime and provider integration #1226).

• darkbloom-cluster-collective-check — both ranks run it; verifies reductions, point-to-point transfers and ordering byte for byte. No model.

• darkbloom-cluster-stage-check — one Mac, one rank, the real artifact, no collective: verified stage load, release, and memory before and after.

• darkbloom-cluster-reference, darkbloom-cluster-pair-check — a single-Mac staged reference, a two-Mac driver (rank 0 local, rank 1 over SSH, never signals a worker) and a comparator with a closed set of verdicts (exact, tokensEqualLogitsDiffer, divergedAtNearTie, diverged, incomparable).

A pair could not decode before this PR: rank 0 handed over its residual while Metal completion handlers still held the buffer, and the sender refuses storage it does not own alone. One GPU stream synchronization on rank 0's path fixes it (fix(cluster): drain the GPU stream before rank 0 hands over its residual). That change is on the serving path and wants a careful read.

A rank's last resort against a native call that never returns is now a deadline thread (ProcessDeadline), not only alarm: on real hardware a rank spinning in the RDMA completion poll ran more than nine minutes past its alarm.

Coordinator. Moved to Layr-Labs/darkbloom-platform (draft pull request 6 there): the member acknowledgement, the pair lifecycle and quarantine release, pair formation, routing an owner's requests to its pair's leader, and the owner views. Nothing under coordinator/ differs from master here.

Provider

• darkbloom cluster (no subcommand): guided link setup. It checks RDMA, waits for a Thunderbolt connection, and if the connected port has no IPv4 address of its own it gives it a link-local one through one macOS approval prompt and installs a small root launchd job (one file under /Library/LaunchDaemons, two system tools, no installed program) that re-adds the address whenever the port has none; then verifies. A one-time address proved not durable on a Mac whose port is a bridge member: it was gone about two hours later with no restart or replug. --temporary keeps the address-only path, --dry-run prints the exact commands and changes nothing, and cluster link --remove removes the job, its file and the address. cluster link --fix, --remove and --watch are the pieces. Darkbloom never calls sudo. The root job has not run behind a real approval yet.
• Restored from Preserve shared distributed runtime and provider integration #1226 (dropped by the extraction): the streamed-completion terminal, the first-token budget on distributed submits, refusal before response headers, member-role registration, member-mode guards on load, prefetch, inference and autopilot, and the --cluster-member start mode; 21 research test files ported.
• Member side of coordinator-formed pairs: cluster_membership in the member registration, the pair approval saved by cluster configure, the member control installed before connecting and attached at acceptance. This build declines every pair preparation on purpose: a coordinator-formed pair needs an owner mode, a worker bootstrap and a rank-1 request channel that are not in the tree, so a member answers 503 instead of half-serving.
• darkbloom cluster console (and bare darkbloom cluster on a terminal that can draw): an interactive screen over the operations above. Readiness comes from the link probe and doctor, pairing and trust from the saved setup with an explicit approval that counts only once the whole question has been on screen for 500 ms, the admitted models from the runtime's own list; it can fix the link, start and stop the local session, recover a stranded journal and write a redacted diagnostic export. handoff/TUI-wiring.md lists 53 screen items: 31 wired to a real operation, 10 whose operation exists but is not connected, 12 with no operation yet, and a check fails if the screen shows anything outside the wired set. --plain and --json print the same state without a terminal.
• darkbloom cluster link [--json], also run by cluster doctor: read-only inspection of RDMA state, port state, interface address, bridge membership and the IPv4-mapped GID, with a closed set of verdicts. No address, MAC or GID is kept from tool output.
• Write provider.toml owner-only and atomically; a file first created by an ordinary save was 0644 and the cluster readers refused it.
• Installed path (start --local --distributed):
  • It could not start a worker at all: the owner always passed the owner-authenticated bootstrap arguments and the worker refuses them. One place now selects the bootstrap; this build selects the runtime's own exchange on the configured link address, and cluster status and cluster doctor say that nothing vouches for the peer that answers.
  • A fence closes the worker's command stream and nothing else. Before, the owner sent SIGTERM at the fence and SIGKILL two seconds later, to a worker that still held its model. The owner now signals only a process that outlived its own hard deadline.
  • The owner clears its own device journal once it has seen its child exit. Before, any failed launch left a journal that refused every later start. darkbloom cluster recover clears a journal whose owner died, and refuses while any process still holds the device or carries the recorded epoch.
  • The worker takes an optional startup deadline and ends itself when it is not ready by then.
  • The plan sets a JACCL progress limit for every rank, and the leader, the owner and cluster doctor refuse a worker binary that does not carry the progress guard.
  • The owner refuses to start a worker while this Mac's ordinary provider holds the instance lock.
  • The plan selects the generation mode beside the prefill schedule: the pipeline by default, so a saved setup behaves as before and its bytes do not change; a mode that a worker's capability record does not advertise is refused before launch. Startup, first-token and shutdown budgets come from the selected registered model, with the 9B's unchanged. The 27B stays closed to a distributed start, and the refusal says it is a policy decision.

Runtime

• A second registered model. The resident adapter selects its model from a closed catalog (QwenResidentModelDefinition), and admission, byte ceilings, capability and the worker's model and cut validation are per registered model. The registered Qwen3.8 27B is admitted with every structural cut; the 9B's receipts, capability bytes and tests are unchanged.

• Phase split, a second generation mode beside the pipeline, for both registered models: the pair prefills with lookahead, rank 0 hands its committed request state to rank 1 after the first token (one checked transfer per state tensor, verified by rank 1 before it adopts, a 20 s limit bound into the agreement), and rank 1, which holds the whole model, decodes alone and relays tokens to rank 0 in batches of 1, 2, 4, 8 then 16: 41 transfers after the first token for 128 outputs against 1,397 in the pipeline. The worker takes --generation-mode and advertises the modes it supports; the pipeline stays the default and its agreement bytes are unchanged and pinned by a test. A correctness-only local-socket transport lets two ranks run on one Mac for tests; it and the fault-injection switch are refused unless a worker is started with --qualification-switches yes.

• Benchmark tooling under scripts/benchmarks/cluster/ (a streamed load generator with a self-test, an isolated serve harness, artifact verification against manifests) and the night's report, docs/reports/2026-10-09-two-mac-baseline-and-model-matrix.md: the single-Mac "before" through the product's serving path on both Macs, and the state of every catalog model on each.

• Six of twelve slices of delivering a follower's stage over the link (design and measurements in handoff/DESIGN-stage-transfer.md): a pinned per-tensor content inventory with a generator and an independent Python cross-check, the pin for the registered 9B, the transfer plan and state machines, a loader seam that separates stage metadata from where the bytes come from (all 14 existing load receipts unchanged), stage metadata derived from the pin with no weight file, and the MLX-side intake with a stage-check transfer mode. In one process on the M3 Ultra a follower's stage (3.98 GB) is received and verified in 1.65 s against 4.72 s for a local load of the same stage, and corrupted, swapped and truncated inputs are refused; the follower is ready about when a local load would be, so the transfer buys a follower with no weights on disk, not time. Nothing has crossed the link or run between two processes, and the load path does not use it yet.

• Host gate: swap left from earlier is admitted while memory pressure is normal and refused under warning pressure. The previous rule refused any Mac with swap in use. Free-page requirements are unchanged.

• Resident admission tests: a real positive path on both ranks from the registered metadata, and one property per refusal. The previous suite could not fail for the reason each case named.

Test plan

Run on an M3 Ultra (256 GB, macOS 27.2) and an M5 Max (128 GB, macOS 27.0), Xcode 27.0, one Thunderbolt 5 cable.

• The focused cluster checks (17 runners, including ClusterLinkChecks with 1,353 expectations)
• swift test --package-path libs/darkbloom-cluster — 22 tests in 5 suites; DarkbloomClusterWorkerTests built for 26.2 — 9 tests
• make provider-test after merging master — the same 75 tests fail as on unmodified master on this machine (same 148 issues, same 56 XCTest errors; mostly a missing pagedattention.metal test resource and MiMo fixtures). The branch adds 142 tests to those targets and none of them fail
• darkbloom-cluster-stage-check on the registered Qwen3.5 9B artifact: every supported cut and both ranks on the M3 Ultra, cut 4 and both ranks on the M5 Max. Each loads in about 3 seconds, releases to a few KB active and 0 cached, and both Macs report the same storage commitment
• darkbloom cluster on both Macs: one reported ready; on the other the approval prompt was accepted, the address added and verified
• darkbloom-cluster-collective-check across the two Macs: 5.36 GiB of reductions and transfers, 0 mismatches on both ranks, IP byte counters at 0.0003% of the payload
• Dead peer on the real link: with the pinned JACCL the surviving rank spins until the deadline thread ends it; with the fork's progress guard it reports the error and exits about 4 seconds after its peer. No process or wired memory left on either Mac
• The real model across both Macs (rank 0 M3 Ultra, rank 1 M5 Max, fork JACCL with the progress guard): nine requests, 30 to 8,192 prompt tokens, cuts 4 and 16, serial and lookahead prefill, all complete with clean shutdown
  • Against each Mac alone: tokens equal for short prompts; for long prompts the first difference, where there is one, is at a tie or within 4 ulp between the top two logits. Not bit-exact across chips, and never a plain divergence. The two Macs alone differ from each other the same way; one Mac is exact across cuts; the pair is exact run to run
  • Prefill at 8,192 tokens: 971 tok/s (M3 Ultra alone), 2,788 (M5 Max alone), 2,160 (pair), 3,069 (pair, lookahead). Decode: 60–65, 77–81, 59–63. Pair figures include control and transport; single-Mac figures are summed frame times
  • Rank 1 ended with SIGTERM mid-decode with both stages loaded: rank 0 reported the JACCL error at its 60 s limit and exited by itself; no process on either Mac; wired memory back at its earlier level; the next run exact
• At 63fd73706 (after the 27B and stage-transfer commits): cluster library 37 tests in 8 suites; worker package 48 XCTest cases; all 18 check runners (11 library, 3 worker package including StageTransferChecks, 4 provider; the link runner has 2,072 expectations in 32 groups); the provider builds
• Registered Qwen3.8 27B, each Mac alone: artifact verified (14 of 14 files) on both; stage load and release for cuts 12, 16 and 20 on both ranks and both Macs with equal storage commitments; single-Mac references at 30, 4,096 and 8,192 prompt tokens deterministic on each Mac and exact across cuts; between the two Macs every token equal, logits differ. Prefill about 300 tok/s (M3 Ultra) and 770–880 (M5 Max), decode 27–29 on both. No pair run: the second Mac's port had no address
• darkbloom cluster --dry-run on both Macs with the new build: the M3 Ultra needs no prompt; the M5 Max prints the fourteen commands the approval would run and changes nothing. The job definition, built by the same command chain, was loaded as a harmless stand-in in a user launchd domain: runs at load and every 10 s, survives six immediate reinstalls, unloads cleanly
• The full provider unit suite at 4e5f70793, built with --build-system native as the release tooling builds it: 4,539 tests in 621 suites, 5 failing, the same five that fail on unmodified master on this machine (three MiMo tests that need CI's fixtures, two that depend on the home directory). With the toolchain's default build system 75 fail on master too, because nested resource bundles are not found
• Library tests 49 in 9 suites, worker package 48 XCTest cases and all 18 check runners at the same head
• At 657a22d8f (phase split and console): library tests 72 in 10 suites; worker package tests (16 and 41); 19 self-contained check runners, the console's with 11,689 expectations in 23 groups on pseudo-terminals; the full provider suite, native build system, 4,544 tests in 622 suites, failing only the five environment tests that fail on master
• Phase split on each Mac separately, both models: 26 of 26 comparisons against that Mac's reference exact (9B at cut 8, 27B at cut 16; two real workers over the local test socket, and the in-process hand-off); a flipped bit, a stalled rank and a rank ended on either side each end both ranks by themselves with nothing left allocated; hand-off 26–71 ms for the 9B and about 125 ms for the 27B's 173 MB at 8,192 tokens
• The console on one Mac: a 23-screen dry-run tour in a pseudo-terminal; q, Ctrl-C, SIGTERM and SIGHUP restore the terminal; a second instance is refused; fifteen existing read-only commands print the same output as before
• At 895aece69 and the test-only 12250136a (memory gate second version, stage-transfer slices 3, 4 and 6, installed-path mode selection): library tests 102 in 14 suites; worker package tests; 20 check runners (the worker startup checks 15 of 15 against a built worker); the full provider suite, native build system, 4,548 tests in 623 suites, failing only the five environment tests that fail on master
• Phase split across the two Macs; a real start, client request and stop from the console
• One HTTP completion through darkbloom start --local --distributed on the two Macs, then the lifecycle faults
• The same through the coordinator with two real members (coordinator side: darkbloom-platform draft 6)
• The 27B across the two Macs; the durable link job behind a real approval

make docs-check reports one broken link, in docs/reports/2026-10-03-all-model-prefix-qualification.md; it is on master (from #1326) and unchanged here. make docs-impact-check passes.

Components touched

• coordinator (Go) — moved to Layr-Labs/darkbloom-platform
• provider (Rust, legacy)
• provider-swift (Swift CLI)
• console-ui (Next.js)
• enclave (Swift)
• infra / CI / release
• docs

Protocol / interface changes

• No protocol/interface changes

• Yes — described above and matching side updated

• The provider registers cluster_membership in the member role and expects cluster_member_accepted; the coordinator side of both is in darkbloom-platform draft 6. A coordinator without it ignores the fields and never sends the acknowledgement, and the provider's negotiation timer then ends a distributed start.

• New CLI: darkbloom cluster (guided setup), darkbloom cluster link, darkbloom cluster recover. cluster doctor now also fails when the local link is not ready or the worker binary has no progress guard. cluster status reports the bootstrap selection and the progress limit.

• New optional worker argument --startup-deadline-uptime-nanoseconds; exit status 123 when it expires. New optional worker arguments --generation-mode and --qualification-switches; supportedGenerationModes in the capability record, omitted for a pipeline-only runtime so existing records keep their bytes. An owner and a worker must come from one tree: an owner without this change refuses a new worker's capability record.

• New CLI: darkbloom cluster console [--json|--plain] [--dry-run] [--temporary].

• New decision-receipt policy identifier registered_dense_selected_stage_load_resources_v2.

• New runtime model ID registered_qwen38_27b; the worker validates --model-id and --stage-cut against the closed catalog. New optional field contentInventorySHA256 on a registered specification, in no existing fingerprint.

• darkbloom cluster gains --dry-run and --temporary; cluster link reports gain addressKept, and guidance can be non-null while the state is ready.

• No release artifact or version change.

Documentation impact

• Canonical documentation updated
• No documentation needed — reason:

docs/reference/cluster-control-protocol.md, protocol-messages.md, configuration.md, docs/provider/installation.md (uninstall removes the link job), and the two package READMEs. Coordinator-only pages are back at master's text; citations of coordinator files that moved are marked as paths in the platform repository. make docs-impact-check passes; make docs-check reports only the broken link that is on master.

Not shown by this PR

• The provider's own distributed start has not served a request on the two Macs. The two defects that stopped it (bootstrap flags the worker refuses, workers ended on two-second timers) are fixed and covered by checks, but the end-to-end run is still to do. The model run above used the qualification driver.
• Against the product's own serving path the pair is not faster end to end for this model. Qwen3.5 9B, 8,192-token prompt, 128 outputs: first token 8.59 s on the M3 Ultra, 2.97 s (rested) to 3.78 s (settled) on the M5 Max, 2.67 s on the pair in the earlier runs; decode 77.9, 101.1 / 97.8 and 61.6 tok/s; total 10.32 s, 4.33 / 5.15 s and 4.75 s. The pair reaches the first token sooner on long prompts, is slower at 1,024 tokens, and decodes at half to four fifths of either Mac's product path, because the cluster runtime's decode is slower than the serving engine's even on one Mac and has no MTP. A phase split (pair prefill, then one rank decodes alone) is in this pull request and proven on each Mac separately; it has not run across the two Macs. A design for handing the pair's prefill to the serving engine is in handoff/DESIGN-pair-prefill-engine-decode.md; by its arithmetic the gain for this model is a few percent at the one measured pair configuration. A layer pipeline helps prefill and models that do not fit one Mac; tensor parallel is not in this runtime.
• When one rank fails the other waits the whole progress limit before failing, because no cancel is sent to it.
• The pinned JACCL has no progress bound and sends stale bytes after a partial payload. The fixes are drafts on the forks: JACCL: zeroed send-frame tails, a progress guard, the advertised GID and the upstream JACCL fixes mlx#33 (with the upstream JACCL fixes) and the pin in build: pin mlx and mlx-c to the JACCL fixes and the owned bootstrap bridge mlx-swift#54. This PR does not move libs/mlx or libs/mlx-swift. Until it does, the guarded JACCL exists only in builds that check out the mlx branch by hand, and the installed path refuses to serve without it.
• The data plane between ranks is plaintext; the authenticated record transport has no caller yet.
• Two registered models are admitted by the runtime, the Qwen3.5 9B and the Qwen3.8 27B. Only the 9B has run across two Macs. The provider still gates the 27B model IDs on M5 plus NAX and a member's capability set never contains NAX, so a distributed start refuses the 27B on any Mac; serving it on a mixed pair needs a decision, not a workaround. A session is bounded to 16 requests and 300 seconds.
• The host memory gate changed; this is its second version, after an independent review of the first. The old gate admitted on truly free pages only and refused a Mac whose memory is mostly file cache, the normal state after a download or a hash pass. The rule here counts file cache above the kernel's file-cache reserve while pressure is normal and the cache is inactive, and stops a load or request at its next check when the Mac's compressed or swapped-out pages have grown since its first sample (handoff/DESIGN-resource-gate-v3.md). The review's concern about the first version was confirmed on hardware: with the cache held mapped and read by another process, a load on the M5 Max compressed 18.4 GiB while the pressure level still read normal; that is what the growth check is for. Run on both Macs for both models: the after-download state, cache made of the model being loaded, cache held by another process, two loads started together, and held loads that must still be refused. Not done: each of its commits built and tested on its own, and the comparison of the final binaries with the ones the hardware runs used. Small Macs are still refused from the after-download state (a 32 GB Mac never admits a 10 GiB stage). This is a resource-admission policy change and wants a decision before it is relied on anywhere else.
• A coordinator-routed request does not reach a real pair yet. The coordinator side (darkbloom-platform draft 6) forms, routes and hands off, tested with a fake member, and the provider member here declines every preparation: it still needs an owner mode for a coordinator-chosen epoch, the worker bootstrap attachment (which needs the mlx-c bridge of Preserve the owned JACCL bootstrap callback bridge mlx-c#14), a rank-1 request channel and the serving seam. Only the owning account's requests are routed; billing and usage rows for a pair are not designed.
• The durable link address is built but unproven: nothing behind the macOS approval has run, including the root job, its Login Items entry, how fast the address and its GID return after a strip, and whether a running session survives the gap. What removed the one-time address is narrowed (no sleep, restart or cable event; the port is a member of a bridge that Internet Sharing runs over) but not proven.

Notes for reviewers

• Refactor pass. The link work had its own refactor pass and a second read-only review. The coordinator half had both before it moved. The library changes for the 27B and the stage-transfer slices have had correctness reviews by their authors' reviewers but not yet the dedicated refactor pass this repository requires; it will run over the library once the branches still in flight (phase split, memory gate) have landed, and this pull request stays a draft until then.

• A wait that could hang. Two provider unit tests in the suite "Owned native member invocation" failed intermittently in CI (12 of 30 isolated runs locally). The cause was a product defect, not the tests: Process.waitUntilExit() for a process started from a dispatch thread returned late and was sometimes never reported, so a member never learned its owner had exited, stayed quarantined and never released its pair. ClusterProcessExit now takes the exit from the termination handler and latches it, at the remote endpoint, the worker process pump and the capability probe; no bound was loosened and no signal added or moved. A check runs 300 sessions from a dispatch thread and fails at about session 47 on the old files. After: 0 of 30 isolated suite runs. Worth a reviewer's eye: the one SIGKILL that can reach a worker is in the worker process pump, after the worker's own hard deadline, SIGTERM and a further margin (the documented exception in ClusterWorkerSignalPolicy); and the endpoint can SIGKILL an owner past its ceiling, which would leave a still-live worker with a closed command stream and nobody to clear the journal.

• AGENT-HANDOFF.md and handoff/ are working documents (status by gate, gap map, evidence ledger, runbook). They are here so the state is reviewable; drop them before merge if they do not belong in the tree.

• The swap-gate change is a resource-admission policy change. The previous rule came from the research measurements; I found no measurement behind it, but it wants a second opinion.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
d-inference	Ready	Preview	Oct 12, 2026 2:21pm UTC
d-inference-console-ui-dev	Ready	Preview	Oct 12, 2026 2:21pm UTC
d-inference-landing	Ready	Preview	Oct 12, 2026 2:21pm UTC
darkbloom-status	Ready	Preview	Oct 12, 2026 2:21pm UTC
eigen-homepages-darkbloom	Ready	Preview	Oct 12, 2026 2:21pm UTC