Skip to content

fix(provider): inline MTP inspection resolves HF-cache symlinks and rejects loudly - #618

Merged
Gajesh2007 merged 2 commits into
masterfrom
fix/specdec-symlink-artifacts
Aug 13, 2026
Merged

Gajesh2007 merged 2 commits into
masterfrom
fix/specdec-symlink-artifacts

Conversation

@Gajesh2007

@Gajesh2007 Gajesh2007 commented Aug 13, 2026 •

Copy link
Copy Markdown
Member

Problem

Bug 1 from Layr-Labs/mlx-swift-lm#106's Discovered bugs: SpecDecStore.inspectInlineArtifact required regularFileSize — which deliberately rejects symlinks — on the checkpoint directory, config.json, model.safetensors.index.json, and every selected shard. HF-cache snapshots materialize all of those as symlinks into blobs/, so on the layout production checkpoints actually arrive in, inline MTP was rejected with status inline_artifact_invalid and zero log output: the drafter silently never activates, the slot looks healthy, and mtp_enabled simply reads false.

This is not hypothetical: it silently poisoned a paired Qwen3.6 measurement session on 2026-08-13 (MTP-on runs were secretly MTP-off; the session was salvaged with APFS-cloned snapshots). It is also the only reason production dodged the 0.70x serial-verify regression fixed by mlx-swift-lm#106.

Change

  1. Symlinks are resolved, not rejected — inline inspection only. Directory, config, index, and shard checks resolve symlinks (URL.resolvingSymlinksInPath) and validate the resolved target (exists, regular file, size bounds). A broken link keeps its unresolvable component, fails the attribute read, and is rejected by name. Operator-provided local overrides (inspectLocalArtifact) still reject symlinks: there the directory itself is untrusted input; the inline directory is the already-verified target checkpoint.
  2. Fail-loud — inspectInlineArtifact now returns Result<SpecDecArtifact, InlineArtifactRejection> where every rejection carries the concrete reason and file path, and is also logged at warning level (darkbloom.SpecDecStore). Both inline_artifact_invalid producers (funnel admission and the pre-load revalidateForLoad gate) now surface the reason; no status-only rejection remains.

Behavior

flowchart LR
  subgraph Before
    A1["HF-cache snapshot<br/>(shards = symlinks into blobs/)"] --> B1["regularFileSize() rejects symlink"] --> C1["status inline_artifact_invalid<br/>ZERO log output<br/>MTP silently inert; measurements poisoned"]
  end
  subgraph After
    A2["HF-cache snapshot<br/>(shards = symlinks into blobs/)"] --> B2["resolve link -> validate target<br/>(exists, regular, size bounds)"] --> C2["valid: MTP activates<br/>broken/invalid: rejection names reason + path,<br/>logged at warning"]
  end
Loading

Code

flowchart LR
  subgraph Before
    D1[SpecDecArtifactFunnel.prepare] -->|"inspectInlineArtifact -> SpecDecArtifact?"| E1["mega-guard, nil on ANY failure"]
    F1[SpecDecStore.revalidateForLoad] -->|same nil| E1
    E1 --> G1["disabled(.inlineArtifactInvalid) — reason lost"]
  end
  subgraph After
    D2[SpecDecArtifactFunnel.prepare] -->|"inspectInlineArtifact -> Result"| E2["per-check rejections via rejectInline()<br/>reason + path, logged (darkbloom.SpecDecStore)"]
    F2[SpecDecStore.revalidateForLoad] -->|"failure detail -> fallback(detail:)"| E2
    E2 --> H2["resolvedRegularFileSize / isDirectoryResolvingSymlinks<br/>(inline path only; local overrides stay symlink-rejecting)"]
  end
Loading

Tests

SpecDecArtifactFunnelTests — 17 tests green (swift test --filter SpecDecArtifactFunnelTests), plus 81 tests across the 7 adjacent SpecDec/MTP suites (SpecDec|ProviderMTPFactory|MTPPostureTelemetry):

  • HF-cache symlinked snapshot passes inspection and admits inline MTP — builds a real blobs/ + snapshots/<rev>/ layout with relative symlinks, then proves the full activation chain short of weight loading: direct inspection succeeds with facts byte-identical to the regular-file baseline (revision, payload bytes, config/index digests), the funnel admits it as .candidate (the status the slot factory consumes), and revalidateForLoad — the last gate before assistant construction in EngineV2SlotFactory+MTP — resolves.
  • Broken symlinked shard is rejected with the concrete reason and path — the Result failure names broken symlink and the shard path; the funnel maps it to the existing fail-open inline_artifact_invalid status.
  • Regular-file inline inspection facts and revalidation are unchanged, and a checkpoint without inline tensors reports no tensors with prefix instead of a bare nil.

Warning logs observed in the test run, e.g.:

warning darkbloom.SpecDecStore: inline MTP artifact rejected: inline shard is missing, not a regular file, or a broken symlink: …/snapshots/0123abcd/model-00001-of-00001.safetensors

Bug 2 from the same inventory (MTP acceptance unobservable outside Datadog Logs) is fixed separately in #619.

…ejects loudly

SpecDecStore.inspectInlineArtifact required regularFileSize on the checkpoint
directory, config.json, index, and every selected shard — but HF-cache
snapshots materialize all of those as symlinks into blobs/, so production
inline MTP was silently rejected with status inline_artifact_invalid and
zero log output (mlx-swift-lm#106 'Discovered bugs' #1; poisoned a paired
measurement session on 2026-08-13, worked around with APFS clones).

- Inline inspection now RESOLVES symlinks and validates the resolved target
  (exists, regular file, size bounds). Operator-provided local overrides
  (inspectLocalArtifact) still reject symlinks: there the directory itself
  is untrusted input.
- inspectInlineArtifact returns Result with a concrete reason + file path;
  every rejection is also logged (darkbloom.SpecDecStore, warning). No
  status-only rejections remain, including the revalidateForLoad gate.
- Tests: HF-style symlinked snapshot (blobs/ + relative links) passes
  inspection, funnel admission, and pre-load revalidation; broken symlink
  fails with the named shard path; regular-file facts unchanged.
@vercel

vercel Bot commented Aug 13, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
d-inference Ready Ready Preview Aug 13, 2026 11:30pm
d-inference-console-ui-dev Ready Ready Preview Aug 13, 2026 11:30pm
d-inference-landing Ready Ready Preview Aug 13, 2026 11:30pm

Request Review

This branch had an error being deployed

1 failed and 3 active deployments
Preview – d-inference-console-ui-dev — adcc6573 Deployed Aug 13, 2026 by vercel[bot]
Preview – d-inference — adcc6573 Deployed Aug 13, 2026 by vercel[bot]
Preview – d-inference-landing — adcc6573 Deployed Aug 13, 2026 by vercel[bot]
benchmarks — adcc6573 Deployed Aug 13, 2026 by Gajesh2007 via E2E Benchmarks #67
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant