Repository navigation
fix(provider): inline MTP inspection resolves HF-cache symlinks and rejects loudly - #618
Merged
Merged
Conversation
…ejects loudly SpecDecStore.inspectInlineArtifact required regularFileSize on the checkpoint directory, config.json, index, and every selected shard — but HF-cache snapshots materialize all of those as symlinks into blobs/, so production inline MTP was silently rejected with status inline_artifact_invalid and zero log output (mlx-swift-lm#106 'Discovered bugs' #1; poisoned a paired measurement session on 2026-08-13, worked around with APFS clones). - Inline inspection now RESOLVES symlinks and validates the resolved target (exists, regular file, size bounds). Operator-provided local overrides (inspectLocalArtifact) still reject symlinks: there the directory itself is untrusted input. - inspectInlineArtifact returns Result with a concrete reason + file path; every rejection is also logged (darkbloom.SpecDecStore, warning). No status-only rejections remain, including the revalidateForLoad gate. - Tests: HF-style symlinked snapshot (blobs/ + relative links) passes inspection, funnel admission, and pre-load revalidation; broken symlink fails with the named shard path; regular-file facts unchanged.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Gajesh2007
had a problem deploying
to
benchmarks
August 13, 2026 22:51 — with
GitHub Actions
Failure
Gajesh2007
had a problem deploying
to
benchmarks
August 13, 2026 23:30 — with
GitHub Actions
Failure
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Bug 1 from Layr-Labs/mlx-swift-lm#106's Discovered bugs:
SpecDecStore.inspectInlineArtifactrequiredregularFileSize— which deliberately rejects symlinks — on the checkpoint directory,config.json,model.safetensors.index.json, and every selected shard. HF-cache snapshots materialize all of those as symlinks intoblobs/, so on the layout production checkpoints actually arrive in, inline MTP was rejected with statusinline_artifact_invalidand zero log output: the drafter silently never activates, the slot looks healthy, andmtp_enabledsimply reads false.This is not hypothetical: it silently poisoned a paired Qwen3.6 measurement session on 2026-08-13 (MTP-on runs were secretly MTP-off; the session was salvaged with APFS-cloned snapshots). It is also the only reason production dodged the 0.70x serial-verify regression fixed by mlx-swift-lm#106.
Change
URL.resolvingSymlinksInPath) and validate the resolved target (exists, regular file, size bounds). A broken link keeps its unresolvable component, fails the attribute read, and is rejected by name. Operator-provided local overrides (inspectLocalArtifact) still reject symlinks: there the directory itself is untrusted input; the inline directory is the already-verified target checkpoint.inspectInlineArtifactnow returnsResult<SpecDecArtifact, InlineArtifactRejection>where every rejection carries the concrete reason and file path, and is also logged at warning level (darkbloom.SpecDecStore). Bothinline_artifact_invalidproducers (funnel admission and the pre-loadrevalidateForLoadgate) now surface the reason; no status-only rejection remains.Behavior
flowchart LR subgraph Before A1["HF-cache snapshot<br/>(shards = symlinks into blobs/)"] --> B1["regularFileSize() rejects symlink"] --> C1["status inline_artifact_invalid<br/>ZERO log output<br/>MTP silently inert; measurements poisoned"] end subgraph After A2["HF-cache snapshot<br/>(shards = symlinks into blobs/)"] --> B2["resolve link -> validate target<br/>(exists, regular, size bounds)"] --> C2["valid: MTP activates<br/>broken/invalid: rejection names reason + path,<br/>logged at warning"] endCode
flowchart LR subgraph Before D1[SpecDecArtifactFunnel.prepare] -->|"inspectInlineArtifact -> SpecDecArtifact?"| E1["mega-guard, nil on ANY failure"] F1[SpecDecStore.revalidateForLoad] -->|same nil| E1 E1 --> G1["disabled(.inlineArtifactInvalid) — reason lost"] end subgraph After D2[SpecDecArtifactFunnel.prepare] -->|"inspectInlineArtifact -> Result"| E2["per-check rejections via rejectInline()<br/>reason + path, logged (darkbloom.SpecDecStore)"] F2[SpecDecStore.revalidateForLoad] -->|"failure detail -> fallback(detail:)"| E2 E2 --> H2["resolvedRegularFileSize / isDirectoryResolvingSymlinks<br/>(inline path only; local overrides stay symlink-rejecting)"] endTests
SpecDecArtifactFunnelTests— 17 tests green (swift test --filter SpecDecArtifactFunnelTests), plus 81 tests across the 7 adjacent SpecDec/MTP suites (SpecDec|ProviderMTPFactory|MTPPostureTelemetry):blobs/+snapshots/<rev>/layout with relative symlinks, then proves the full activation chain short of weight loading: direct inspection succeeds with facts byte-identical to the regular-file baseline (revision, payload bytes, config/index digests), the funnel admits it as.candidate(the status the slot factory consumes), andrevalidateForLoad— the last gate before assistant construction inEngineV2SlotFactory+MTP— resolves.Resultfailure namesbroken symlinkand the shard path; the funnel maps it to the existing fail-openinline_artifact_invalidstatus.no tensors with prefixinstead of a bare nil.Warning logs observed in the test run, e.g.:
Bug 2 from the same inventory (MTP acceptance unobservable outside Datadog Logs) is fixed separately in #619.