Build definitive MLX porting standard#12
Open
Amal-David wants to merge 20 commits into
Open
Conversation
Close the DNS-rebind TOCTOU in validate_sources.py by resolving once, vetting every address, and pinning vetted sockaddrs into the actual connection while TLS verification stays on the original hostname. Shared transport applied to contributor/top-model collectors; env proxies disabled in the pinned opener; redirects re-vet and re-pin. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
generate_claim_catalog.py no longer trusts recomputed_median_ratios from receipt_assessments.json: the field is structurally validated and, when receipts are available, recomputed from the re-hashed receipt payloads and compared exactly; mismatch fails closed. Adds a tamper regression test (inflated ratio, intact fingerprint). Promotion stays sealed: 10 observations / 0 promotion-ready / 1 rejected, 9 withheld. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
generate_evidence_index.py now enforces https link schemes, percent- encodes link-destination breakout characters without double-encoding, and escapes raw HTML in table cells. Hostile-fixture regression tests added; regenerated EVIDENCE_INDEX.md is byte-identical for the current clean sources. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
install_skill.py now uses a deterministic .<name>.backup directory for the force-replace swap and recovers at startup: restores the backup when the target is missing, finishes cleanup when the replacement already landed, and warns about legacy random-suffix backups without touching them. Adds four interrupted-state regression tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test_worker_retirement.py now catches real workers.dev hosts and any mlx-model-advisor*.pages.dev reference while allowing the current site host, and scans the whole repo with an explicit ignore-list instead of a hardcoded surfaces allowlist. daily-research.yml jobs get timeout-minutes and lose a dead variable. New contract tests pin site/data.js summary counts to live asset counts and verify knowledge-graph node/edge counts and edge endpoints. profile_eligible_range was left intact: it is consumed by recommend_optimizations.py, contrary to the review finding. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
First execution tool: captures token ids, embeddings, per-layer hidden states, attention/MLP intermediates where hookable, final norm, logits, and greedy continuations from a local PyTorch/HF checkpoint into a bounded .npz plus a deterministic digest manifest. torch/transformers stay optional lazy imports (fail closed with install guidance); trust_remote_code is impossible; output paths reject symlink components. Stable tensor-key scheme documented for the MLX-side mirror. 10 new contract tests (7 dependency-free, 3 torch-gated). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Generates a user-owned MLX implementation package (config parser, RMSNorm/RoPE incl. linear+dynamic-NTK scaling, GQA/MHA attention with KV cache, gated MLP, greedy generate CLI, oracle-mirroring capture mode) from a trusted inspection for the dense-decoder family via an explicit per-family registry. Computation-relevant config outside the feature allowlist fails closed with runbook guidance; generator is dependency-free; parameter names mirror HF conventions for the weight converter. Tests: 6 dependency-free contracts + real MLX tiny-model forward/generate/cache-consistency execution. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
safetensors-to-MLX conversion driven entirely by an explicit WEIGHT_MAP.json (schema 2: rename/transpose/permute/reshape/slice/ split/merge/cast with keep|f16|bf16|f32 dtype policies; legacy maps still accepted). Bounded pure-python safetensors header validation is always on; hostile headers, partial shards, unmapped or missing tensors, post-transform shape mismatches, and draft maps fail closed. Emits a validate_weight_map-compatible target manifest and a strict JSON conversion report; draft-map mode pairs only exact names. Tests: 10 dependency-free + 2 real mlx.core.load round-trips. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
capture_mlx.py mirrors the oracle tensor-key scheme from a scaffolded package + converted weights (package drift detection, --allow-modified, lazy mlx). run_parity.py orchestrates capture_oracle -> capture_mlx -> compare_tensors through bounded subprocesses and emits a strict-JSON parity ladder that stops at the first failing rung with a debug target. Shared capture/manifest logic factored into _capture_common.py; scaffold_port.py now emits scaffold-manifest.json. Keystone E2E test builds a tiny stock Llama in both frameworks, drives the real chain, passes clean parity, and catches a seeded transposed q_proj at layer.0.hidden (test init strengthened to std=0.20 attention projections after real-hardware verification exposed uniform-softmax insensitivity at default init). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Full chain on real bytes, fully offline from the local HF cache:
inspect -> plan -> oracle capture -> scaffold -> explicit 290-entry
WEIGHT_MAP -> convert (f32 parity artifact + bf16 variant) -> weight-map
validation -> 29-rung parity ladder -> exact greedy-token match across
torch oracle, scaffolded MLX port, and offline mlx_lm 0.31.1
(' The sky is blue because it reflects sunlight').
Two real tool bugs fixed: inert sliding-window metadata now allowed
only with use_sliding_window=false, and Q/K/V/O attention biases
inferred independently from tensor coverage (Qwen2 has QKV bias, no O
bias). Parity used a documented model-specific tolerance for
bf16-arithmetic source vs f32-arithmetic target; defaults and exact
token gate unchanged. Schema-2 receipt pair checked in and honestly
classified: 13 receipts = 12 observations / 0 promotion-ready / 1
rejected, execution_attested=false. Worked example (no weights,
portable paths) under examples/worked-port-qwen2.5-0.5b-instruct/.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New attested benchmark lane: a repository-owned runner (assets/benchmarks/runners/attested_mlx_port.py) whose bytes are digest-pinned at argv position 1 under -I -B isolation, parent-owned per-invocation random challenges binding label/phase/run/argv/quality bytes, dependency attestation from actually-loaded module bytes, and independent validator re-derivation of every digest. Five tamper scenarios fail closed with specific gate tokens; unattested lanes stay sealed. Real M4 Pro re-run of the Qwen2.5-0.5B pair yields the first legitimate promotion: bf16-weight-cast promoted at 1.0x-1.8122x, fingerprint-scoped, quality window and residual trust assumptions documented in VALIDATION.md. 13 receipts = 11 observations / 1 promotion-ready / 1 rejected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
recommend_optimizations.py now surfaces the knowledge graph as a clearly-labeled unreviewed-research-signals section (bounded 2MiB reads, schema fail-closed without blocking canonical advice, no numeric imports, execution_allowed=false). contributor-refresh.json feeds knowledge_curator source selection as review-only nodes. Orphan collect_top_models.py and its write-only snapshot removed. research_backlog.json is now deterministically reconciled from graph + update candidates with --reconcile-backlog/--check-backlog drift gates; graph grew to 697 nodes / 499 edges from checked-in candidates. New asset-consumer contract test prevents future write-only drift. Promoted claim machinery untouched (10 claims, 1 promoted). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every doc surface reconciled with reality: dense-decoder transformers are tooled end to end and proven by one real Qwen2.5-0.5B port; the other 16 families keep executable intake/routing/planning/validation with runbook-guided module implementation; domain evaluation beyond exact-output parity remains future work. Counts updated everywhere (29 scripts, 66 techniques, 423 tests, 13 receipts = 11/1/1, 10 claims with 1 fingerprint-scoped promotion, graph 697/499). Public-claims linter and 17/17 golden scenarios pass; no overclaims introduced. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner
Author
|
@coderabbitai full review @greptileai review |
…e lint statistics.pstdev became correctly-rounded in CPython 3.12, so 3.10 recomputed committed assessment bytes 1 ulp differently and failed the drift gate. All stored/diffed aggregates now use version-stable IEEE helpers (stable_mean/stable_median/stable_pstdev via math.fsum) in _common.py; zero statistics.* calls remain. Drift checks verified byte-identical on Python 3.10, 3.12, and 3.14. New .gitattributes marks byte-exact attestation evidence (-diff -merge) so the patch whitespace gate no longer lints dependency snapshots; evidence bytes and the promoted fingerprint are unchanged (13 receipts stay 11/1/1). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner
Author
|
@coderabbitai full review @greptileai review |
|
Important Review skippedToo many files! This PR contains 276 files, which is 126 over the limit of 150. To get a review, narrow the scope: Upgrade to a paid plan to raise the limit. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (30)
📒 Files selected for processing (276)
You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
✅ Action performedFull review finished. |
Owner
Author
|
@greptileai review |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
…city An adversarial full-PR review found the execution-attestation gate was author-forgeable: it verified unkeyed digests, structure, and nonce uniqueness but had no external signer or challenge issuer, so a receipt author could fabricate a coherent bundle (the old positive test did exactly that) and get execution_attested=true without running MLX. A sound gate needs an external trust root that cannot live in-repo. Re-seal: a missing-external-attestation-signature blocker keeps execution_attested false for every in-repo receipt. The bf16 Qwen2.5 receipt returns to performance_observation and all 10 effective claims return to withheld (12 obs / 0 promoted / 1 rejected). The runner and real receipts are kept as honest reproducibility evidence, not proof. The fabricating positive test now asserts the gate stays sealed. Docs corrected: no first-legitimate-promotion / content-signed / independently-proved wording; the 1.8122x figure is framed as one reproducible local observation, not a reliable speedup; VALIDATION.md and benchmarking.md state the real trust boundary and name an external signer as the prerequisite for any future attested promotion. .gitattributes narrowed so only binary payloads are diff-opaque; the validator rejects unreferenced attestation files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two correctness bugs in the code scaffold_port.py generates. Dynamic-NTK RoPE: cached keys were rotated under the pre-threshold base, so crossing original_max_position_embeddings mixed bases; generated models now reject KV-cache reuse past the threshold and recompute the full sequence, guaranteeing one consistent base (speed tradeoff documented in the generated README). Float16 padding: attention scores promote to float32 before the finite -1e9 mask (no float16 -inf), and fully-masked query rows are explicitly zeroed with where() so 0*NaN cannot propagate; capture_oracle (torch) and capture_mlx (mlx) mask consistently. New tests fail against the old template and pass on real Metal; generator identity 1.0.0 -> 1.0.1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…data recover_interrupted_install ran on every non-force install and treated a deterministic .<name>.backup sibling as installer-owned, deleting or moving unrelated user data of that name. Force replacement now uses a randomized backup plus an O_EXCL|O_NOFOLLOW recovery journal recording the exact target/backup/stage paths and device/inode/type identities, fsynced before the destructive rename. Recovery acts only when a journal and its recorded identities match; unjournaled .backup* siblings are warned about but never deleted, moved, or overwritten. Destination directory is fsynced after every rename for power-loss durability. Regression tests prove an unrelated legacy or random .backup survives a normal install; verified independently with a real copy-install round-trip. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ry bound Three converter findings from the ultra review. (A) validate_weight_map modulo-wrapped axes while convert rejected out-of-range, so some maps validated then failed conversion; both now share one schema/shape executor (_weight_map_common.py) with single normalized axis/negative/ default/slice semantics, so validation accepts exactly what conversion accepts. (B) mapping and shard-index files were parsed then reopened for their attested digest (TOCTOU); bytes are now read once and hashed from the same captured bytes. (C) the 16GiB cap was per-tensor with no aggregate bound; added configurable --max-working-set-bytes with preflight aggregate/peak estimates and one-tensor-at-a-time streaming output so a hostile sparse file fails closed before allocating. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three remaining ultra-review findings. generate_evidence_index.py now escapes Markdown link-label delimiters and neutralizes emphasis/code delimiters (code fences chosen longer than any hostile backtick run), so a title like x](attacker) cannot hijack a rendered link destination; tests assert the parsed destination, not just escaped source. validate_sources.py enforces one monotonic deadline across DNS, redirects, connection, TLS, and reads and caps attempts to four pinned addresses per host, closing the many-black-holed-address time amplification while keeping DNS-rebind pinning. MLX_KEYSTONE_REQUIRED=1 turns missing-MLX skips of the parity and generated-model-math keystones into failures; both Ubuntu jobs explicitly acknowledge the uncovered MLX math and VALIDATION.md documents the required Apple-Silicon path, so the coverage gap is loud, not hidden. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR ships two releases:
v0.4.0 — architecture-aware, evidence-gated MLX porting standard (original scope): rebuilds the app as an evidence-gated advisory standard, retires the public Worker for a deterministic static site, expands routing to 17 golden architecture families, centralizes numeric-claim authority in generated artifacts, and hardens every intake/benchmark/research/network/installer/CI surface.
v0.5.0 — the execution toolchain (added after a full multi-agent review + efficacy audit of v0.4.0): closes the "guides but never builds" gap and turns the skill into a working porting tool for dense-decoder transformers, proven by a real port.
v0.5.0 highlights
capture_oracle.py(PyTorch source-tensor capture),scaffold_port.py(generates a runnable MLX package: config parser, RoPE incl. scaling, GQA attention + KV cache, gated MLP, greedy generate, capture hooks),convert_checkpoint.py(safetensors→MLX via explicit WEIGHT_MAP schema 2 with rename/transpose/split/merge/cast, fail-closed),capture_mlx.py,run_parity.py(one-command parity ladder that stops at the first failing rung), shared_capture_common.py.mlx_lm(examples/worked-port-qwen2.5-0.5b-instruct/, no weights checked in). Two tool bugs found and fixed on real bytes.-I -B, parent-owned per-run random challenges, dependency-byte attestation from actually-loaded modules, independent validator re-derivation; five tamper scenarios fail closed). Result: 13 receipts = 11 observations / 1 promotion-ready / 1 rejected, and the first promoted claim —bf16-weight-castat 1.0x–1.8122x, fingerprint-scoped. The other 9 claims remain withheld.research_backlog.jsonis deterministically reconciled and drift-gated; every asset now has a consumer (contract-enforced); orphancollect_top_models.pyremoved.--forceinstall, a retirement IOC guard that matches realworkers.devhosts, CI job timeouts, and site/knowledge-graph drift-guard tests.Honest proof boundary (v0.5.0)
VALIDATION.md, not hidden.Verification
Supersedes #11.
🤖 Generated with Claude Code