Repository navigation
[engram-parser] GH#10 — safetensors: coordinate ecosystem / sibling parser extraction from corinth-canal (separate from GGUF #7; one-way copy, no dep on corinth-canal) #10
Description
Activity
- addedmodularizationWork to make repos more modular and overlappingWork to make repos more modular and overlappingapiAPI changes or trait workAPI changes or trait workextractionMain extraction coming from rmems/corinth-canalMain extraction coming from rmems/corinth-canal
on Jul 2, 2026 - changed the title
[-]safetensors: coordinate ecosystem / sibling parser extraction from corinth-canal (separate from GGUF #7; one-way copy, no dep on corinth-canal)[/-][+][engram-parser] GH#10 — safetensors: coordinate ecosystem / sibling parser extraction from corinth-canal (separate from GGUF #7; one-way copy, no dep on corinth-canal)[/+]on Jul 26, 2026 Decision reversed: safetensors lands in engram-parser behind a cargo feature
Charter reversal (2026-08). This issue, the engram-parser README, corinth-canal
docs/MODULE_STATUS.md, and cortex-tensor#9 all previously stated that the extractable safetensors surface would land in a new dedicatedsafetensors-parsercrate, and that "engram-parser charter remains GGUF-only." That is reversed. The extractable set —manifest,discovery,json,paths,validatefrom corinth'ssrc/moe/safetensors/— is ported into engram-parser behind an off-by-defaultsafetensorscargo feature. Nosafetensors-parserrepo was or will be created.Why: the charter was never "GGUF". It is zero-dependency checkpoint deserialization plus MoE raw-weight extraction — and safetensors header inspection is exactly that shape: header-only parse, deterministic manifest, name/shape MoE candidate discovery, raw bytes out, no math, no mmap. A third crate would have duplicated this crate's error type, dtype model, MSRV policy, CI, Docker, and release plumbing to host ~1.4k lines that share every one of its invariants. Consumers had already converged on this home without being asked: rmems/hybrid-fusion#27 states "Concrete memmap / header parse / payload extract →
engram-parser(same rule as GGUF)". A cargo feature buys the same isolation a separate crate would have: default builds stay byte-for-byte GGUF-only, and the zero-dependency guarantee holds in every feature combination —[dependencies]stays empty, and the upstreamsafetensorsandserde_jsoncrates remain forbidden dependencies.Unchanged, and still governing: this is a one-way copy from inspiration. engram-parser takes no dependency on corinth-canal, and corinth-canal takes no dependency on engram-parser for safetensors. corinth-canal keeps its unmodified reference copy of
src/moe/safetensors/and keeps using it in its Router /CheckpointBackendexperiment paths. The corinth-specificconfig(HFconfig.json) andmap(mmap load / token extract) modules are not ported.What this changes in this issue
The body is preserved unedited under a superseded banner rather than rewritten, so the original intent stays legible. Four acceptance criteria needed their text changed, not just their boxes ticked:
- AC 9 is directly contradicted. "engram stays GGUF-only" is the exact thing being reversed. Its other clause, "Limen-Neural org for reusable libs", is also stale —
engram-parserandcortex-tensorwere transferred to rmems. The two surviving clauses (one-way copy, no dep) are preserved verbatim everywhere. - AC 5's "MODULE_STATUS for safetensors layer toward frozen" is dropped. The module stays at
referencepermanently: corinth's ladder definesfrozenas "the modular crate becomes the source of truth", and that is not happening — both copies stay live and are expected to diverge. corinth'sdocs/PROMOTION_RULES.mdgains a One-way extractions section documenting the departure. - AC 6's "Linear LIM-88" is stale — LIM-88 is archived with status Duplicate. The live tracker is RM-344.
- AC 10 has no committable target. There is no
plan.mdat repo root, anddocs/superpowers/is gitignored by design (.gitignore:15, documented inREVIEW.md§7). The committed record is insteadREADME.md§ Origin / modularization ([engram-parser] GH#10 — safetensors: coordinate ecosystem / sibling parser extraction from corinth-canal (separate from GGUF #7; one-way copy, no dep on corinth-canal) #10) +CHANGELOG.md. Noting this so it is not re-litigated later as an unmet AC.
AC 4 is reinterpreted: it asks for an "ecosystem/sibling note", but there is no sibling. It becomes an in-crate charter note.
Most cross-linking ACs were already satisfied
corinth#116, cortex-tensor#9, and engram#7 already carried the #10 link and the "no dep on corinth-canal" language before this pass. What was actually missing was the reversal.
Not blocked by #45
This surface is header-only — no mmap, no dequant. It does not wait on #45, which blocks the GGUF consumer story in corinth-canal#115. Worth stating explicitly since RM-344 is already
relatedTothe mmap issues in Linear.In flight
engram-parser docs #57 corinth-canal docs rmems/corinth-canal#156 the code port follow-up PR, branch feat/safetensors-10-portStill stale elsewhere
rmems/cortex-tensorREADME.md:79/:113andsrc/moe/mod.rs:33, plus theengram-parserandcortex-tensorGitHub wikis, still describe thesafetensors-parsercrate. Tracked separately.(Housekeeping: the bead id
raulmc-yqjcited in comments on corinth#116 and engram#7 does not exist — the real bead iscorinth-canal-le53qh.)- AC 9 is directly contradicted. "engram stays GGUF-only" is the exact thing being reversed. Its other clause, "Limen-Neural org for reusable libs", is also stale —
- added a commit that references this issue
on Aug 15, 2026 5 remaining items
- added 6 commits that reference this issue
on Aug 15, 2026 rmems commented
on Aug 22, 2026 OwnerAuthorMore actions2026-08-22 architecture clarification
Current intent is stronger than the historical "copy forever, no dependency in either direction" wording preserved in this issue body.
engram-parseris the canonical reusable home for Safetensors header deserialization, deterministic manifests, and MoE router/expert discovery behind its off-by-defaultsafetensorsfeature.engram-parserremains zero-dependency: this means supporting/parsing the Safetensors format; it does not mean adding the upstream Rustsafetensorscrate as a dependency.engram-parsermust never depend oncorinth-canal.- Corinth may retain its local Safetensors reference implementation while the reusable feature is incomplete, but under
rmems/corinth-canal#161the long-term architecture should re-evaluate Corinth consuming engram-parser once the required reusable surface, parity, mmap/payload boundaries, and performance gates are adequate. Permanent duplication is no longer the desired default. - No separate
safetensors-parserrepository should be created.
This lane belongs to Corinth's
v0.3.0extraction program (rmems/corinth-canal#161, milestone #11).A focused README cleanup issue is being created so the README stops describing Safetensors as "copy, never a dependency" while still being precise that the feature is not shipped until implementation lands.
Planning attribution: GPT-5.6 Sol (ChatGPT/OpenAI) with @rmems, 2026-08-22.
@coderabbitai plan
coderabbitai commented
on Aug 23, 2026 coderabbitaiboton Aug 23, 2026 – with coderabbitaiContributorMore actionsCoding Plan
Summary
Treat GH#10 as a coordination and documentation ticket. Add no Rust code, no
safetensorsfeature, and no dependency, per the ticket non-goals.Audit and align the two committable in-repo records,
README.mdandCHANGELOG.md, so both carry the four required clarifications, the reversal statement, the source/exclusion boundaries, and the cross-links.Confirm
REVIEW.mddoes not contradict the aligned documentation. Leave the intentionaldocs/superpowers/gitignore untouched.Route all cross-repository and issue-tracker actions to the follow-up section, because they are not committable in this repository.
Follow-up actions outside this repository (not committable here)
The following acceptance criteria cannot be satisfied by changes in the
engram-parsercodebase. Record them for the owner or a coordination agent to execute in the named external systems. Use the same clarification language in every place: one-way copy from inspiration;rmemsorg for reusable libraries; no dependency oncorinth-canalin either direction;engram-parserowns safetensors behind an off-by-defaultsafetensorscargo feature.- Create or confirm the dedicated GitHub issue
Limen-Neural/engram-parser#10with full cross-links and the "no dep on corinth-canal" language. - Add or update cross-link comments on
corinth-canal#116,cortex-tensor#9, and this repo's#7. - Update
corinth-canaldocs: retargetdocs/MODULE_STATUS.mdandmanifests/proven_components.tomltormems/engram-parser(featuresafetensors), keep statusreference(not "toward frozen"), and add adocs/PROMOTION_RULES.md"One-way extractions" section. Add explicit "no dep" language oncorinth-canal#115andcorinth-canal#116. - Update Linear
rmems/engram-parser#10with the reversal. Note thatLIM-88is archived/Duplicate andrmems/engram-parser#10is its live successor. - Create or update beads in
corinth-canalfor the separate Safetensors extraction. - Acknowledge coordination with a comment in
cortex-tensor#7or#9.
Design Choices
Design Choice 1: Code, feature-flag, or module scaffolding requirement
Options Considered:
- Scaffold
src/safetensors/and add asafetensors = []feature now. - Treat the ticket as documentation and coordination only, with no code.
Chosen Option: 2
Rationale: The ticket states that it does not change engram-parser's GGUF-only implementation surface. It states that no code has landed yet and that this entry records the decision only. The last acceptance criterion names
README.mdandCHANGELOG.mdas the only committable record.Design Choice 2: Scope of in-repo work given existing documentation
Options Considered:
- Assume the docs are complete and make no changes.
- Audit the existing docs against the ticket's explicit clarification checklist and refine any gaps.
Chosen Option: 2
Rationale: The acceptance criteria list four specific clarifications that every place must carry. The safe action is a targeted audit-and-align pass, not an assumption of completeness.
💡 User Tips
Regenerate the plan with different choices with
@coderabbitai <feedback>.Implementation Steps
In-repo documentation alignment
This phase makes the two committable records in this repository consistent with the superseded-plan decision. The goal is to guarantee that every required clarification appears, that cross-links render, and that the wording matches prior reviews. This phase is the full in-repo scope of the ticket. It adds no source code, no feature flag, and no dependency.
Task 1: Audit and align the README ecosystem/sibling note
Confirm that the safetensors sections carry all four ticket clarifications. Correct any gaps.
- Review
README.mdsections "Planned / in flight — Safetensors", "Origin / modularization — Safetensors (#10)", the "Scope / boundaries" table, and the "Ecosystem / promotion model". - Verify the four required clarifications are present and explicit: (1) one-way copy from inspiration; (2)
rmemsorg for reusable libraries; (3) no dependency oncorinth-canalin either direction; (4)engram-parserowns safetensors behind an off-by-defaultsafetensorscargo feature. - Verify the reversal statement remains: the previous "permanently GGUF-only" and "separate
safetensors-parserrepository" plan is superseded. No separate parser repository is created. - Verify the source-boundary text names
corinth-canal/src/moe/safetensors/as the promote-from surface. It must list the "do not promote" exclusions (Corinthconfig.jsonadapter policy, runtime/payload mmap, GPU registration/dequantization, SAAQ orchestration, machine-local config). - Verify the zero-dependency charter statement stays: no upstream
safetensorscrate, noserde_json,[dependencies]stays empty in every feature combination. - Verify cross-reference links to
engram-parser#7,engram-parser#10,engram-parser#61,corinth-canal#115,corinth-canal#116,corinth-canal#161, andcortex-tensor#9are present and render. - If any clarification, link, or exclusion is missing or inconsistent, edit the prose to add it. Keep the existing section structure and heading names.
Task 2: Align the CHANGELOG Unreleased entry
Confirm the
[Unreleased]entry records the decision with the required clarifications and issue references.- Review the
[Unreleased] → ### Changed"Charter reversal (#10)" entry inCHANGELOG.md. - Verify the entry states: safetensors ships inside this crate behind an off-by-default
safetensorscargo feature; not in a separatesafetensors-parsercrate; the zero-dependency guarantee is unchanged; the extraction is a one-way copy with no dependency in either direction; no code has landed yet. - Verify the entry cites the relevant issues and cross-repo docs already referenced (
#10,#45,corinth-canal docs/MODULE_STATUS.md,cortex-tensor#9). - Keep the Keep-a-Changelog format: bolded short label, issue references, bullet under
## [Unreleased]. - If the entry omits any of the four ticket clarifications, extend the existing bullet. Do not add a duplicate entry.
Task 3: Confirm REVIEW.md consistency
Ensure
REVIEW.mddoes not contradict the aligned README and CHANGELOG.- Review
REVIEW.mdcharter language and its §7.gitignorenote. - Confirm
REVIEW.mddescribes the charter as "GGUF today, plus safetensors headers once#10lands", consistent withREADME.md. - Confirm no remaining unqualified "GGUF-only" charter claim exists that contradicts the reversal.
- If a contradicting claim exists, update the prose to match the README wording. If none exists, make no change.
- Do not change the §7 note about
docs/superpowers/. It is intentional background, not an actionable item.
🤖 Prompt for AI agents
Implement Phase 1: In-repo documentation alignment. This phase is documentation-only. Add no source code, no feature flag, and no dependency. Task 1 — Audit and align the README ecosystem/sibling note: - Review `README.md` sections "Planned / in flight — Safetensors", "Origin / modularization — Safetensors (`#10`)", the "Scope / boundaries" table, and the "Ecosystem / promotion model". - Verify these four clarifications are present and explicit: (1) one-way copy from inspiration; (2) `rmems` org for reusable libraries; (3) no dependency on `corinth-canal` in either direction; (4) `engram-parser` owns safetensors behind an off-by-default `safetensors` cargo feature. - Verify the reversal statement is present: the previous "permanently GGUF-only" and "separate `safetensors-parser` repository" plan is superseded, and no separate parser repository is created. - Verify the source-boundary text names `corinth-canal/src/moe/safetensors/` as the promote-from surface, and lists the "do not promote" exclusions: Corinth `config.json` adapter policy, runtime/payload mmap, GPU registration/dequantization, SAAQ orchestration, machine-local config. - Verify the zero-dependency charter statement is present: no upstream `safetensors` crate, no `serde_json`, `[dependencies]` stays empty in every feature combination. - Verify cross-reference links to `engram-parser#7`, `engram-parser#10`, `engram-parser#61`, `corinth-canal#115`, `corinth-canal#116`, `corinth-canal#161`, and `cortex-tensor#9` are present and render correctly. - Edit the prose to add any missing clarification, link, or exclusion. Keep the existing section structure and heading names unchanged. Task 2 — Align the CHANGELOG Unreleased entry: - Review the `[Unreleased] → ### Changed` "Charter reversal (`#10`)" entry in `CHANGELOG.md`. - Verify the entry states: safetensors ships inside this crate behind an off-by-default `safetensors` cargo feature (not a separate `safetensors-parser` crate); the zero-dependency guarantee is unchanged; the extraction is a one-way copy with no dependency in either direction; no code has landed yet. - Verify the entry cites `#10`, `#45`, `corinth-canal docs/MODULE_STATUS.md`, and `cortex-tensor#9`. - Keep the Keep-a-Changelog format: bolded short label, issue references, bullet under `## [Unreleased]`. - If any of the four ticket clarifications is missing, extend the existing bullet instead of adding a duplicate entry. Task 3 — Confirm REVIEW.md consistency: - Review the `REVIEW.md` charter language and its §7 `.gitignore` note. - Confirm `REVIEW.md` describes the charter as "GGUF today, plus safetensors headers once `#10` lands", matching `README.md`. - Confirm no unqualified "GGUF-only" charter claim remains that contradicts the reversal. - If a contradicting claim exists, update the prose to match the README wording; otherwise make no change. - Leave the §7 note about `docs/superpowers/` unchanged, since it is intentional background.Research
engram-parseris a pure-Rust, zero-dependency crate.src/lib.rsdeclareserror,gguf, andmoemodules. No#[cfg(feature)]gating exists today.Cargo.tomlhas empty[dependencies]and[features] default = []. No safetensors code, tests, or examples exist.README.mdalready carries a "Planned / in flight — Safetensors" section, an "Origin / modularization — Safetensors (#10)" section, a scope/boundaries table, and an ecosystem/promotion model.CHANGELOG.mdalready carries a single[Unreleased] → ### Changed"Charter reversal (#10)" entry.REVIEW.md§7 documents the intentionaldocs/superpowers/gitignore.The in-repo work for this ticket is to audit and align existing documentation against the ticket's required clarifications. It does not introduce new concepts.
🚀 Next Steps
🤖 All AI agent prompts combined
Implement Phase 1: In-repo documentation alignment. This phase is documentation-only. Add no source code, no feature flag, and no dependency. Task 1 — Audit and align the README ecosystem/sibling note: - Review `README.md` sections "Planned / in flight — Safetensors", "Origin / modularization — Safetensors (`#10`)", the "Scope / boundaries" table, and the "Ecosystem / promotion model". - Verify these four clarifications are present and explicit: (1) one-way copy from inspiration; (2) `rmems` org for reusable libraries; (3) no dependency on `corinth-canal` in either direction; (4) `engram-parser` owns safetensors behind an off-by-default `safetensors` cargo feature. - Verify the reversal statement is present: the previous "permanently GGUF-only" and "separate `safetensors-parser` repository" plan is superseded, and no separate parser repository is created. - Verify the source-boundary text names `corinth-canal/src/moe/safetensors/` as the promote-from surface, and lists the "do not promote" exclusions: Corinth `config.json` adapter policy, runtime/payload mmap, GPU registration/dequantization, SAAQ orchestration, machine-local config. - Verify the zero-dependency charter statement is present: no upstream `safetensors` crate, no `serde_json`, `[dependencies]` stays empty in every feature combination. - Verify cross-reference links to `engram-parser#7`, `engram-parser#10`, `engram-parser#61`, `corinth-canal#115`, `corinth-canal#116`, `corinth-canal#161`, and `cortex-tensor#9` are present and render correctly. - Edit the prose to add any missing clarification, link, or exclusion. Keep the existing section structure and heading names unchanged. Task 2 — Align the CHANGELOG Unreleased entry: - Review the `[Unreleased] → ### Changed` "Charter reversal (`#10`)" entry in `CHANGELOG.md`. - Verify the entry states: safetensors ships inside this crate behind an off-by-default `safetensors` cargo feature (not a separate `safetensors-parser` crate); the zero-dependency guarantee is unchanged; the extraction is a one-way copy with no dependency in either direction; no code has landed yet. - Verify the entry cites `#10`, `#45`, `corinth-canal docs/MODULE_STATUS.md`, and `cortex-tensor#9`. - Keep the Keep-a-Changelog format: bolded short label, issue references, bullet under `## [Unreleased]`. - If any of the four ticket clarifications is missing, extend the existing bullet instead of adding a duplicate entry. Task 3 — Confirm REVIEW.md consistency: - Review the `REVIEW.md` charter language and its §7 `.gitignore` note. - Confirm `REVIEW.md` describes the charter as "GGUF today, plus safetensors headers once `#10` lands", matching `README.md`. - Confirm no unqualified "GGUF-only" charter claim remains that contradicts the reversal. - If a contradicting claim exists, update the prose to match the README wording; otherwise make no change. - Leave the §7 note about `docs/superpowers/` unchanged, since it is intentional background.💡 Iterate on the plan with:
`@coderabbitai` <feedback>Example Feedback - `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2. - `@coderabbitai` For design choice 1 go ahead with option 3 and replan.
- Create or confirm the dedicated GitHub issue
- moved this from Backlog to In progress in Limen-Neural Consolidation
on Sep 15, 2026 - added a commit that references this issue
on Sep 15, 2026 - moved this from In progress to Done in Limen-Neural Consolidation
on Sep 15, 2026
Metadata
Metadata
Assignees
Labels
Projects
- StatusShow more project fieldsDone
Source
Synced / linked to GitHub for 1:1 mirror (2026-07-26).
Limen-Neural/engram-parserGH#10Summary
Track the separate extraction (one-way copy of implementation as inspiration) of the reusable Safetensors header inspection, deterministic manifest generation, and MoE router/expert candidate discovery logic from the experimental reference implementation in
rmems/corinth-canal(parallel to the GGUF work in this repo's #7).This is the separate dedicated issue in engram-parser (per user request) for the Safetensors case. Companion/parallel to GGUF #7 / corinth#115. See corinth-canal#116 (the supporting/bootstrap issue in the source experimental repo) and Limen-Neural/cortex-tensor#9 (coord).
Important clarification (per modularization intent and prior reviews): This is one-way extraction/copying. We are not making corinth-canal depend on any new or target crate. corinth-canal remains self-contained as the experimental end-to-end reference (per its own PROMOTION_RULES.md: after "frozen", it keeps an unmodified reference copy of the logic). "we are not making corinth-canal dep. We are simply copying and implementing code from inspiration." "corinth-canal repo more of my experimental repo. Limen-Neural organization repos are more about modular reusable libraries. Ideally I am trying to extract useful code from corinth-canal repo."
Context
src/moe/safetensors.rs+safetensors/discovery.rs) is a self-contained pure-Rust implementation (header parse without the officialsafetensorscrate for the header/inspect path, sharded/HF index/dir support,inspect_safetensors_checkpoint/write_safetensors_manifest,SafetensorsManifestwith tensor records (name/dtype/shape/byte_size/offsets/shard) + metadata + checkpoint source, MoE candidate labeling via name/shape heuristics for routers/experts/groups (classify_tensor,discover_candidates,SafetensorsRouterCandidate,SafetensorsExpertGroup, layout family detection e.g. "generic_moe", "deepseek_v3_family", "phimoe" etc.), plusMappedSafetensorsCheckpointfor loading tensor bytes on demand).examples/safetensors_manifest.rs, verification checklist, and dual-backend (with GGUF) in the Router/Checkpoint for experiments.rmems-moe).Related
Goal
Ensure the Safetensors extraction from corinth-canal (as inspiration, one-way copy, no dep) is tracked with a dedicated separate GitHub issue in this repo (engram-parser), parallel to #7. This provides visibility in the parser-related issues here, cross-links the full story (corinth source #116, cortex coord #9, future reusable crate work, LIM-9), and follows the precedent of explicit GitHub issues for such modularization.
(The actual reusable crate implementation can be a new Limen-Neural/safetensors-parser or placed per future decision; this issue tracks the extraction effort and ecosystem from engram's perspective.)
Non-goals
Acceptance criteria
docs/MODULE_STATUS.md+manifests/proven_components.tomlretargeted tormems/engram-parser(featuresafetensors), status staysreference(toward frozen— a one-way copy is not a source-of-truth handoff), plus adocs/PROMOTION_RULES.md"One-way extractions" section; explicit "no dep" language on rmems/corinth-canal#115 / rmems/corinth-canal#116.LIM-88is archived/Duplicate — not the tracker; [engram-parser] GH#10 — safetensors: coordinate ecosystem / sibling parser extraction from corinth-canal (separate from GGUF #7; one-way copy, no dep on corinth-canal) #10 is its live successor.Limen-Neural— both repos were transferred); no dep on corinth-canal in either direction; engram-parser owns safetensors behind an off-by-default cargo feature (engram stays GGUF-only— reversed).Plan.md updated with the new issue number.No committable target: there is no rootplan.md, anddocs/superpowers/is gitignored by design (.gitignore:15, perREVIEW.md§7). The committed record isREADME.md§ *Origin / modularization (*[engram-parser] GH#10 — safetensors: coordinate ecosystem / sibling parser extraction from corinth-canal (separate from GGUF #7; one-way copy, no dep on corinth-canal) #10) +CHANGELOG.md.Validation
Suggested branch
docs/safetensors-sibling-extraction(orfeat/safetensors-ecosystemif any engram updates).References
Created as the separate GitHub issue in engram-parser for the Safetensors extraction from corinth-canal (parallel to #7; per user request for the issue here, one-way copy, no dep on corinth-canal, following modularization pattern from LIM-9 and GGUF precedent).
Siblings (cross-linked/updated together):