safetensors: adopt upstream crate and add mmap-backed raw tensor access #86

Summary

Promote Safetensors from metadata/manifest-only support to a peer checkpoint backend beside GGUF.

Dependency decision: engram-parser should adopt the upstream Rust safetensors crate as an optional production dependency behind the existing off-by-default safetensors feature. The upstream crate becomes the canonical Safetensors format parser/validator; engram-parser keeps ownership of its public checkpoint/tensor abstractions, sharding/discovery policy, normalized metadata, and downstream topology-facing API.

Operational tracking: https://linear.app/rpd-34/project/annsnn-hybrid-69612b4fafb8

Scope

• Add upstream safetensors as an optional Cargo dependency and wire features.safetensors = ["dep:safetensors"].
• Replace overlapping home-grown Safetensors format parsing/validation where the upstream crate already provides the canonical behavior; do not maintain two independent parsers without a documented reason.
• Keep upstream types behind engram-owned public types so consumers do not depend directly on the upstream API.
• Preserve deterministic single-file, shard-index, and directory checkpoint discovery.
• Expose validated raw tensor payload access by tensor name.
• Support large sharded checkpoints without loading every shard into RAM.
• Provide mmap-backed/raw borrowed access where practical.
• Normalize dtype/shape/offset/shard data into engram-owned types.
• Allow existing MoE discovery results to resolve to real raw tensors.

Dependency boundary

The dependency direction is:

safetensors → internal format backend used by engram-parser.

It must not invert into safetensors knowing about engram-parser, and it must not leak upstream implementation types into engram-parser's stable public contract.

This issue does not add gguf-rs, gguf-rs-lib, neuromod, nir-rs, myelin-accelerator, or other SNN/GPU crates to the core dependency graph.

Non-goals

• No Hugging Face config.json model policy.
• No CUDA/GPU registration.
• No matmul/inference runtime.
• No GOZ1.
• No Grok-specific quantization policy.
• No replacement of the native GGUF backend.

Acceptance criteria

• Cargo.toml contains an optional upstream safetensors dependency.
• The safetensors Cargo feature activates dep:safetensors.
• --features safetensors actually uses the upstream crate internally for canonical Safetensors parsing/validation.
• Redundant in-tree parsing code is removed or retained only with an explicit documented responsibility that upstream does not cover.
• Safetensors tensors can be looked up and read as validated raw payload bytes.
• Sharded checkpoints remain deterministic and path-safe.
• Large-checkpoint path does not require whole-checkpoint copies.
• Public API does not leak upstream implementation types unnecessarily.
• Tests cover single-file and sharded checkpoints plus malformed/bounds cases.
• README/CHANGELOG retire the zero-dependency Safetensors claim.
• Default builds remain free of the Safetensors dependency when the feature is disabled.

Release

Target: v0.3.0 — first crates.io checkpoint-substrate release.