Z-Space Core

Content-addressable tensor memory for efficient, exact, and fault-tolerant checkpoint storage.

Z-Space Core stores tensor checkpoints as a content-addressed graph instead of treating every checkpoint as an unrelated file. It combines deterministic tensor descriptors, reversible deltas, XOR+Zstandard compression, persistent packfiles, and integrity verification.

Why it matters

Training checkpoints can consume more storage than the model itself because most consecutive checkpoints contain highly related weights. Z-Space Core captures those changes while preserving exact reconstruction.

The design targets:

• Lossless checkpoint versioning.
• Deduplicated, content-addressed tensor storage.
• Exact reconstruction after restart.
• Durable append-only storage with recovery after interrupted writes.
• Efficient storage of large `fp32` and `bf16` tensor deltas.

Highlights

• SHA-256 addresses for descriptors and content nodes.
• Merkle-DAG-style checkpoint graph.
• `RAW`, `SPARSE`, `SVD`, and optional TensorLy codecs.
• Progressive loading: approximate with `exact=False`, verified reconstruction with `exact=True`.
• Reversible `add`, `mul`, and `patch` deltas.
• Tensor-wise `XOR+zstd` deltas with chunking for large tensors.
• Safe `mscs` serialization without `pickle`.
• Append-only packfile with a persistent journal/index.
• Group durability barriers using `fsync`.
• Integrity verification when reading content.
• LRU caching that returns clones to preserve content immutability.

Benchmark snapshot

The results below are measurements already recorded in this repository. Storage is shown in decimal GB. Results depend on model, dtype, checkpoint interval, hardware, and compression policy; they are not universal guarantees.

Recorded 100-checkpoint Pythia-410M benchmark

Configuration: `EleutherAI/pythia-410m`, `bf16`, 100 checkpoints over 5,000 training steps, with exact reconstruction.

Method	Stored data	Ratio vs. `torch.save`	Savings
PyTorch checkpoints	81.076 GB	100.00%	0.00%
DVC content-addressable baseline	81.076 GB	100.00%	0.00%
Safetensors + zstd	62.669 GB	77.30%	22.70%
Z-Space, XOR+zstd, no periodic full	5.346 GB	6.59%	93.41%
Z-Space, XOR+zstd, `full_every=8`	14.400 GB	17.76%	82.24%

The operational `full_every=8` policy reconstructed the final checkpoint in 18.456 seconds. The maximum-compression policy reconstructed it in 533.337 seconds. This is the central trade-off: periodic full checkpoints use more storage but make random recovery much faster.

See the full methodology and baselines in docs/paper_benchmark_metrics.md.

Current large-run validation

An additional 10-checkpoint `bf16` run with Pythia-410M used synthetic data and the repaired persistent store implementation:

Metric	Result
PyTorch checkpoint total	8.108 GB
Z-Space store	2.059 GB
Storage savings	74.60%
Final reconstruction	Exact (`true`)
Average commit time	12.315 s
Final reconstruction time	11.054 s
Total wall time	164.219 s

This shorter synthetic run is a durability and implementation check, not a replacement for the 100-checkpoint training benchmark.

Store validation suite

The validation suite currently checks 25 versions and reported:

• Exact reconstruction for every version.
• Successful close and reopen recovery.
• Corruption detection.
• Recovery after an interrupted write.
• Chunked large-delta handling.
• 157.439 MB of full snapshots versus 101.799 MB in Z-Space: 35.34% savings for that test workload.

Run it with:

python scripts/validation_suite.py

Installation

python -m venv .venv
.\\.venv\\Scripts\\Activate.ps1
pip install -r requirements.txt

`mscs` is required for component and delta serialization. `tensorly` and `lz4` are optional at runtime; the core codecs continue to work without them.

Quick example

import torch
from z_space_core import DecompType, ZSpace

space = ZSpace(cache_size=1 << 28)

a = torch.arange(120, dtype=torch.float32).reshape(20, 6)
b = torch.arange(90, dtype=torch.float32).reshape(6, 15) / 31
tensor = a @ b

descriptor = space.register(
    "low_rank_matrix",
    tensor,
    decomp_type=DecompType.SVD,
    target_ratio=0.4,
    exact=True,
)

approx = space.load("low_rank_matrix", exact=False)
exact = space.load("low_rank_matrix", exact=True)

assert torch.equal(exact, tensor)
print(descriptor.address_hex)

Tests and benchmarks

Run the unit and recovery tests:

python -m unittest discover -q

Run the smoke tests:

python scripts/smoke_2m_model.py
python scripts/smoke_model_versioning.py

The benchmark scripts are in scripts/, including the independent comparison and validation suite. Generated stores and benchmark outputs are excluded from Git by default.

Repository layout

z_space_core.py                 Core storage and reconstruction engine
tests/                          Unit, persistence, corruption, and model tests
scripts/                        Smoke tests, validation, and benchmarks
docs/                           Benchmark reports and technical postmortems

Status

The project is an experimental research-grade storage engine. The persistence path is covered by restart, corruption, interrupted-write, and exact-reconstruction tests, but production deployment should still add workload-specific durability, concurrency, backup, and disaster-recovery testing.

License

Z-Space Core is released under the MIT License.