ipld-bucket-test

Benchmark harness comparing go-pail against Bluesky's MST as candidate content-catalog structures for an S3-compatible object store. Investigation for Work Area 5 of the Storacha S3/MSP scoping brief.

Not a library. Not production code. A comparative benchmark and a report.

What this is

• A common Backend interface implemented by both go-pail and a vendored/patched copy of indigo's mst package.
• A latency-simulating blockstore interposer that adds configurable fixed + jitter delay per block fetch, modelling "every shard fetch is a network round trip."
• A stackable LRU cache layer with configurable size.
• A seeded workload generator producing S3-style keys in two profiles (nested paths and flattened single-slash) at 100k / 1M / 10M scale.
• A harness that runs scenarios across {backend × profile × scale × latency × cache × operation} and emits JSON results.
• REPORT.md — the hand-authored analysis and recommendation produced from the JSON output.

Running

# Populate fixtures and run the small matrix:
go run ./cmd/bench -config configs/full-matrix-small.yaml -out results/small.json

# Medium (1M keys):
go run ./cmd/bench -config configs/full-matrix-medium.yaml -out results/medium.json

# Reduced matrix at 10M:
go run ./cmd/bench -config configs/reduced-10m.yaml -out results/10m.json

Fixture files land in ./fixtures/{backend}-{profile}-{scale}-{seed}/ and are reused across runs. Expect fixture directories to reach multi-GB sizes at 10M scale; they are gitignored.

Report

See REPORT.md for methodology, results, structural observations, and the recommendation.

Status

Investigation in progress. See .specs/active/ipld-bucket-test/ in the parent repo for the feature brief, design, and task list driving this work.