Design reference-safe retention and regeneration for media variants

[58bits]

Context

Soft delete now retains source files and generated variants instead of deleting external storage after the database transaction. This is intentional: immutable versions and duplicated documents can share stored-file paths, so deleting an object merely because one document was soft-deleted is unsafe.

This design follow-up was split from #69 and the implementation plan in PR #71. Until this work is designed and approved, generated variants have the same retention behavior as their source files. There is no supported purge operation.

Problem

Byline cannot safely reclaim or regenerate generated media variants without first defining their identity, reproducibility, compatibility, and reference model across immutable document history and external storage.

IStorageProvider currently supports upload and delete, but has no provider-neutral read/download primitive. Persisted variant metadata records output name, path, dimensions, and format, but not the full recipe that produced the bytes. A source or variant path may also be retained by multiple immutable versions or duplicated documents.

Decisions required

• Decide whether variant storage paths remain persisted immutable identities or are derived dynamically from current rules.
• Define migration and compatibility behavior when variant path or naming rules change.
• Decide whether regenerated output must be byte-identical or only contract-equivalent.
• Define the generation recipe that must be persisted, including requested dimensions, fit, format, quality, and processor/version.
• Design a provider-neutral way to read retained source bytes.
• Define reference safety when immutable versions or duplicated documents share stored-file paths.
• Define behavior for legacy variants whose complete generation recipe is unavailable.
• Define retry and idempotency behavior across external storage and database state.
• Decide whether cleanup applies only to current-version variants or every immutable version.

Constraints

• Do not infer exclusive ownership of an object from a single document or version.
• Do not make external-storage deletion part of ordinary soft delete.
• Preserve adapter and storage-provider neutrality.
• A partial failure must not leave database metadata pointing at missing objects or delete paths still referenced by retained history.
• Existing retained variants must remain readable while any migration or regeneration is in progress.

Design deliverables

• An explicit object identity and reference model for source files and generated variants.
• A persisted recipe/versioning contract, including a legacy-data policy.
• Provider-neutral read, regenerate, write, and cleanup interfaces.
• Transaction/compensation and idempotency rules spanning database state and external storage.
• A rollout and compatibility plan for existing installations.
• Test scenarios covering shared references, immutable history, duplicates, retries, collisions, partial failures, and legacy variants.
• A decision on whether purge and regeneration are one operation family or separate workflows.

Relationship to #69

#69 releases the live path namespace on soft delete and deliberately retains stored assets. This issue owns the follow-on design needed before any irreversible source or generated-variant cleanup can be supported.

[58bits]

Two prerequisites that I think have to land before the purge/regeneration design can be settled. Both are small, both are already half-present in the tree.

1. store_file.file_hash is declared and indexed, but never populated

The column exists in both adapters, with an index and a stated purpose:

// packages/db-postgres/src/database/schema/index.ts
file_hash: varchar('file_hash', { length: 64 }), // SHA-256 hash for deduplication
...
index('idx_file_hash').on(table.file_hash),

It is threaded end to end — storage-flatten.ts, storage-restore.ts, store-manifest.ts, storage-row-types.ts, the Zod builder, and the insert/query paths in db-postgres and db-mysql. The copy-forward SQL in storage-commands.ts carries it across both restore-version and copy-to-locale, which are precisely the operations that multiply references to one stored object.

Nothing computes it. packages/core/src/services/field-upload.ts synthesises the envelope with:

fileHash: undefined,

The buffer is in scope a few lines above, and the file is already being read for metadata extraction and variant generation, so the digest is close to free at that point.

Why it matters here specifically. The blocker this issue names is proving exclusive ownership before reclaiming an object. Today that requires reasoning across versions, duplicated documents, and per-locale copies — a graph traversal with no cheap index. With the column populated it collapses to:

SELECT count(*) FROM byline_store_file WHERE file_hash = $1;

Shared-reference analysis becomes a query rather than a design problem. Content-addressed dedup at write time (probe the index before writing bytes) also becomes available without changing the human-readable storage-key scheme, since filename / originalFilename stay per-row.

The secondary benefit is fixity in the digital-preservation sense: recording a checksum at ingest and re-verifying on a schedule is a graded capability in the NDSA Levels and a first-class concept in PREMIS. Institutional buyers ask about it, and it is currently a column that promises something the code does not do.

Backfill for existing rows needs (2).

2. IStorageProvider has no read capability

The interface is upload / delete / getUrl, plus optional move and exists. There is no way to get bytes back out of a provider. That blocks three things at once:

• verifying fixity (re-hash and compare)
• backfilling file_hash for objects stored before (1)
• any regeneration path that needs the original bytes, which this issue already identifies

Proposal: an optional download(storagePath): Promise<Readable> (or getStream), following the existing optional-capability convention already established by move and exists — first-party providers implement it, custom providers may not, callers feature-detect. storage-local and storage-s3 both implement it trivially.

Suggested ordering

1. Add the optional read capability to IStorageProvider; implement in storage-local and storage-s3.
2. Compute SHA-256 in field-upload.ts and populate fileHash on the envelope. No migration required — the column and index already exist.
3. Backfill existing rows via (1), streaming, resumable.
4. Then design purge on top of a populated, indexed file_hash.

Steps 1 and 2 are independently useful and don't commit the project to any particular purge semantics, so they can merge ahead of the wider design.