Migrate retained substrates directly into indexed graph roots #822

1. Background Context

Source: git-warp #822. Audited against main 94b40dac64034cd8caab9bb05efe14a0c22bd735. Template: feature; work type: type:feature.

Current scope and disposition: Migration completion requires indexed reduction, attachment streams, executable recursive traversal and retained child-root ownership; these were incompletely modeled in the previous graph.

Reported context (historical names and counts require the current disposition above):
Plan: https://github.com/git-stunts/git-warp/blob/plan/streaming-recursive-materialization/docs/plans/streaming-indexed-recursive-warp.md
Design PR: #813

Discussion evidence, including corrections:

• Comment ABYi6F_A

2. Problem Description

Migration completion requires indexed reduction, attachment streams, executable recursive traversal and retained child-root ownership; these were incompletely modeled in the previous graph.

Historical source report; the current disposition above supersedes obsolete claims:
Replace the v18-to-v19 monolithic replay-basis translation with a script-owned streaming migration that constructs dictionaries, address pages, Roaring roots, causal roots, attachments, graph references, and a retained manifest directly.

• Keep all legacy decoding under scripts/.
• Integrate graph discovery, version status, one-pass confirmation, Bijou progress, capacity estimates, and multi-graph isolation from Make v18-to-v19 migration discover graphs before confirmation #806–Define and satisfy the v19 migration guide requirements #810.
• Stream legacy checkpoint components and writer patches through bounded reducers.
• Verify bounded reopen, append, property reads, attachment streaming, neighborhood and recursive traversal, receipts, and semantic equivalence in scratch.
• Refresh every source ref and promote through compare-and-swap with additive recovery refs.
• Re-run the same verification after promotion and roll back safely on failure.

2b. Proposed Solution

Migration completion requires indexed reduction, attachment streams, executable recursive traversal and retained child-root ownership; these were incompletely modeled in the previous graph.

Historical approach to reconcile:
Current architecture rules take precedence: parsing/encoding stays at adapters, domain concepts are runtime-backed, no trust casts are introduced, and tests run in Docker.
Replace the v18-to-v19 monolithic replay-basis translation with a script-owned streaming migration that constructs dictionaries, address pages, Roaring roots, causal roots, attachments, graph references, and a retained manifest directly.

• Keep all legacy decoding under scripts/.
• Integrate graph discovery, version status, one-pass confirmation, Bijou progress, capacity estimates, and multi-graph isolation from Make v18-to-v19 migration discover graphs before confirmation #806–Define and satisfy the v19 migration guide requirements #810.
• Stream legacy checkpoint components and writer patches through bounded reducers.
• Verify bounded reopen, append, property reads, attachment streaming, neighborhood and recursive traversal, receipts, and semantic equivalence in scratch.
• Refresh every source ref and promote through compare-and-swap with additive recovery refs.
• Re-run the same verification after promotion and roll back safely on failure.

2c. Alternatives considered and rejected

No additional alternatives are recorded as decided. Reject duplicate ownership, private-import escape hatches and a broken intermediate mainline; retain original alternatives below when present.

2d. Acceptance Criteria

• Migration never constructs or publishes a full-state replay basis.
• No production full-state codec is called.
• A generated legacy state exceeding 5 MiB migrates under a reviewed heap cap.
• Normal execution performs expensive migration work once.
• A disposable real Think copy passes before the authoritative graph is retried.
• Make v18-to-v19 migration discover graphs before confirmation #806–Define and satisfy the v19 migration guide requirements #810 are completed against the new representation.

2e. Test Plan

Golden: Migrate a disposable repository, reopen and compare graph semantics through the supported boundary.

Edges: Empty/multiple graphs, symlink paths, stale refs, unsupported formats and missing frontier evidence.

Known failure modes: CAS refusal, interruption and post-promotion failure preserve verifiable recovery.

Fuzz and stress: Generated deep/wide/path-sensitive repositories under a Docker memory cap; no host repository mounts.

All tests and benchmarks execute in COPY-based Docker containers without host repository or Git-directory mounts. This planning audit does not claim those checks were run.

3. Prerequisites

Completion prerequisites; preparatory work may start earlier.

• #817: Migration must construct indexed retained roots with deterministic causal reduction and commitments rather than a full-state replay basis.
• #818: Migration explicitly verifies attachment streaming and must avoid eager legacy asset collection. The common internal stream-only and bounded-collector contract must precede a completed migrator.
• #820: Migration explicitly verifies recursive traversal in scratch and repeats that verification after promotion. Reference encoding alone cannot establish cycle termination, shared-child handling or bounded traversal.
• #821: Migration publishes graph-reference roots and must preserve them through reopen and promotion. Child-root retention, lease and cross-repository authority behavior must exist before this migration is complete.
• #819: The migrator explicitly constructs and verifies typed graph references; preserve this direct requirement even though recursive traversal also depends on it.
• #811: Assign the migration CLI/help/documentation contract to Generate migration guidance from one canonical contract #811. Migrate retained substrates directly into indexed graph roots #822 extends that source for indexed-root migration and derives its operator guidance from it; do not add another independently maintained flag/example contract. Review basis: accepted-ownership.

4. Scope

In: Migration completion requires indexed reduction, attachment streams, executable recursive traversal and retained child-root ownership; these were incompletely modeled in the previous graph.

Out: unrelated domain work and any expansion beyond the issue’s stated observable outcome.

Safe intermediate state: the PR builds and passes relevant checks after its listed prerequisites; existing supported behavior remains usable. Any preparatory step must be independently mergeable.

Domain review, 2026-10-01 — accepted ownership / scope correction:

Consume #811 canonical CLI/documentation contract and extend it for indexed-root migration. Preserve existing graph discovery, confirmation and recovery requirements without copying independently owned help/example semantics.

5. Why now

Maintainer ordering: memory correctness first, supported attachments next, then land eligible PRs. Preserve this card’s existing priority unless a separately recorded scope decision changes it.

6. Risks

Main risk: implementing the historical description instead of the current runtime contract. Preserve compatibility, causal/ownership invariants and bounded behavior relevant to migration.

7. Definition of Done

The issue-specific acceptance checks pass, relevant validation evidence is attached, and the issue links the coherent PR and resulting mainline integration commit. No open item is hidden in a later repair PR.

8. Stakeholders

James Ross: maintainer, assignee and acceptance owner. git-warp contributors and consumers rely on this domain’s supported contract and reproducible evidence.

9. Related Issues

• Blocks completion of #823: Release evidence must exercise the real streaming migration/reopen/append paths, including a disposable retained repository.

Historical paths, counts, release names and shell examples in source material are evidence to reconcile, not authority to restore retired documentation or run host tests.