Goalpost: streaming indexed recursive WARP without full-state materialization #824

1. Background Context

Source: git-warp #824. Audited against main 94b40dac64034cd8caab9bb05efe14a0c22bd735. Template: nine-section-card; work type: type:goalpost.

Current scope and disposition: Tracking container for indexed streaming/recursive slices. Its checklist is delivery coverage, not one extra implementation PR or evidence every child must wait for every other child.

Reported context (historical names and counts require the current disposition above):
Plan: https://github.com/git-stunts/git-warp/blob/plan/streaming-recursive-materialization/docs/plans/streaming-indexed-recursive-warp.md
Design PR: #813

Discussion evidence, including corrections:

• Comment ABUXHZ3A
• Comment ABYcThhQ

Planning role: tracking container; not an additional executable PR.

Additional source requirements/context:

Hill:
Make every git-warp operation independent of complete process-resident graph state. Roaring indexes select stable graph-local ordinals, bounded address pages resolve those ordinals to immutable CAS payloads, attachments remain streams, and graph-valued attachments recursively traverse other WARP graph bases with cycle safety, budgets, backpressure, and receipts.

Why this supersedes the immediate release roadmap:
A real Think migration completed scratch rewrite and verification, promoted prepared refs, and then failed promoted-runtime verification because a 23,995,927-byte logical replay basis reached the ordinary 5 MiB CBOR decoder ceiling. Rollback succeeded and recovery refs were retained.

The failure proves that the v19 bounded-reading work left a whole-state escape hatch in retained resume, hashing, checkpoint, comparison, BTR, migration, and attachment paths. Progress UI and migration documentation cannot make that substrate safe.

Freeze v19.0.1 publication while this goalpost is open. The existing v20.0.0 milestone already owns the streaming/materialization/traversal thesis and becomes the next functional release target.

Child slices:

• Eliminate production full-state materialization contracts #814 — eliminate production full-state materialization contracts
• Define retained graph manifests and bounded ordinal address pages #815 — define retained graph manifests and bounded ordinal address pages
• Remix Roaring indexes as the retained-state selection plane #816 — remix Roaring indexes as the retained-state selection plane
• Stream causal reduction and replace state hashes with root commitments #817 — stream causal reduction and replace state hashes with root commitments
• Make attachment APIs stream-only and bound all byte collection #818 — make attachment APIs stream-only and bound all byte collection
• Add typed recursive WARP graph attachment references #819 — add typed recursive WARP graph attachment references
• Traverse recursive WARP graphs with cycle-safe bounded receipts #820 — traverse recursive WARP graphs with cycle-safe bounded receipts
• Retain and diagnose recursive graph references across CAS boundaries #821 — retain and diagnose recursive graph references across CAS boundaries
• Migrate retained substrates directly into indexed graph roots #822 — migrate retained substrates directly into indexed graph roots
• Prove bounded memory across global jobs, attachments, migration, and recursion #823 — prove bounded memory across global jobs, attachments, migration, and recursion

Existing issue lineage:
Corrective completion of #626–#635, #734–#742, and #758–#760.

Related open debt: #185, #186, #192, #211, #225, #226, #228, #248, and #260.

Migration UX and guidance retained from #806–#812. #810 is blocked on the retained-state architecture; #811 and #812 remain follow-on work.

Closed predecessor issues remain historical records. This issue does not silently redefine what their original PRs proved.

Non-solution:
Increasing the global CBOR limit, embedding truncated CAS hashes in Roaring integers, or renaming a complete-state object does not satisfy this goalpost.

2. Problem Description

Plan: https://github.com/git-stunts/git-warp/blob/plan/streaming-recursive-materialization/docs/plans/streaming-indexed-recursive-warp.md
Design PR: #813

3. Proposed Solution

Tracking container for indexed streaming/recursive slices. Its checklist is delivery coverage, not one extra implementation PR or evidence every child must wait for every other child.

4. Prerequisites

No unsatisfied open-issue prerequisite established by this review. This is not proof that an unresolved design or external readiness condition is satisfied.

5. Scope

In: Tracking container for indexed streaming/recursive slices. Its checklist is delivery coverage, not one extra implementation PR or evidence every child must wait for every other child.

Out: unrelated domain work and any expansion beyond the issue’s stated observable outcome.

Safe intermediate state: the PR builds and passes relevant checks after its listed prerequisites; existing supported behavior remains usable. Any preparatory step must be independently mergeable.

This container owns coverage and reconciliation; child executable cards own implementation. Do not create duplicate PRs for already-owned outcomes.

6. Acceptance Criteria

• No production port, adapter, service, command, or proof accepts or returns a complete graph state.
• Full-state codecs, replay-basis storage, eager attachment APIs, and unbounded asset collection are deleted.
• Targeted reads and patches use Roaring selection plus bounded address/payload pages.
• Checkpoint, resume, comparison, BTR, export, migration, and diagnostics stream or use bounded external workspaces.
• State identity is a deterministic commitment over retained roots.
• Typed pinned/live graph attachments recursively traverse with cycle, sharing, budget, capability, redaction, and receipt semantics.
• Recursive retention and doctor are cycle-safe and cross-repository authority remains explicit.
• Generated oversized state and attachment fixtures pass under reviewed heap/RSS caps without checking large repositories into Git.
• A disposable copy of the real Think repository passes migration, reopen, append, readings, attachment streaming, recursive traversal, receipts, and doctor.
• Migration discovery, one-pass confirmation, progress, disk guidance, Wesley guidance, and API diagrams from Make v18-to-v19 migration discover graphs before confirmation #806–Define and satisfy the v19 migration guide requirements #810 are complete against the new representation.
• Merged-main release evidence is current and green.

7. Definition of Done

Every required outcome maps to a merged executable child with evidence, or an explicit accepted deferral/removal. Reconcile stale claims before closing this container.

8. Test Plan

Golden: Reopen the same retained coordinate and compare roots/readings without replaying covered history.

Edges: Missing/wrong-kind roots, page limits, changed frontiers and alternate shard geometry.

Known failure modes: Refuse corrupt or unavailable bases; cancellation leaves retained roots valid.

Fuzz and stress: Generated graph/page growth under a fixed Docker heap; compare deterministic semantic results.

All tests and benchmarks execute in COPY-based Docker containers without host repository or Git-directory mounts. This planning audit does not claim those checks were run.

9. Stakeholders

James Ross: maintainer, assignee and acceptance owner. git-warp contributors and consumers rely on this domain’s supported contract and reproducible evidence.

Related work:
No additional downstream blocker is established. Shared domain membership alone is not a prerequisite.

Historical paths, counts, release names and shell examples in source material are evidence to reconcile, not authority to restore retired documentation or run host tests.