RFC-029: Write-protocol lifetime ownership and recovery barriers

[ragnorc]

What & why

This RFC proposes three coordinated changes to close structural defects in write-protocol cancellation safety and recovery residual handling:

1. W1 — Engine-owned write-protocol lifetime: Shield armed write protocols from caller cancellation by spawning protocol execution as a detached task. Caller disconnection means "stopped waiting for the result", never "abandoned the protocol".

2. W2 — Full recovery at the write-entry barrier: Escalate rollback-class recovery residuals to RecoveryMode::Full under exclusive admission at the write-entry heal, instead of parking them for process restart. Resolves RecoveryRequired as a transient condition.

3. W3 — Boot supervision: Replace silent graph exclusion on open failure with an observable quarantined registry state and a capped-backoff re-open loop, making recovery from transient substrate errors automatic.

Together these establish the invariant: Caller fate is not a protocol participant. An armed graph-write protocol completes or compensates under engine ownership regardless of caller cancellation, and resolving any recovery residual requires only exclusive gate acquisition — never process restart.

Backing issue / RFC

• Implements / is an accepted RFC: docs/rfcs/0029-write-lifetime-and-recovery-barriers.md

Motivation

A validated production incident showed the chain of failures: client timeout interrupted a multi-statement delete mid-protocol, leaving an Armed sidecar; the live-handle heal is roll-forward-only and parks rollback-class residuals; the next write hit a stale manifest version; boot recovery hit a transient S3 error and silently quarantined the graph until redeploy. Every link was validated against current code (Lance 9.0.0-rc.1 at pinned rev).

The RFC documents this incident in detail (§1) and validates the mechanism at each step. The long-run liability: cancel-safety is maintained by audit today, and every write-path change must re-reason about cancellation at every await. W1 replaces that with a structural property.

Design summary

W1 (Stage 1 — server boundary): Apply the existing spawn-and-clone idiom from server_export to every _as writer handler. The HTTP response contract is unchanged; on client disconnect, the spawned task runs the protocol to terminal state (success or error) while the request future drops.

W1 (Stage 2 — engine level): Move the shield into the engine via Arc<Self> receivers or an Arc<OmnigraphInner> split (shape TBD in review), so the property holds for embedded SDK callers.

W2: When the write-entry heal encounters a rollback-class sidecar, escalate it to RecoveryMode::Full under exclusive admission + existing schema → branch → table gates, instead of parking it. The heal re-reads the sidecar under those gates (existing behavior), dispatches with Full mode (new), and the triggering write proceeds from a fresh capture.

W3: A graph whose open fails enters Quarantined { since, attempts, last_error, retry_at } in the registry. One supervision task per quarantined graph retries the full open_single_graph with capped exponential backoff (proposed: 5 s initial, ×2, 10 min cap). GET /graphs gains a per-graph status field (serving | quarantined) with quarantine detail.

Safety argument

• W1: The spawned task holds the Arc and the admission permit; caller cancellation releases neither. The per-actor admission bound is preserved (admission bounds concurrent work, and the work genuinely continues).
• W2: Restore needs exclusive authority over affected tables while restore-and-publish completes. In-process writers are excluded by exclusive admission + table gates (stronger than open-time). In-process readers are safe by construction: reads pin manifest-published versions; restore appends a new version; read caches are keyed by version/e_tag. Foreign processes are exactly as (un)protected as at open time (documented single-writer-process boundary unchanged).
• W3: The retry unit is the whole idempotent open; recovery

https://claude.ai/code/session_01EqR7rPVfAT7UMiy85pG9b5

---

Note

Low Risk
Documentation-only RFC addition; proposed behavioral changes are described for future PRs, not shipped here.

Overview
Adds draft RFC-029 (docs/rfcs/0029-write-lifetime-and-recovery-barriers.md) — design-only; no runtime changes in this PR.

The RFC responds to a validated production chain (client timeout → Armed sidecar → writes wedged until restart → boot quarantine on transient S3). It keeps RFC-022’s commit protocol and proposes three landable workstreams:

W1 — Cancellation shielding: Spawn write-protocol work so disconnect means “stopped waiting,” not “aborted mid-arm.” Stage 1: tokio::spawn at server _as handlers (same pattern as server_export). Stage 2: engine-level 'static execution for SDK callers.

W2 — Write-entry full recovery: Escalate rollback-class sidecars from roll-forward-only live heal to RecoveryMode::Full under exclusive admission plus existing gates, so RecoveryRequired clears on the next write instead of requiring restart.

W3 — Boot supervision: Registry quarantined state, capped-backoff re-open loop, and additive GET /graphs status (serving | quarantined).

Proposes invariant “caller fate is not a protocol participant,” documents contract/behavior ledger (503 lifetime, admission on disconnect, quarantine visibility), test-first evidence plan, and a four-PR rollout order.

^{Reviewed by Cursor Bugbot for commit bb55888. Bugbot is set up for automated code reviews on this repo. Configure here.}

Greptile Summary

This PR defines RFC-029 for cancellation-safe writes and automatic recovery. The main changes are:

• Engine-owned execution for armed write protocols.
• Full recovery under the live write-entry barrier.
• Observable quarantine state with supervised reopen attempts.
• New tests, documentation, and rollout stages for the three changes.

Confidence Score: 4/5

The RFC's admission ownership and quarantine lifecycle need fixes before merging.

• The Stage 1 example lets request cancellation release admission while detached work continues.
• An all-quarantined startup exits before retries can begin.
• Retry tasks have no generation or cancellation rule for stale registry state.

docs/rfcs/0029-write-lifetime-and-recovery-barriers.md

Important Files Changed

Filename	Overview
docs/rfcs/0029-write-lifetime-and-recovery-barriers.md	Defines write lifetime ownership, live full recovery, and boot supervision, but leaves admission transfer and quarantine lifecycle gaps.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    C[HTTP or SDK caller] --> A[Admission]
    A --> T[Engine-owned write task]
    T --> P[Arm recovery sidecar]
    P --> E[Apply table effects]
    E --> M[Publish manifest]
    M --> D[Delete sidecar]
    C -. disconnect .-> X[Caller stops waiting]
    X -. task and admission continue .-> T
    R[Next write] --> H[Exclusive write-entry heal]
    H --> F[Full recovery]
    F --> T
    B[Server boot] --> O[Open graph]
    O -->|success| S[Serving]
    O -->|failure| Q[Quarantined]
    Q -->|backoff retry| O
    Q --> G[GET /graphs status]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    C[HTTP or SDK caller] --> A[Admission]
    A --> T[Engine-owned write task]
    T --> P[Arm recovery sidecar]
    P --> E[Apply table effects]
    E --> M[Publish manifest]
    M --> D[Delete sidecar]
    C -. disconnect .-> X[Caller stops waiting]
    X -. task and admission continue .-> T
    R[Next write] --> H[Exclusive write-entry heal]
    H --> F[Full recovery]
    F --> T
    B[Server boot] --> O[Open graph]
    O -->|success| S[Serving]
    O -->|failure| Q[Quarantined]
    Q -->|backoff retry| O
    Q --> G[GET /graphs status]

Loading

_{Reviews (1): Last reviewed commit: "Add RFC-029: write-protocol lifetime own..." | Re-trigger Greptile}

Greptile also left 3 inline comments on this PR.

Context used:

• Context used - AGENTS.md (source)
• Context used - CLAUDE.md (source)

[greptile-apps]

Detached Work Releases Admission

The example captures only the engine call, while the current handler owns its admission guard outside this spawned future. When the client disconnects, the request drops that guard even though the write continues, so repeated disconnects can exceed the actor's concurrency limit and contradict the admission contract stated below.

Context Used: AGENTS.md (source)

[greptile-apps]

All-Quarantined Boot Cannot Recover

When every configured graph fails its initial open, the current startup path aborts before the server and supervision loop can run. A single-graph deployment with the transient S3 failure described by this RFC therefore still exits instead of entering quarantined and retrying; default subset mode must explicitly permit zero serving graphs when quarantined entries exist.

Context Used: AGENTS.md (source)

[greptile-apps]

Retry Tasks Lack Generation Fencing

The proposed per-graph retry loop has no cancellation or generation rule for a graph removed or reconfigured while it is quarantined. If runtime registry replacement is added as specified, an older delayed retry can keep running or publish a handle from stale startup configuration after a newer registry state exists, causing an obsolete graph or URI to become serving again.

Context Used: AGENTS.md (source)

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bb55888a10

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Move admission guard into the spawned task

In the current run_mutate path the per-actor permit is acquired before this call (handlers.rs:686-689). If Stage 1 follows this snippet, the spawned task owns only the engine and inputs; _admission stays in the handler and is dropped when the client disconnects, releasing the slot while the write continues. That violates this RFC's admission invariant and lets disconnecting clients bypass per-actor concurrency limits, so the spawned future needs to own the guard or the whole admitted operation must be spawned.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Move engine-cancellation test to Stage 2

PR-1 is scoped to the HTTP server-boundary shield, but requiring both W1 tests here also includes the failpoint test above that drops the engine caller future directly. That case remains cancellable until PR-4's engine-level Arc/inner-handle restructure, so PR-1 cannot turn its required red test green. Split the plan so PR-1 only tests a dropped HTTP client and reserve the direct future-drop test for Stage 2.

Useful? React with 👍 / 👎.