feat(local-container): fixed idempotent lease IDs (#1493)

* feat(local-container): fixed idempotent lease IDs

The local-container provider now advertises SupportsRequestedLeaseID and
routes requested lease IDs through the durable AcquireFixedLease helper:
existing containers are adopted only after verifying lease, slug, container
name, runtime, image, daemon identity, and intent fingerprint; recovery
mirrors the aws fixed-lease fail-closed semantics (live containers resume,
unresolved attempts, missing or dead containers, and released tombstones
return lease_id_conflict). The intent fingerprint covers runtime and daemon
scope, image and checkpoint identity, resources, mounts, network and socket
access, desktop settings, architecture, slug, retention, lifecycle timing,
and the SSH public key, so a materially different configuration conflicts
instead of silently reusing a lease.

* docs(local-container): document fixed idempotent lease IDs

* fix(local-container): preserve downgrade-safe fixed lease claims

* fix(local-container): exclude released fixed claims from slug routing

* fix(local-container): preserve fixed lease duration precision

* test(local-container): cover fixed checkpoint fork acquisition

* fix(local-container): bind fixed lease intent to normalized pond

* fix(local-container): bind stopped fixed lease recovery before conflict

Author: steipete

CHANGELOG.md

@@ -2,6 +2,10 @@
 
 ## 0.46.1 - Unreleased
 
+### Added
+
+- Added fixed idempotent `--lease-id` replay to local-container warmups, with exact container-intent matching and single-use released IDs.
+
 ### Fixed
 
 - Redacted coordinator URL credentials, query parameters, and fragments from run-context portal and logs links.

docs/commands/warmup.md

@@ -15,6 +15,7 @@ crabbox warmup --browser
 crabbox warmup --tailscale
 crabbox warmup --slug update-flow-smoke
 crabbox warmup --provider aws --lease-id cbx_abcdef123456 --slug update-flow-smoke
+crabbox warmup --provider local-container --lease-id cbx_abcdef123456 --slug update-flow-smoke
 crabbox warmup --pond alpha --slug db
 crabbox warmup --provider aws --target windows --desktop
 crabbox warmup --provider azure --target windows
@@ -50,13 +51,14 @@ Warmup requires an explicit provider selection from `--provider`,
 recorded lease route. With no selection it exits before provider initialization
 and points to `crabbox providers recommend`.
 
-For `local-container`, the default `--keep=true` also covers SSH readiness
-failure: once Docker has returned an exact container identity, Crabbox persists
-a scoped `provisioning` claim before inspecting the container or waiting for
-SSH. Cancellation or timeout retains that pending lease and prints exact
-inspect, reclaim, and cleanup commands.
-`warmup --keep=false` rolls the container and all per-lease local state back
-instead.
+For ordinary `local-container` warmups without `--lease-id`, the default
+`--keep=true` also covers SSH readiness failure: once Docker has returned an
+exact container identity, Crabbox persists a scoped `provisioning` claim before
+inspecting the container or waiting for SSH. Cancellation or timeout retains
+that pending lease and prints exact inspect, reclaim, and cleanup commands.
+`warmup --keep=false` rolls an ordinary container and all per-lease local state
+back instead. Fixed-ID local-container warmups retain their durable create
+attempt so an interrupted operation can be safely replayed.
 
 ## Lifetime: TTL and idle timeout
 
@@ -70,28 +72,29 @@ instead.
 it and may append a short suffix if an active lease already uses that slug.
 
 `--lease-id cbx_<12 lowercase hex>` is the automation idempotency contract for
-providers that explicitly support fixed identities. Direct AWS and managed
-coordinator leases accept it. Replaying the same normalized create intent
+providers that explicitly support fixed identities. Direct AWS, Machine0, and
+local-container leases, managed coordinator leases, and explicitly capable
+external providers accept it. Replaying the same normalized create intent
 returns or joins the same lease, including after the creating process loses its
 response. Reusing the ID with a different provider, slug request, SSH key,
-machine shape, capabilities, lifetime, or other immutable create input fails
-with `lease_id_conflict` before another provider create. Slugs remain display
-aliases and are never used as the idempotency key.
+machine or container shape, capabilities, lifetime, or other immutable create
+input fails with `lease_id_conflict` before another provider create. Slugs
+remain display aliases and are never used as the idempotency key.
 
 Coordinator-backed fixed-ID creation uses a versioned `PUT /v1/leases/<id>`
 route. An older coordinator therefore rejects the request before provisioning;
 the CLI never falls back to slug lookup or legacy create behavior. After an
 ambiguous fixed create response, the CLI repeats that exact PUT to atomically
 confirm the same intent before it may poll lease status with GET.
 
-A fixed lease ID is single-use. If direct AWS acquisition completed and its
-bound instance later disappears, replay fails closed instead of relying on EC2
-client-token retention. Successful stop and missing-resource cleanup replace
-the live local claim with a compact terminal tombstone, so the ID remains
-rejected after release. Use a new operation ID for every later lease.
-If an AWS launch attempt was durably recorded but its instance is not yet
-visible, replay fails closed without resubmitting it; retry later to adopt the
-resource after provider inventory converges.
+A fixed lease ID is single-use. Direct AWS, Machine0, and local-container
+acquisitions fail closed if their bound resource later disappears. Successful
+stop and missing-resource cleanup replace the live local claim with a compact
+terminal tombstone, so the ID remains rejected after release. Use a new
+operation ID for every later lease. If an AWS launch or local-container create
+attempt was durably recorded but its resource is not yet visible, replay fails
+closed without resubmitting it; retry later to adopt the resource after
+provider inventory converges.
 
 `--pond <name>` tags a new lease into a named pond (stored as a reserved
 provider label); `crabbox list --pond <name>` filters by it. When combined with

docs/features/identifiers.md

@@ -57,20 +57,30 @@ private token and generation never appear in public lease records. Fixed-ID
 to own replay, and caller cancellation never releases them.
 
 Automation may instead supply the canonical ID with `warmup --lease-id`. For
-direct AWS, direct Machine0, and managed coordinator leases, that ID is an
-immutable create identity: an identical semantic replay returns the same lease,
-while intent drift returns `lease_id_conflict`. The coordinator durably stores
-a versioned normalized request hash. Direct AWS durably stores the intent and
-current resolved EC2 attempt in the normal lease claim before `RunInstances`,
-then uses a deterministic regional/zonal client token. Neither path uses the
-slug to decide replay ownership.
+direct AWS, direct Machine0, direct local-container, and managed coordinator
+leases, that ID is an immutable create identity: an identical semantic replay
+returns the same lease, while intent drift returns `lease_id_conflict`. External
+providers also accept requested IDs when their protocol explicitly advertises
+idempotent lease identity support. The coordinator durably stores a versioned
+normalized request hash. Direct AWS durably stores the intent and current
+resolved EC2 attempt in the normal lease claim before `RunInstances`, then uses
+a deterministic regional/zonal client token. No path uses the slug to decide
+replay ownership.
 
 Direct Machine0 binds the intent to its deterministic VM name before creation;
 the durable attempt binds the first visible match to its Machine0 resource ID,
 and every later adoption requires that exact recorded ID. Its fixed claims use
 the downgrade-safe `machine0-fixed-v1` marker alongside AWS's `aws-fixed-v1`
 marker.
 
+Direct local-container binds the intent to its runtime and daemon scope,
+normalized container configuration, and deterministic container name before
+creation. Replay adopts only the exact recorded container when its lease, name,
+runtime identity, and intent fingerprint still match; an unresolved attempt or
+missing acquired container fails closed instead of starting another container.
+Its fixed claims use the downgrade-safe `local-container-fixed-v1` marker, so
+older clients cannot mistake them for ordinary local-container claims.
+
 After the direct AWS launch attempt is durable, Crabbox never submits that
 attempt again. An ambiguous replay with no visible tagged instance fails closed;
 a later replay can adopt the one instance after inventory converges only when
@@ -79,11 +89,11 @@ match the persisted attempt exactly. Fixed AWS
 claims use the downgrade-safe local discriminator `aws-fixed-v1`; current
 clients map it to runtime AWS, while older clients skip/refuse it.
 
-Fixed IDs are single-use operation identities. Direct AWS and Machine0 keep a
-compact terminal claim tombstone after successful destroy release or exact
-missing-resource cleanup. Tombstones contain only the ID, slug, provider scope,
-versioned intent hash, timestamps, and terminal state; automatic provider
-cleanup never prunes them.
+Fixed IDs are single-use operation identities. Direct AWS, Machine0, and
+local-container keep a compact terminal claim tombstone after successful
+destroy release or exact missing-resource cleanup. Tombstones contain only the
+ID, slug, provider scope, versioned intent hash, timestamps, and terminal
+state; automatic provider cleanup never prunes them.
 There is no time-based reuse window. Explicitly deleting local Crabbox claim
 state forfeits this replay protection, so automation must instead mint a new
 operation ID.

docs/provider-backends.md

@@ -138,6 +138,22 @@ remote commands itself, use `DelegatedRunBackend`.
 
 Add optional capabilities as small interfaces instead of widening every backend.
 
+Requested fixed lease IDs are optional:
+
+```go
+type IdempotentLeaseIDBackend interface {
+	SupportsRequestedLeaseID() bool
+}
+```
+
+Direct AWS, Machine0, and local-container backends implement this capability;
+coordinator-backed leases support it through the coordinator wrapper. External
+backends support it only when their configured protocol explicitly advertises
+idempotent lease IDs. `crabbox warmup --lease-id` rejects other backends before
+provisioning. Built-in direct adapters reuse `core.AcquireFixedLease` for
+durable intent and replay mechanics while keeping resource creation,
+reconciliation, and identity validation provider-owned.
+
 Cleanup is optional:
 
 ```go
@@ -386,8 +402,9 @@ Acquisition already shares the mechanics that have provider-neutral contracts:
 - `core.AcquireFixedLease`, `core.FixedAcquireOptions`,
   `core.FixedLeaseBinding`, and `core.FixedLeaseKind` already share durable
   fixed-ID intent locking, replay validation, acquired-state commit, and
-  terminal tombstones for AWS and Machine0. Their adapters still own exact
-  create attempts, provider reconciliation, and immutable resource identity.
+  terminal tombstones for AWS, Machine0, and local-container. Their adapters
+  still own exact create attempts, provider reconciliation, and immutable
+  resource identity.
 
 The transaction boundary deliberately remains inside each adapter:
 
@@ -415,9 +432,9 @@ The transaction boundary deliberately remains inside each adapter:
   back failed creation even with `Keep`, so retention cannot be a global rule.
 - **`OnAcquired` placement:** Lume acknowledges an early provisional identity,
   Vast acknowledges after readiness but before its final claim, and fixed-ID
-  Machine0 acknowledges only after durable commit and lock release. Moving the
-  callback changes when controller ownership transfers and which transaction
-  must clean up if acknowledgment fails.
+  Machine0 and local-container acknowledge only after durable commit and lock
+  release. Moving the callback changes when controller ownership transfers and
+  which transaction must clean up if acknowledgment fails.
 - **Security-critical bootstrap ordering:** Hyper-V locks down guest SSH before
   attaching networking; Lume pins authenticated guest identity before accepting
   SSH; Vast probes initial access, installs required tools, and only then proves

docs/providers/local-container.md

@@ -81,6 +81,25 @@ Docker Desktop-specific APIs. Crabbox detects an installed `docker` or `podman`
 CLI and uses that runtime. Set `localContainer.runtime` when you need a specific
 CLI.
 
+### Fixed-ID replay
+
+```sh
+crabbox warmup --provider local-container \
+  --lease-id cbx_abcdef123456 --slug my-app-operation
+```
+
+A fixed lease ID makes warmup replay-safe across process restarts. Crabbox
+persists the runtime identity and normalized container-create intent before
+starting the container, then reuses an existing container only when its lease,
+slug, runtime scope, and intent fingerprint match. Changing the image or other
+container-shaping configuration returns `lease_id_conflict` instead of silently
+reusing the lease. An unresolved create attempt or a missing previously
+acquired container also fails closed without starting a second container.
+
+Stopping a fixed lease preserves a terminal local tombstone, so the same
+operation ID cannot create another container after release. Use a new fixed
+lease ID for each later operation.
+
 ## Configuration
 
 ```yaml

internal/cli/claim.go

@@ -76,6 +76,7 @@ type FixedCreateIntent struct {
 
 const FixedAWSClaimProvider = "aws-fixed-v1"
 const FixedMachine0ClaimProvider = "machine0-fixed-v1"
+const FixedLocalContainerClaimProvider = "local-container-fixed-v1"
 
 const maxLocalClaimInventoryFileBytes int64 = 1 * 1024 * 1024
 
@@ -1303,6 +1304,8 @@ func canonicalClaimProvider(provider string) string {
 		return "aws"
 	case FixedMachine0ClaimProvider:
 		return "machine0"
+	case FixedLocalContainerClaimProvider:
+		return "local-container"
 	case "exec-provider":
 		return "external"
 	}

internal/cli/claim_test.go

@@ -56,6 +56,44 @@ func TestFixedMachine0ClaimProviderCanonicalizes(t *testing.T) {
 	}
 }
 
+func TestFixedLocalContainerClaimProviderCanonicalizesWithoutOverwritingMarker(t *testing.T) {
+	t.Setenv("XDG_STATE_HOME", t.TempDir())
+	const leaseID = "cbx_abcdef123465"
+	err := withDurableLeaseClaimLock(leaseID, func(claim *leaseClaim, _ bool, persist func() error) error {
+		claim.LeaseID = leaseID
+		claim.Slug = "fixed-local-container"
+		claim.Provider = FixedLocalContainerClaimProvider
+		claim.ProviderScope = "runtime:docker/context:default"
+		claim.RepoRoot = "/repo"
+		claim.FixedCreateIntent = &FixedCreateIntent{
+			Version: 1, Fingerprint: strings.Repeat("a", 64), ProviderScope: claim.ProviderScope,
+			Slug: claim.Slug, CreatedAt: time.Now().UTC().Format(time.RFC3339Nano), State: "acquired",
+		}
+		return persist()
+	})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got := canonicalClaimProvider(FixedLocalContainerClaimProvider); got != "local-container" {
+		t.Fatalf("fixed local-container marker canonicalized to %q", got)
+	}
+	resolved, ok, exact, err := resolveLeaseClaimForProviderWithExact(leaseID, "local-container")
+	if err != nil || !ok || !exact || resolved.Provider != FixedLocalContainerClaimProvider {
+		t.Fatalf("resolved=%#v ok=%t exact=%t err=%v", resolved, ok, exact, err)
+	}
+	if err := claimLeaseForRepoProvider(leaseID, "fixed-local-container", "local-container", "/repo", time.Minute, false); err != nil {
+		t.Fatal(err)
+	}
+	after, exists, err := readLeaseClaimWithPresence(leaseID)
+	if err != nil || !exists || after.Provider != FixedLocalContainerClaimProvider {
+		t.Fatalf("runtime local-container claim update overwrote marker: claim=%#v exists=%t err=%v", after, exists, err)
+	}
+	peer := bridgePeerFromClaim(after, TransportNone)
+	if peer.Provider != "local-container" {
+		t.Fatalf("fixed marker displayed as provider %q", peer.Provider)
+	}
+}
+
 func TestClaimEndpointReservationDeadlineStartsAfterClaimLockAcquired(t *testing.T) {
 	t.Setenv("XDG_STATE_HOME", t.TempDir())
 	const leaseID = "cbx_reservation_lock"

internal/cli/pond_test.go

@@ -29,6 +29,9 @@ func TestNormalizePondName(t *testing.T) {
 		if got := normalizePondName(tc.in); got != tc.want {
 			t.Fatalf("normalizePondName(%q)=%q want %q", tc.in, got, tc.want)
 		}
+		if got := NormalizePondName(tc.in); got != tc.want {
+			t.Fatalf("NormalizePondName(%q)=%q want %q", tc.in, got, tc.want)
+		}
 	}
 }
 

internal/cli/provider_exports.go

@@ -580,6 +580,10 @@ func NormalizeLeaseSlug(value string) string {
 	return normalizeLeaseSlug(value)
 }
 
+func NormalizePondName(value string) string {
+	return normalizePondName(value)
+}
+
 func RenderTailscaleHostname(template, leaseID, slug, provider string) string {
 	return renderTailscaleHostname(template, leaseID, slug, provider)
 }