Stage a component build with activate:false and deploy it later by deployment_id

[dawsontoth]

deploy_component gains two optional fields. activate: false builds, installs and certifies a release and then stops, leaving a dormant artifact under .deploy-staging; deployment_id makes that exact artifact live later, resolving, fetching and installing nothing. The work that takes time and can fail happens when the operator chooses, and the cut-over itself is two directory renames. Both calls replicate as ordinary deploys, so every node stages and every node activates — no cluster barrier, and #2301's stage/activate protocol stays retired.

flowchart TB
  REQ["deploy_component"]
  REQ -->|"package / payload"| DEP["mode = deploy<br/>publish root config, build, certify"]
  REQ -->|"package / payload<br/>+ activate: false"| STG["mode = stage (new)<br/>same build, but no file: directory sources<br/>and no links that leave the tree"]
  REQ -->|"deployment_id"| ACT["mode = activate (new)<br/>verify the artifact and its descriptor<br/>nothing is resolved, fetched or installed"]

  STG --> CMP2["write .artifact.json, then .complete"]
  CMP2 --> DORM(["dormant artifact, nothing swapped<br/>.deploy-staging/&lt;deployment_id&gt;/&lt;component&gt;<br/>returns status staged + deployment_id"])
  DORM -.->|"later, surviving restarts"| ACT

  DEP --> JRN
  ACT --> JRN

  subgraph SWAP["swap"]
    JRN["write .activation.json"] --> B1["1. live moved to .deploy-aside"]
    B1 --> PUB1["publish root config (activation only)"]
    PUB1 --> B2["2. candidate renamed to live — commit point"]
  end

  B2 --> DONE["returns status success + deployment_id<br/>(an activation's row also records activated_from)"]

  classDef added stroke:#2da44e,stroke-width:3px
  class STG,ACT,CMP2,DORM,PUB1 added

Loading

Green marks what this PR adds; everything else is the path a deploy already took.

The mode is chosen from the request, validated so an activation carries no build inputs, and the activate path branches around every payload, credential and install step — without that branch a replicated activation looks exactly like a blob-backed deploy and every peer waits out deployment_timeout for a payload its row never had.

A staged artifact outlives an ordinary deploy of the same component, so staging is also how a release becomes revertible: deploy V1, stage V2, deploy V3, and activating V2's id later puts V2 back. What cannot be recovered is a version that was only ever deployed — its artifact is consumed by the swap and the tree it displaced is swept — which is what #2315 step 5 is for.

Every refusal on these paths carries a status code rather than reaching the caller as a 500, through the same ClientError the rest of the deployment operations use: 400 for a component that cannot be staged at all (a file: directory source), and 409 for a conflict with what is on disk — a deployment id already taken, an artifact whose descriptor this build cannot activate, one whose tree stopped being what was certified, an incomplete build, an unsettled verdict, an in-flight journal, or another component's artifact. 404 now means one thing only: nothing on this node answers to that id, so a caller can tell "never existed, or already used" from "present, but not activatable".

This is step 6 of #2315. Step 4 bounded how many unactivated staged builds accumulate; nothing on main produced one, so its integration test had to plant them. This is the producer.

Four things make the delay safe, and each exists only because an immediate deploy did not need it. The artifact directory is named by the public deployment id while the deploy-lifecycle token stays per-invocation, because DeployLifecycle de-duplicates starts by the announced id and releases watcher suppression on the first matching end — two overlapping activations of one artifact sharing that id would count as one owner. A mandatory versioned .artifact.json is written before .complete, carrying the root-config entry the build would have published (explicitly null for a payload deploy), installationIsOpaque, and the admitted isolation intent; an optional record could not tell a payload build from a package build whose record was lost. Claiming an id is exclusive, because a public id can be repeated where a minted UUID could not. Claiming an id publishes ownership as part of the claim rather than at certification, and an unattributed directory is refused, never reclaimed: a build can hold an empty directory for minutes while it resolves and packs, so an empty readdir cannot tell a claim in flight from an abandoned one. And staging owns its bytes: a file: directory source is refused, and so is any symlink whose walk leaves the built tree at any prefix — certification follows no links, and the post-swap relocation repair runs past the commit point and can only warn.

Certification moved out of the swap primitive into prepareApplication, so a delayed activation does not re-walk and re-fsync a node_modules it certified at build time while holding the preparation lock. An activation that fails before it commits returns its artifact to dormant — journal removed, its parent synced, and the published root-config entry taken back — because otherwise the next preparation reads live-plus-candidate-plus-journal as an abandoned activation and deletes the artifact, or rolls a first deploy forward unasked.

For the human reviewer

The planning gate never cleared on one axis, and it is the first thing to rule on.

1. Mixed-version clusters: detection, not prevention — settled, not open. A stage replicates as an ordinary deploy_component, so a peer running a pre-5.3 build ignores activate: false and deploys the release immediately. The origin reports it afterwards from a staged: true marker each node returns; nothing in core or in harper-pro's replicator carries a peer version or capability, so it cannot refuse in advance. The reviewer's alternative — replicate the stage as a distinct operation an old peer rejects as unknown — was declined by @dawsontoth: Harper clusters are kept at consistent versions, the UI will steer operators away from the mixed case, the docs will state the prerequisite, and adding a second wire path to cover a case that does not occur in practice would encumber the 99.9% path with more code that can break. The planning gate's better-alternative-exists verdict on this axis is recorded as overruled by the owner, not carried as open. Reversing it later is a wire-format change, not an API change: the two request fields stay as they are.

2. Where the root-config publish sits, and the window that remains. A package artifact's entry is published between B1 (the live tree moved aside) and the commit rename — the only point where the on-disk state recovery would find rolls forward to the certified artifact. Publishing any earlier, journal or no journal, leaves live present and the candidate present with no rollback record, which settlement reads as an activation that never started: it deletes the artifact and the next boot re-resolves that package from the registry instead. Three review rounds landed on this from different angles before it was right, so it is the first thing to check if you disagree. The window that remains is the inverse — a crash after the roll-forward state exists but before the publish leaves the certified artifact live under the previous release's config, because rollForward() publishes nothing. That includes its isolation intent, which is a containment boundary rather than a version string: a component staged to run isolated comes back non-isolated after an ordinary crash, with nothing in the operation reporting it. Closing it needs config to be an effect of the journal, which is step 3 — and that, or gating delayed activation to payload artifacts until step 3 lands, is the open call on @kriszyp's thread.

3. A component with absolute or escaping internal symlinks cannot be staged. Staging refuses any link whose walk leaves the built tree at any prefix, including absolute ones under node_modules — where npm writes them on Windows for file:/workspace dependencies. The alternative was to trust repairRelocatedDependencyLinks, but that runs past the commit point and only warns (it logs and continues on a failed re-point, and skips a whole subtree on EACCES/EMFILE), so a component could go live holding a link to a path that no longer exists while the operation reported success. The cost is that such a component deploys immediately but cannot be staged until its links are relative; the error says so. The same rule is re-run at activation rather than trusted from .complete, which is a durability marker over the bytes and not a seal on them — a dormant artifact can be edited while it waits.

4. Staged artifacts share deployment_stagingRetention_maxCount with incidental dormant builds, and only the in-flight request is pinned — so an operator's staged release can be aged out by unrelated activity. One filter in pruneDormantBuilds if you want them to have their own lifetime.

5. A dormant artifact's contents are not sealed, only its links are re-checked. Activation re-runs the link rule and the load validation before the swap, but validateComponentLoads is a no-op on the main thread and the operations API deploys there, so an artifact whose files were edited while it sat dormant still activates. Detecting that means binding .complete to a manifest or digest of the tree — a whole-tree hash at stage time and again at activation, on a path where that cost is affordable. @kriszyp asked for it; I did the link and load re-checks he named as the minimum and left the manifest out, because what it should cover (every file? the installed tree? the descriptor too?) and how it interacts with repairRelocatedDependencyLinks rewriting links past the commit point are design questions, not a patch.

6. A deployment id names one artifact only while that artifact exists. Activation consumes it (the swap is a rename) and retention can prune it, after which the id is free to name different bytes. Durable identity after removal would need a tombstone store with its own retention and recovery. Cheap to add later; impossible to remove once operators script against the stronger promise.

7. A redelivered stage is refused, not answered idempotently. A replayed replication hits the .complete refusal and that peer records a failed deploy for an artifact that is staged and usable. Idempotent success was available and is what the reviewer preferred. I kept the refusal because _deploymentId is operator-forgeable, and answering "success" for bytes that were never staged is the worse failure. Backward-compatible to change later.

8. staged is a terminal status on its own row, and activation gets a second row linked by activated_from, rather than one row transitioning staged→success. This is what makes delete_deployment_payload and SSE tails behave, but it is the shape operators will script against, and it is hard to reverse after release.

9. The config undo is best-effort, and for an existing component a failed undo strands config. addConfig does not fsync and there is no publication lock, so a failed activation restores the previous entry without durability guarantees — that is Staged component deploys: land build-aside-then-swap as a sequence of small PRs #2315 step 3's work, deliberately not started here. It takes back only what it published, re-reading from disk first because set_configuration writes the file without refreshing this process's config object, so a change acknowledged while the swap was retrying is not overwritten by a stale snapshot. That narrows the window rather than closing it. An undo that fails keeps the journal only for a first-ever deploy — the one shape recovery can still roll forward from, because the live path is absent. For a component that already had a tree, compensation has just put that tree back and taken the rollback record with it, so the next settle returns the artifact to dormant whatever the journal says: config names the staged release, the previous one is live, and an operator has to resolve it.

10. A partially successful clustered activation cannot converge by retrying the same id. The origin activates locally before peer fanout, and the swap consumes its staged directory — so if this node activates and a peer fails, retrying the same deployment_id is refused here (nothing is staged under it any more) before the retry ever reaches the peer that still holds the dormant artifact. The operator's route back is a fresh deploy, not a retry. Making the retry converge means either deferring the local swap until peers confirm, which gives up "every node activates the same way it deploys", or answering a deployment_id this node already activated as success, which needs the deployment row to be authoritative for what is live. Neither is small; both are cheaper before release than after. A local run turned up the same gap without a cluster: crash a first-ever activation after its journal lands, and recovery rolls the artifact forward at the next boot — the component comes up live from exactly those bytes, and the operator's retry of that deployment_id then answers 404, because nothing is staged under it any more. Their activation succeeded; what they are told is that it never existed.

Smaller carried items: concurrent deploys of different components can over-admit the global isolated-worker budget (adjudicated pre-existing — the same race beforePrepare already had); activated_from is registered for fresh installs only, matching every other attribute on hdb_deployment; an activation re-walks the artifact tree under the preparation lock, measured by a reviewer at 11,615 directories and 148 links in 380ms warm; every refusal of a deployment_id is a 404, so a caller cannot tell "never existed" from "already used"; and a crash inside the claim window — between the exclusive mkdir and the sidecar write, microseconds with only an lstat between them — burns that deployment id and leaves an empty directory nothing reclaims, since refusing an unattributed directory is what removed the only self-heal that path had. A claim that cannot record who owns it now takes back the directory it made, so only a process death in that window — not an ordinary write failure — burns the id. Closing what remains needs claims serialized on the staging root; a staleness threshold would contradict this code's rule against wall-clock verdicts.

Verification

Unit tree — run per directory rather than as one test:unit:main, because unitTests/components/applicationSpawn.test.js wedges on this machine at a process-group termination test (it wedges identically at the merge base, and the diff touches zero lines of that code), and .mocharc.json sets timeout: 0, so one hang swallows the whole run. Excluding that file: resources 2500 pass, components 1662 pass / 5 fail, server 960 pass / 10 fail, utility 759 pass, config 314 pass / 1 fail, validation 260 pass / 1 fail, bin 247 pass, sqlTranslator 67, upgrade 19.

Every one of those failures is pre-existing. The same three directories were run at the merge base (3ece37294) for comparison: server and config have identical failure sets, and components fails 16 at base against 5 here — every branch failure is also a base failure, none the other way. The named failures are RuntimeModuleTracker, nonInteractiveSpawn git credential scoping, uWS UDS adapter, process-group reclaim ordering, package-specifier resolution and RootConfigWatcher; two are explained outright by this environment — validation's is getDomainSocketPathLengthWarning, which this worktree's own path length trips, and dataLayer's suite fails to load at all without HARPER_STORAGE_ENGINE=lmdb.

Live server, not only tests. A Harper 5.3.0-alpha.1 instance on an isolated data root, driven through the operations API, with three builds of a component serving {"build":"V1|V2|V3"} over REST plus a real component (three as a dependency). A normal deploy returned its deployment_id; activate: false left V1 serving and wrote .artifact.json/.complete/.component beside the tree; the artifact survived a full Harper restart; activating it made V2 live carrying a sentinel file planted in the dormant tree, which is what proves the staged bytes were swapped rather than rebuilt; restart: true on the activation brought the server back serving V3. On the real component the split was 1.7s to stage, 0.1s to activate, with a sentinel written inside the stage-time node_modules/three present in the live tree.

Refusals were exercised the same way: a consumed id, an unknown id, another component's artifact (preserved), restart with activate: false, deployment_id alongside package/payload, a non-boolean activate, a non-UUID id, an id already holding a completed build, an id another component owns, an empty unattributed claim directory (refused and left untouched — the concurrent-claim fix), an artifact whose tree was removed, and five malformed descriptors including one whose isolation intent disagreed with what it publishes. An artifact tampered with after staging — ln -s /etc planted in the dormant tree — was refused and preserved. Retention settled at maxCount + the pinned in-flight build and stayed there across further stages; drop_component reclaimed that component's artifacts and left its neighbour's alone; a stage whose package could not resolve left no residue; delete_deployment_payload on a staged row succeeded and the artifact still activated afterwards, which is the point of certifying at stage time. SSE phases are symmetric for both calls (prepare/start → prepare/done → replicate → staged|success).

Recovery was driven on a real boot by planting crash states and restarting: an interrupted activation returned its artifact to dormant ("Returned the staged build … to dormant after an activation that never moved its live tree aside"), and a described artifact carrying a stale .unsettled with no journal had the verdict cleared ("Cleared a stale unsettled verdict … its activation was already settled"). Both artifacts activated successfully afterwards. Two rounds of this exercise are what produced the status codes above; nothing else it found needed a code change.

Docs: HarperFast/documentation#670.

Complexity: complicated

_{Review-Coverage: authored=claude; ran=gemini,codex,cursor-grok,cursor-composer; adjudicated=domain; rounds=20; full=8 @ e7e1a91}

_{Human-Review-Need: 4 (decisions: exported-for-testability, caught-failures-reclaim-crashes-do-not, one-404-class-for-missing-and-consumed, isolation-loss-documented-not-closed, artifact-consumed-by-rename, failed-clear-fails-closed, fail-open-peer-confirmation, dangling-internal-links-fail-closed) @ e7e1a91}

[gemini-code-assist]

Code Review

This pull request implements the ability to stage a component build and activate it later (deploy_component { activate: false }). It introduces a mandatory, versioned .artifact.json descriptor to record build decisions, enforces exclusive claims on deployment IDs, and restricts external or absolute symlinks within staged trees to ensure integrity. The activation process is updated to swap the pre-certified artifact into the live path without rebuilding or reinstalling. A review comment suggests optimizing the staging process by caching the result of readlink to avoid duplicate filesystem reads.

[claude]

1 blocker, unchanged, still carried over from kriszyp's open threads (r4009459237, r4019827364): root config — isolation included — still isn't a journaled, durable effect, so a crash after roll-forward, or a failed config-undo for an existing component, can leave config naming a release that isn't live. Both threads remain unresolved; this push doesn't touch that code path, and only updates the test fixture to assert the divergence explicitly rather than close it.

This push's two targeted fixes are correct and each has a new test that fails against the prior commit: link-target splitting now treats \ as a separator only on Windows, so a POSIX file literally named ..\asset no longer false-triggers the escape guard; and publishClaimOwnership now removes the directory it just created (and only that one) if it can't record ownership, so a write failure between the exclusive mkdir and the .component write no longer permanently burns the deployment id. No new blockers or suggestions from this push.

[kriszyp]

_{🤖 Reviewed with Codex}

[kriszyp]

_{🤖 Reviewed with Codex}

[kriszyp]

Excellent!
_{🤖 Reviewed with Codex}