Improved deployment visibility #526

Problem

deploy_component is a synchronous HTTP request that returns a single JSON result only after the full pipeline completes: upload → decompress → install dependencies (npm/pnpm/yarn/custom) → component load validation → replication to peers → worker restart. With non-trivial packages this routinely takes minutes. The CLI shows nothing during that time, and when something fails — network drop, npm error, lock conflict, peer rejection — the user gets one terse error string with no indication of which phase broke, on which node, or how far the cluster got.

We need step-level progress and clear error attribution across every phase, on the origin and each replica.

Plan

Transport: text/event-stream response on deploy_component

When the CLI sends Accept: text/event-stream (and once the request matches an operation that supports it), the operations API returns SSE instead of a single JSON response. The SSE serializer already exists at server/serverHelpers/contentTypes.ts:104 and is selected by Accept in REST handling — this work wires deploy_component (and the relayed peer deploys) to that path.

Event shape (each is an SSE event: line plus a JSON data: payload):

event: phase
data: { phase: "upload"|"extract"|"install"|"load"|"replicate"|"restart",
        status: "start"|"progress"|"done"|"error", node?, ... }

event: upload
data: { bytes, total? }

event: install
data: { manager: "npm"|"pnpm"|"yarn"|"custom", stream: "stdout"|"stderr", line: "..." }

event: replicate
data: { node, phase, status, attempt?, ... }      // relayed from each peer's SSE

event: error
data: { phase, node?, message, code? }

event: done
data: { result, peers: [{ node, status, error? }, ...] }

CLI rendering:

• A byte-based progress bar for upload (origin only; peer uploads happen server-side and surface as replicate phase events).
• A spinner + the latest log line for install per node, with full lines streamed through verbose mode.
• A per-node status grid for replicate: each peer shows its current phase and any retry attempt count.
• A clear, attributed error block (phase + node + message) for any failure, distinguishing application errors (npm install failed) from transport errors that were retried (transient HTTPS errors per Support for deployments larger than 2GB #524).

Why SSE instead of a richer job record

We considered the "create a job and poll get_job for state" pattern. It doesn't fit well here:

• The most useful signal during install is the live npm install stdout/stderr. Persisting every chunk into a job record so it can be polled is wasteful, and polling cadence trades latency against load.
• The CLI is already connected; SSE reuses that single connection. Polling adds round-trips and forces every progress event to be persisted somewhere queryable just so it can be retrieved seconds later.

We still emit a job record so post-hoc inspection works (get_job <id>), but live progress comes through SSE. Phase transitions and final status are mirrored into the job record; per-chunk install output is not (too noisy to persist).

Emitting progress in deployComponent

The deployComponent handler in components/operations.js receives a progress emitter from the SSE response wrapper. Emit points (see implementation map in #524 discussion):

• Upload: byte count tracked on the server-side multipart parser as the file part is consumed.
• Extract: components/Application.ts:217 — emit start/done around the gunzip-maybe + tar-fs pipeline.
• Install: components/Application.ts:255–410 — installApplication() already spawns the package manager (Application.ts:622–645 handles stdout/stderr chunks for debug logging); we forward those chunks as install events instead of (or in addition to) the existing debug logs.
• Load: components/operations.js:405–418 — emit around componentLoader.loadComponent().
• Replicate: emit per-peer events around each direct-HTTPS deploy started by the new relay (see Support for deployments larger than 2GB #524). Each peer's SSE stream is re-emitted on the origin's SSE response, tagged with the peer node id.
• Restart: emit around manageThreads.restartWorkers('http') / rolling-restart job dispatch.

Per-peer relay visibility (with #524)

Origin opens a streaming HTTPS deploy to each peer with Accept: text/event-stream. Each peer's SSE events are re-emitted on the origin's SSE response with the peer node id attached. Transient transport failures that trigger a retry surface as replicate events with an attempt counter so the user sees the retry happening rather than the operation appearing stalled. Final success/failure per peer is included in the terminating done event.

Backward compatibility

If the client doesn't send Accept: text/event-stream, behavior is unchanged: single JSON response after completion. External operations API callers that don't know about SSE continue to work.

Files touched (approximate)

• bin/cliOperations.js — request text/event-stream, render progress/grid (likely a small renderer module under bin/).
• server/operationsServer.ts / server/serverHelpers/serverHandlers.js — wire SSE response when Accept: text/event-stream and operation is deploy_component; thread a progress emitter into the operation handler.
• components/operations.js, components/Application.ts — emit phase + chunk events from each step; forward spawn stdout/stderr as install events.
• Replication relay (see Support for deployments larger than 2GB #524) — re-emit peer SSE events on the origin stream, with retry attempt counts.
• Job records — mirror phase transitions and final per-peer status into the existing job record so get_job is still useful after the fact.

Related

• Support for deployments larger than 2GB #524 — the multipart/HTTPS-relay transport this depends on for both origin upload progress and per-peer replication visibility.