Edge sync (spoke→hub replication): phase 1 manual core

[xe-nvdk]

Tracking issue for edge-to-cloud sync — promised for 26.09.1, currently design-only.

Design: docs/progress/2026-06-04-edge-sync-architecture-converged.md (converged from two independent designs; read that one, it supersedes the two siblings). Re-validated 2026-08-06: all eight §14 reuse targets still exist and the cited primitives are current.

Why

Arc runs at the edge today — a standalone binary with local storage in a rocket, a tractor, a factory cell, a forward operating base — with no first-class way to get data out to a central Arc. Backup is a DR snapshot, not incremental sync. Import re-ingests rows through the Arrow buffer and writes new files with new checksums, which breaks end-to-end integrity and double-counts on retry, so it can't be the receive door.

The field differs from the datacenter in one decisive way: connectivity is the exception, not the rule. The transport must treat "disconnected" as steady state and "connected" as the event.

Shape

Sync immutable parquet files, not rows. Spoke initiates (edges are behind NAT). At-least-once delivery + idempotent receive = exactly-once effect.

Two phases with opposite round-trip economics — the key insight of the merged design:

• Discovery is batched — one POST /api/v1/sync/reconcile returns missing/present/conflicts for the whole backlog. After a long dark period with 5,000 pending files that is one request, not 5,000.
• Transfer is per-file — each file its own request so X-Arc-Sync-Offset byte-resume is mechanically real. A contact window will close mid-file, so resume is the common case.

Licensing

Per enterprise product definition line 27: manual export/import is OSS; the automatic scheduled agent is Enterprise. Phase 1 below is entirely OSS. The auto-agent gate and receive-endpoint registration are independent — a box can be an OSS manual spoke, an OSS manual hub, an Enterprise auto spoke, or an Enterprise auto hub.

Phase 1 breakdown (small PRs)

Each row is one PR. Ordered by dependency; each should be independently reviewable and mergeable.

• 1. Sync ledger (SQLite) — sync_ledger + sync_history tables in the shared DB, keyed UNIQUE(hub_id, path) for multi-hub readiness. bytes_sent on the row as the resume checkpoint. Modeled on internal/tiering/metadata.go. Includes the in_flight → pending revert on startup. Subject to the SQLite Review Checklist — batched deletes, no app mutex across DB I/O, v.SetDefault for every new key.
• 2. SyncTransport interface + local/no-op impl — Reconcile(ctx, hubID, pending) (*ReconcileResult, error) and PutFile(ctx, hubID, f, body, offset) error. Defining this cleanly now is what keeps the S3-relay and sneakernet transports open later without touching the agent or ledger.
• 3. ComputeSyncHMAC + validation — binds spoke_id, hub_id, path, content-hash, timestamp. NUL-delimited, modeled directly on ComputeFetchHMAC (internal/cluster/security/auth.go:123) which already binds a path into the signed payload and has a matching constant-time validator with freshness tolerance.
• 4. Hub receive: POST /api/v1/sync/file — stream body → WriteReader to a temp path, SHA256 tee'd as it streams, atomic-rename into the namespaced final path only on match, then register in the manifest. Verify-before-commit. filereplication/fetch_client.go:190-215 already does exactly this including resume via a caller-supplied prefix hasher.
• 5. Hub receive: POST /api/v1/sync/reconcile — batch delta from the manifest, O(N) lookups, no I/O on parquet bytes. Must stream both request and response — 100k pending entries is ~20MB and neither side may buffer it whole.
• 6. Spoke namespacing + 409 conflict — hub rewrites incoming paths to {spoke_id}/{original}; spoke_id is bound into the HMAC so a spoke cannot write into another's namespace. Same path + different SHA is a 409, never an overwrite — this deliberately overrides Arc's internal last-write-wins (raft/fsm.go), which is right for intra-cluster replication and wrong for cross-edge fan-in.
• 7. Hub registry — sync_spokes table, per-spoke shared secret, POST/GET /api/v1/sync/spokes. Revoking one spoke must not re-key the fleet.
• 8. One-shot sync trigger + status/ledger endpoints — the manual OSS trigger. No scheduler. Shipped in feat(edgesync): spoke agent and manual sync pass (#569) #579 as POST /api/v1/spoke-sync/run (plus /status and /ledger), not /api/v1/sync/run as planned: Fiber's Group().Use() matches by string prefix, not path segment, so an operator group under /api/v1/sync inherited the hub group's body limit and middleware. A sibling prefix removes the coupling; TestEdgeSyncSpoke_PrefixDoesNotCollideWithHubGroups pins the property.
• 9. Air-gap export/import bundle — shipped across four PRs: feat(edgesync): add StateExported to the spoke ledger (#569) #580 (ledger StateExported), feat(edgesync): air-gap bundle export (#569) #581 (export), feat(edgesync): hub-side air-gap bundle import (#569) #582 (import + (spoke_id, bundle_id) dedup + Raft batching at 1000 ops), feat(edgesync): bundle acknowledgment, closing the air-gap loop (#569) #583 (acknowledgment). Endpoints are POST /api/v1/spoke-sync/export, POST /api/v1/bundle-import, and POST /api/v1/spoke-sync/ack — not /api/v1/sync/imports as planned: Fiber's Group().Use() matches by string prefix, so anything under /api/v1/sync inherits the spoke-facing HMAC group's middleware and body limit. Bundles are a signed directory (manifest + entries.jsonl + data/), chosen over an archive because resume is free and the contents are auditable with ls and sha256sum.

Non-negotiables

• Every endpoint gets auth middleware. Receive, reconcile, and the HEAD fallback all carry HMAC validation → license → handler. Even reconcile leaks "what data exists on the hub" if unauthenticated.
• The hub must not assume clusterCoordinator != nil. "The manifest" is Raft-backed in cluster mode and a local store standalone. This is the classic independently-enabled-subsystem trap.
• Never delete edge data fire-and-forget. Post-sync retention is phase 3, off by default, and only after a verified acknowledged round-trip.

Open design question (decide before PR 1)

Row-group granularity. Compacted files can be large, and compaction output is bounded by file count not bytes (#568), so a daily partition can produce a multi-hundred-MB file. Over a contested link a whole-file unit means nothing is queryable until the entire file lands, and newest-first ordering loses meaning within one big file.

Parquet is internally chunked at 122,880 rows per row group (set explicitly in the compaction COPY), so row groups are a natural transfer unit that costs nothing — original compressed bytes, no decode, checksum chain intact per group, and each group carries its own statistics for time-based prioritization. This would change the ledger schema and the reconcile protocol, so it should be settled before PR 1, not retrofitted.

Rejected alternative: extracting rows and shipping msgpack. That inflates bandwidth 3–5× (row-major, self-describing per record, no cross-row compression) on the resource the edge has least of, costs edge CPU, and breaks the end-to-end checksum by forcing re-ingestion on the hub.

Later phases

Phase	Scope	License
2	Connectivity-adaptive agent, FeatureEdgeSyncAuto gate, cluster-manifest receive, bandwidth cap, 429 backpressure, multi-hub	Enterprise
3	S3/Azure relay transport, delete_after_sync	mixed
4	Metrics, per-spoke dashboard, sync-lag alerting, hub-as-relay	Enterprise

🤖 Generated with Claude Code

[xe-nvdk]

Decision: whole-file granularity for phase 1 (row groups deferred)

Settled against the code rather than the design doc. Reversing my earlier recommendation — I argued for row-group granularity when this came up; tracing the manifest shows it doesn't pay for itself in phase 1.

Why not row groups

raft.FileEntry (internal/cluster/raft/fsm.go:178-189) carries Path, SHA256, SizeBytes — per file, no row-group information. That is precisely what makes §5.1's reconcile cheap: the hub answers missing/present "from its manifest in O(N) lookups, no I/O on the parquet bytes."

Row-group sync breaks that on both ends:

1. The spoke would have to open every pending file and read its footer to enumerate groups and compute per-group digests. The manifest cannot answer it. Reconcile stops being a metadata operation and starts touching every candidate file on the smallest machine in the system — exactly the cost §5.1 was designed to avoid.
2. It needs a second integrity chain. The manifest's per-file SHA is the authority the whole design leans on (§6.1: (namespaced path, SHA256) *is* the file's identity). Per-group hashes have no authority to verify against, so we'd be inventing and persisting a parallel one.
3. Idempotency gets harder. §6.1's three-case receive logic (absent / same-SHA no-op / different-SHA 409) is stated over whole files. Partial-file presence turns a 3-case check into a range-merge problem, and the lost-ack recovery path (§6.2) stops being "one extra entry in the next reconcile batch."

What we lose, and why it's acceptable

The real concern was a contested link yielding nothing queryable until a large file completes. Two things blunt it:

• Byte-offset resume (§5.2, X-Arc-Sync-Offset) already means a dropped link doesn't restart from zero — bytes_sent on the ledger row is the checkpoint. A large file still drains across multiple contact windows.
• Compaction output is bounded by file count (max_files_per_batch, default 30 — see feat(compaction): make batch size configurable; remove dead TargetSizeMB #568), and a partition exceeding that already produces multiple independent output files. Lowering that key on edge deployments is a working lever today for smaller sync units, requiring no protocol change at all.

So the mitigation for the motivating problem is a config knob we already shipped, not a protocol redesign.

Not closing the door

Nothing in phase 1 precludes it. sync_ledger is keyed UNIQUE(hub_id, path); adding a nullable row-group range later is an additive migration, and the reconcile response can gain a partial-presence field backward-compatibly. Revisit when there is a real deployment where resume-across-windows proves insufficient — with measurements, not speculation.

Unblocks PR 1 (sync ledger).

[xe-nvdk]

PR 1/9 (sync ledger) is up: #570 — spoke-side sync_ledger + sync_history, SQL-enforced state machine, atomic retry cap, resume checkpoint, batched prune. 23 tests, no production callers yet.

Next up: PR 2 — SyncTransport interface. Worth settling the package-name question before it lands, since the transport is a likely stdlib-sync importer: internal/sync shadows the stdlib and forces importers to alias. Renaming to internal/edgesync costs a git mv now and gets more annoying with each stacked PR.

[xe-nvdk]

PR 4 scope decision: staging + copy-promote, all backends

§5.2 specifies "stream to a temp path, verify the SHA, atomic-rename into the final namespaced path". That is not expressible through Arc's storage layer as written, so recording how PR 4 resolves it.

The two constraints

1. Backend has no rename/move. WriteReader, Read, Delete, Exists, List, StatFile — nothing relocates an object. LocalBackend does stage to a .part file and os.Rename internally (local.go:201), but that is inside one WriteReader call, with no seam to verify a checksum between staging and promotion.

2. Resume is local-only. Byte-offset resume needs AppendingBackend.AppendReader, implemented only by LocalBackend (local.go:374). S3 and Azure return ErrResumeNotSupported — you cannot append to a block object.

Resolution

Verify-before-commit is preserved on every backend, via copy-promote instead of rename-promote. WriteReader accepts an arbitrary path, so the hub stages to .sync-staging/{spoke_id}/{path}, hashes while streaming, verifies against the spoke's declared digest, then streams staging → final and deletes the staging object. The copy uses the same pipe pattern the tiering migrator already relies on (tiering/migrator.go:310 copyFileStreaming).

Trade-off: promotion is a copy rather than a rename, so on object storage a promoted file costs an extra read+write. Atomicity is weaker than a rename — a crash mid-promote can leave a staging object, which is recoverable (the staging prefix is swept, and the transfer simply retries) and never leaves corrupt bytes at the final path.

Resume degrades rather than failing. Where the backend implements AppendingBackend, a dropped transfer resumes from X-Arc-Sync-Offset. Where it does not (S3, Azure), the partial is discarded and the transfer restarts from zero — the same fallback the peer-fetch puller already implements for ErrResumeNotSupported (filereplication/puller.go:858).

What this means for operators

	Local hub	S3 / Azure hub
Verify-before-commit	yes	yes
Byte-offset resume	yes	no — restarts from zero
Promote cost	rename	extra read + write

An S3 hub is supported, not refused. The resume gap only bites when a contact window is shorter than a single file transfer — the aerospace profile. Deployments on object storage with intermittent links should lower compaction.max_files_per_batch (#568) so individual files stay small enough to cross a window.

An earlier version of this analysis concluded the hub had to be local-only, mirroring Iceberg. That was wrong: it confused "no rename" with "no verify-before-commit", and would have refused a configuration that works. Recording it here so the reasoning is not re-derived.

[xe-nvdk]

PR 5 scope decisions: hub-side received index + paged reconcile

Two places where §5.1's assumptions do not hold against the code as built. Recording both so they are not re-derived.

1. There is no manifest to answer from in standalone

§5.1 says the hub answers missing/present/conflicts "from its manifest in O(N) lookups, no I/O on the parquet bytes." That manifest is Coordinator.GetFileEntry (cluster/coordinator.go:3125) — an O(1) lookup carrying the per-file SHA256 — and it exists only in cluster mode. GetFileEntry returns (nil, false) when raftFSM is nil.

In OSS standalone there is no file index at all. Telling present from conflicts would mean reading and hashing every candidate file (Receiver.hashStored), so a spoke returning from a long outage with thousands of pending files could cost gigabytes of disk reads per reconcile — on the request that exists precisely to be cheap.

Resolution: the hub persists its own received-files index. PR 4's receiver already has every file's path, SHA256, and size at commit time; it now records them in SQLite alongside the ledger tables. Reconcile becomes an O(1) lookup on any backend, cluster or standalone, with no parquet I/O — which is what §5.1 actually promised.

Costs one table and one small write per received file, on a path that has already done a full hash and a promote. It also gives the hub somewhere to answer from that does not depend on Raft being up, and sits beside where the spoke registry will live (PR 7).

2. The reconcile body cannot stream

§5.1 requires streaming the body both ways, noting 100k entries ≈ 20MB. Arc constructs its Fiber app with StreamRequestBody: false (api/server.go:124) so the gzip ingest path can handle bodies itself — established in #574 — which means fasthttp buffers the entire request body before routing, and therefore before authentication. Streaming the request is not available without a server-wide change that would alter behavior for every existing endpoint including the ingest hot path.

Resolution: cap entries per request and let the spoke page. A new edge_sync.max_reconcile_entries (default 10,000, ≈2MB) is enforced on Content-Length and on the decoded entry count. A spoke with a larger backlog sends several reconcile requests.

This keeps the property that actually matters — discovery cost is O(batches), not O(files), so 5,000 pending files is one request rather than 5,000 — while bounding what an unauthenticated caller can make the hub hold. The alternative of reusing the 512MiB upload cap would have let a reconcile claim orders of magnitude more memory than any reconcile should.

Deviation from §5.1 to note for later: the response is also materialized rather than stream-written. At the capped batch size that is a few hundred KB, which is not worth an iterator API. If a deployment ever needs larger batches, streaming the response is the first thing to revisit.

[xe-nvdk]

PRs 6 + 7 combined: spoke registry

PR 6 (spoke namespacing + 409) is already merged. NamespacedPath, the 409-never-overwrite rule, spoke-ID validation, and the reconcile-time conflict list all shipped in #574 and #575, because the receive and reconcile endpoints could not be correct without them. Opening a thin PR to re-test merged code would add process without adding safety, so the remaining piece — proving a spoke cannot write into another's namespace — folds into the registry work here, where the per-spoke secret binding actually lives.

Design decision: secret_hash cannot be a hash

§8.1 specifies secret_hash TEXT NOT NULL in sync_spokes. That does not work. HMAC verification recomputes the MAC from the secret, so the hub needs the plaintext at request time. A one-way hash — Arc's PBKDF2 API-token scheme, for instance — supports verification of a presented value, not recomputation of a MAC over a fresh message.

Resolution: encrypted at rest, not hashed. Secrets are stored AES-256-GCM encrypted using Arc's existing AESEncryptor (internal/mqtt/encryption.go), keyed on ARC_ENCRYPTION_KEY — the same mechanism MQTT already uses for broker passwords, which is the closest existing analogue: a credential Arc must be able to use, not merely check.

A hub with edge_sync.enabled=true and no ARC_ENCRYPTION_KEY refuses to start. There is deliberately no plaintext fallback: a silent downgrade would leave every spoke's write credential readable in a database that also holds audit logs, and the failure would be invisible until someone copied the file.

The column is named secret_encrypted rather than secret_hash so the storage model is not misread from the schema.

Design decision: the hub generates the secret

POST /api/v1/sync/spokes generates a cryptographically random secret and returns it once, in the registration response. It is never readable again — a subsequent GET returns metadata only.

This matches how Arc issues API tokens, and it removes the failure mode where an operator (or an automation) supplies a weak or reused secret. Rotation is a separate endpoint that generates a new secret and returns it the same way.

[xe-nvdk]

PR 8 scope decisions: hash at discovery, one-shot only

1. Discovery has no manifest to read digests from

§8.2 step 1 says the agent "walks the manifest for files not yet in sync_ledger". Same gap PR 5 hit on the hub side: that manifest is the Raft FSM, so a standalone spoke has none. storage.ObjectLister.ListObjects is implemented by all three backends and gives path, size, and mtime — but no SHA256, and the digest is the integrity anchor the entire design rests on (§6.1 makes (path, sha256) the file's identity).

Resolution: hash each file once, at discovery. The spoke walks storage, computes the digest for each newly-discovered file, and stores it in the ledger. That is one full read per file over its entire lifetime — and the spoke has to read those bytes to send them anyway, so the marginal cost is a single extra pass amortized across however long the file lives.

The alternative of reading digests from the Raft manifest when present would give two discovery paths behaving differently depending on deployment shape, for a saving that only applies to clustered spokes. Deferring the hash to send time is worse: reconcile could not ask "do you have this exact content?" until a file had already been sent once, which defeats the lost-ack recovery reconcile exists for.

2. Phase 1 is manual; the background agent is Enterprise

§8.2 describes a connectivity-adaptive ticker, but §9 and the enterprise product definition put the automatic agent in phase 2 under FeatureEdgeSyncAuto, with manual export/import in OSS.

Resolution: PR 8 ships the one-shot only.

• POST /api/v1/sync/run — discover → reconcile → stream, once, returning a summary.
• GET /api/v1/sync/status — pending count, pending bytes, last sync, sync lag.
• GET /api/v1/sync/ledger — query ledger entries for troubleshooting.

No ticker, no scheduler, no license plumbing. The agent internals land here so phase 2 is a ticker plus a license gate rather than a rewrite — which is also what keeps the OSS/Enterprise boundary exactly where §9 puts it, rather than shipping licensed behaviour in a phase-1 PR.

Deliberately out of scope for this PR, from §8.2: the connectivity gate, the token-bucket rate cap (sync.max_bytes_per_sec), and 429/Retry-After backoff. All three are properties of a loop that runs unattended; a one-shot an operator triggers has none of them to govern. They land with the agent in phase 2.

[xe-nvdk]

PR 8: spoke-side configuration decisions

The secret is env-only, and a config-file secret is refused

A spoke signs every request with its hub-issued secret, so that secret has to live on the edge box. Viper auto-binds every config key to an env var (SetEnvPrefix("ARC") + AutomaticEnv), so a edge_sync.spoke.secret key would be settable from either arc.toml or ARC_EDGE_SYNC_SPOKE_SECRET — and the TOML path means a live write credential sitting in a file that gets committed, copied, and backed up.

ARC_ENCRYPTION_KEY already sets the opposite precedent: read via os.Getenv, never a config key.

Resolution: the same treatment. The spoke secret is read from ARC_EDGE_SYNC_SPOKE_SECRET only, and Arc refuses to start if a secret is found in the config file rather than quietly preferring the environment. A silent preference would leave the committed copy looking load-bearing while doing nothing — and still leaked.

The spoke gets its own config block

An Arc instance can be a hub, a spoke, both, or neither, and the two roles share no settings: a hub has hub_id and upload limits, a spoke has a hub URL and its own identity. Flattening them into one block would leave an operator unable to tell which keys apply to their deployment, and would force renaming enabled to disambiguate.

[edge_sync]              # hub receive side
enabled = true
hub_id = "ground-station"

[edge_sync.spoke]        # push side
enabled = true
hub_url = "https://ground-station.example.com"
spoke_id = "rocket-01"
# secret comes from ARC_EDGE_SYNC_SPOKE_SECRET, never from this file