feat(edgesync): spoke agent and manual sync pass (#569)

[xe-nvdk]

Re-targets #578 at main. #578 was stacked on fix/api-test-race-timeouts and merged into that branch rather than retargeting when #577 landed, so the spoke agent never reached main. This is the same commit (6338b19) cherry-picked onto main — no content changes, re-verified from a clean build.

Summary

Completes the edge-sync story: an edge Arc now pushes its Parquet files to a hub on demand. The hub receive side, ledger, transport interface, HMAC scheme, reconcile index, and spoke registry landed in #570–#576; this is the client that drives them. Closes item 8 of #569.

A pass recovers transfers interrupted by a crash, discovers new files, reconciles the backlog in one round-trip, then streams what the hub lacks — newest first, so a contact window that closes mid-backlog has already delivered the freshest telemetry. It pages until the backlog drains, so one pass on a spoke returning from a long outage moves everything rather than the first batch.

Operator surface (admin-only, on /api/v1/spoke-sync):

Endpoint	Purpose
POST /run	Run one pass, return what it did
GET /status	Pending/synced/failed counts and sync lag
GET /ledger	Per-file state, attempts, and last error

The issue planned /api/v1/sync/run; it ships under /api/v1/spoke-sync because Fiber's Group().Use() matches by string prefix, not path segment — see the fourth row below.

Config lives in [edge_sync.spoke]; the secret is environment-only via ARC_EDGE_SYNC_SPOKE_SECRET, and Arc refuses to start if one appears in the config file. This is the manual form and is OSS; the scheduled agent is Enterprise and lands later.

Bugs found and fixed during review

Each has a regression test verified to fail against the pre-fix code:

Bug	Impact
Secret guard used v.IsSet	Consults the environment under AutomaticEnv, so Arc refused to start in the one configuration the guard exists to require. Now v.InConfig, which reads only the file.
Run fetched a single page	Sent batch_size files, reported success, and silently stranded the rest.
Negative batch_size	Reached make()'s capacity argument and panicked the spoke on its first pass. Clamped in NewAgent, rejected at load.
Fiber group prefix collision	Group().Use() matches by string prefix, so the hub's /api/v1/sync group also matched /api/v1/sync-spoke/* and its body limit ran on the operator routes. Moved to /api/v1/spoke-sync, with a test pinning the property rather than the name.
Ledger view showed only pending	Hid the exhausted files it exists to diagnose. Added Ledger.Unfinished.
Reconcile-path conflicts stayed pending	Rode along in every future reconcile payload forever. Added Ledger.MarkConflicted — MarkFailed requires in_flight and silently matched no rows.

Also: max_concurrent is bounded (64), cancellation is checked before the select rather than as a case (both being ready made it a coin flip, ~50% of transfers still launching), hub_url is parsed rather than prefix-matched, and every new key has a v.SetDefault.

Test plan

• 37 new tests across agent, spoke API, e2e, and config
• E2E drives the real agent against real hub handlers over real HTTP
• go test -count=1 -race ./internal/edgesync/ ./internal/api/ — clean on the rebuilt tree
• Two live processes: 20 files at batch_size=7, all byte-identical on the hub, idempotent re-run, no staging residue
• Non-default max_attempts / max_concurrent / batch_size exercised together
• Auth boundary: forged MAC → 401, unknown spoke → 401, wrong secret writes nothing
• ARC_ENCRYPTION_KEY rotation → startup warning + spoke fails closed
• Negative batch_size refused at startup with a clear message
• §6.1 identity rule verified live: conflicting content → reported, 0 bytes sent, hub copy unchanged
• go vet, gofmt clean

Follow-ups

• Docs ship in docs(edge-sync): document both sides of edge sync docs.basekick.net#20.
• PR 9 (air-gap export/import bundle) closes out phase 1 of Edge sync (spoke→hub replication): phase 1 manual core #569.

🤖 Generated with Claude Code