feat(sync): spoke-side ledger for edge-to-cloud replication (#569 PR 1/9)

[xe-nvdk]

First of nine PRs for edge sync — see #569 for the phase-1 breakdown and docs/progress/2026-06-04-edge-sync-architecture-converged.md for the design.

Summary

Adds internal/sync with the spoke-side ledger: the durable record of which Parquet files have reached which hub, and how far a partial transfer got.

• sync_ledger keyed UNIQUE(hub_id, path) from day one — multi-hub becomes config later, not a schema migration on a disconnected box.
• State transitions enforced in SQL, not by trusting callers.
• MarkFailed's retry cap decided inside a single UPDATE — no read-then-write race.
• bytes_sent as the resume checkpoint, preserved across failure, clamped to size_bytes.
• PruneSynced batched at 1000 rows, PK-ordered, ctx.Err() between batches, no vacuum.
• UTC normalization on write and read.

No production callers. Nothing constructs the ledger, so its schema is never created. Wiring lands with /sync/run (PR 8).

Design decision recorded on the issue

Granularity is whole-file, not row-group — decision comment. raft.FileEntry carries SHA256/SizeBytes per file with no row-group data, which is what makes §5.1's reconcile answerable from the manifest with no I/O on parquet bytes. Row-group sync would force the spoke to open every pending file's footer, need a second integrity chain with no authority to verify against, and turn §6.1's three-case receive into a range-merge problem. Revisit later is cheap: a row-group range is an additive column.

Configuration matrix

Configuration	Reaches new code?	Preconditions established?
OSS standalone, any config	No — grep NewLedger outside the package returns nothing	n/a; no live derefs
Cluster mode, any config	No — no wiring in cmd/arc/main.go	n/a
Tests only	Yes — ledger_test.go is the sole caller	test DB via sql.Open, t.Cleanup closes

Every row is "no", which inverts the usual risk profile: no config reaches this code, so no nil-deref or subsystem-interaction bug is possible. Risk sits entirely in SQL correctness and the state machine, which is where review was pointed.

Config keys introduced: none. Deref check: NewLedger rejects a nil *sql.DB; no other pointer is dereferenced internally.

Review

Two adversarial passes. The first found 3 blockers + 6 high, all fixed; I verified each empirically with probes before fixing rather than taking them on faith:

• Lost-update race in MarkFailed — read-then-write let a concurrent worker push attempts past the cap between statements; the losing writer resurrected the entry to pending, retrying forever. Now one atomic UPDATE with the CASE inside.
• A factually wrong time-domain comment — probed the real stored format: "2026-08-06 22:40:43.006808+00:00", space-separated with an offset, neither RFC3339 nor datetime() output. The comment pointed a future maintainer the wrong way.
• discovered_at not UTC-normalized — probed: the same instant stored two ways gave SQL equality 0.
• Every illegal transition permitted — MarkInFlight on a synced row succeeded with synced_at still set, double-counting in Stats.
• PendingBytes went negative (−4900 with an over-large offset), and since it is a SUM, one bad offset silently cancelled other files' real pending bytes.
• Full scan + temp B-tree on the prune cutoff and status query — fixed with a second index, both re-verified with EXPLAIN QUERY PLAN.

Two reviewer claims I checked and declined: the prune's state filter is redundant-by-construction rather than untested (removing either filter alone still protects unsynced rows — deliberate belt-and-braces on a DELETE), and database as a column name doesn't actually need quoting (probed unquoted in WHERE/SELECT/GROUP BY/ORDER BY; four existing Arc tables already do this).

The second pass found no bugs. It suggested an explicit pending → synced reconcile-path test, which is added.

I also found one of my own tests was vacuous — the TrackBatch rollback test never triggered a rollback because empty strings satisfy NOT NULL. Replaced with a trigger-based abort and verified it fails pre-fix (reported 1 inserted but rollback discarded every row).

Test plan

• go build ./cmd/... ./internal/...
• go test ./internal/sync/ — 23 tests
• go test -race ./internal/sync/
• go vet / gofmt -l clean
• Query plans verified with EXPLAIN QUERY PLAN (no full scans, no temp B-trees)
• New regression tests verified to fail pre-fix
• Binary run — N/A. No startup path reaches this code; there is nothing to exercise. The usual run-the-binary gate applies to PR 8, which first gives the ledger a caller.

Notes

• Docs: no docs.basekick.net change — nothing here is configurable or observable. Docs land with PR 8's endpoints.
• Package name: internal/sync shadows stdlib sync; importers needing sync.Mutex must alias. Renaming to edgesync is cheap now and progressively more annoying as PRs 2–9 stack. Happy to do it before PR 2.
• Deferred to PR 8: RecoverInFlight has no age guard and Pending has no claim/lease, so two workers could hand out the same row. Both are real but belong with the agent, where concurrency actually exists — building a lease with no consumer now would be speculative.

🤖 Generated with Claude Code