Repository navigation
feat(edgesync): spoke registry with encrypted-at-rest secrets (#569 PRs 6+7/9) - #576
Merged
Merged
Conversation
Combines PRs 6 and 7 of nine for edge sync (#569). PR 6's spoke namespacing and 409-never-overwrite shipped in #574 and #575 — the receive and reconcile endpoints could not be correct without them — so what remained was the per-spoke secret binding, which lives here. This makes the hub usable. Until now it rejected every request for want of a way to register a spoke. §8.1 specifies a `secret_hash` column. That cannot work: HMAC verification recomputes the MAC from the secret, so the hub needs the plaintext at request time — unlike an API token, which is only ever checked against a value the caller presents. Secrets are therefore encrypted at rest with AES-256-GCM under ARC_ENCRYPTION_KEY, the same mechanism MQTT uses for broker passwords, and the column is named secret_encrypted so the storage model cannot be misread from the schema. The hub generates the secret and returns it once, from registration or rotation. It is never readable again — Spoke carries no secret field, so metadata paths cannot leak it by construction. Generating rather than accepting one removes the failure mode where an operator or an automation picks something weak or reuses it across a fleet. Adversarial review found two blockers, both in claims I had made and not properly checked. The comment asserting the admin routes inherit the sync group's middleware was wrong. I had tested it with the sync group registered first; main.go registers admin first, and under that order the middleware does not run. Worse, the behavior flips on reordering two lines, at which point a registration whose JSON exceeds the sync body limit would get a 413 from the wrong group before auth ran. Admin routes now sit at /api/v1/sync-spokes, so the isolation is structural rather than a property of registration order. The cipher guard did not guard. mqtt.NewPasswordEncryptor returns a non-nil pass-through for an empty key, which satisfies the interface, so a caller that forgot to check the key would have stored every secret in plaintext while construction reported success. NewRegistry now round-trips a canary and refuses a cipher that does not encrypt. Three mutations survived because the tests used a single spoke: a RecordActivity that updates every row, a List that drops its ORDER BY, and a scan that always reports enabled. All three are invisible with one registration. Multi-spoke tests now catch each, and List has a stable tiebreaker so bulk provisioning cannot produce arbitrary order. Also: spoke names are bounded and reject control characters, since they are echoed into logs and every list response; a decrypt failure now logs at Warn rather than Debug, because it is a hub misconfiguration affecting every spoke rather than the scan noise an unknown spoke ID is; and the startup path verifies the configured key still decrypts what is stored, refusing to start on a changed key rather than letting the hub look healthy while nothing can authenticate. With auth.enabled=false the admin routes are unauthenticated, matching every other Arc admin endpoint in that mode. Review's recommendation was to keep the pattern rather than introduce a lone inconsistency, and to make the posture visible: startup now warns explicitly that anyone reaching the port can mint write credentials. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
10 tasks done
xe-nvdk
added a commit
that referenced
this pull request
Aug 7, 2026
Completes the edge-sync story: an edge Arc now pushes its Parquet files to a hub on demand. The hub receive side, ledger, transport interface, HMAC scheme, reconcile index, and spoke registry landed in #570-#576; this is the client that drives them. A pass recovers transfers interrupted by a crash, discovers new files, reconciles the backlog in one round-trip, then streams what the hub lacks — newest first, so a contact window that closes mid-backlog has already delivered the freshest telemetry. It pages until the backlog drains, so one pass on a spoke returning from a long outage moves everything rather than the first batch. Operator surface (admin-only, on /api/v1/spoke-sync): POST /run run one pass, return what it did GET /status pending/synced/failed counts and sync lag GET /ledger per-file state, attempts, and last error Config lives in [edge_sync.spoke]; the secret is environment-only via ARC_EDGE_SYNC_SPOKE_SECRET. This is the manual form and is OSS; the scheduled agent is Enterprise and lands later. Bugs found and fixed during review, each with a regression test verified to fail against the pre-fix code: - Secret guard used v.IsSet, which consults the environment under AutomaticEnv — so Arc refused to start in the one configuration the guard exists to require. Now v.InConfig, which reads only the file. - Run fetched a single page: it sent batch_size files, reported success, and silently stranded the rest. - A negative batch_size reached make()'s capacity argument and panicked the spoke on its first pass. Clamped in NewAgent, rejected at load. - Fiber's Group().Use() matches by string prefix, not path segment, so the hub's group at /api/v1/sync also matched /api/v1/sync-spoke/*, and its body limit ran on the operator routes. Moved to /api/v1/spoke-sync, with a test pinning the property rather than the name. - The ledger view showed only pending entries, hiding the exhausted files it exists to diagnose. Added Ledger.Unfinished. - Reconcile-path conflicts stayed pending and rode along in every future reconcile payload forever. Added Ledger.MarkConflicted, since MarkFailed requires in_flight and silently matched no rows. Also: max_concurrent is bounded (64), cancellation is checked before the select rather than as a case (both being ready made it a coin flip), hub_url is parsed rather than prefix-matched, and every new key has a v.SetDefault. Test plan: - [x] 37 new tests across agent, spoke API, e2e, and config - [x] e2e drives the real agent against real hub handlers over real HTTP - [x] go test -race ./internal/edgesync/ — clean - [x] two live processes: 20 files, batch_size=7, all byte-identical on the hub, idempotent re-run, no staging residue - [x] non-default max_attempts/max_concurrent/batch_size exercised - [x] auth boundary: forged MAC 401, unknown spoke 401, wrong secret writes nothing - [x] negative batch_size refused at startup with a clear message - [x] go vet, gofmt clean
10 tasks done
xe-nvdk
added a commit
that referenced
this pull request
Aug 7, 2026
Completes the edge-sync story: an edge Arc now pushes its Parquet files to a hub on demand. The hub receive side, ledger, transport interface, HMAC scheme, reconcile index, and spoke registry landed in #570-#576; this is the client that drives them. A pass recovers transfers interrupted by a crash, discovers new files, reconciles the backlog in one round-trip, then streams what the hub lacks — newest first, so a contact window that closes mid-backlog has already delivered the freshest telemetry. It pages until the backlog drains, so one pass on a spoke returning from a long outage moves everything rather than the first batch. Operator surface (admin-only, on /api/v1/spoke-sync): POST /run run one pass, return what it did GET /status pending/synced/failed counts and sync lag GET /ledger per-file state, attempts, and last error Config lives in [edge_sync.spoke]; the secret is environment-only via ARC_EDGE_SYNC_SPOKE_SECRET. This is the manual form and is OSS; the scheduled agent is Enterprise and lands later. Bugs found and fixed during review, each with a regression test verified to fail against the pre-fix code: - Secret guard used v.IsSet, which consults the environment under AutomaticEnv — so Arc refused to start in the one configuration the guard exists to require. Now v.InConfig, which reads only the file. - Run fetched a single page: it sent batch_size files, reported success, and silently stranded the rest. - A negative batch_size reached make()'s capacity argument and panicked the spoke on its first pass. Clamped in NewAgent, rejected at load. - Fiber's Group().Use() matches by string prefix, not path segment, so the hub's group at /api/v1/sync also matched /api/v1/sync-spoke/*, and its body limit ran on the operator routes. Moved to /api/v1/spoke-sync, with a test pinning the property rather than the name. - The ledger view showed only pending entries, hiding the exhausted files it exists to diagnose. Added Ledger.Unfinished. - Reconcile-path conflicts stayed pending and rode along in every future reconcile payload forever. Added Ledger.MarkConflicted, since MarkFailed requires in_flight and silently matched no rows. Also: max_concurrent is bounded (64), cancellation is checked before the select rather than as a case (both being ready made it a coin flip), hub_url is parsed rather than prefix-matched, and every new key has a v.SetDefault. Test plan: - [x] 37 new tests across agent, spoke API, e2e, and config - [x] e2e drives the real agent against real hub handlers over real HTTP - [x] go test -race ./internal/edgesync/ — clean - [x] two live processes: 20 files, batch_size=7, all byte-identical on the hub, idempotent re-run, no staging residue - [x] non-default max_attempts/max_concurrent/batch_size exercised - [x] auth boundary: forged MAC 401, unknown spoke 401, wrong secret writes nothing - [x] negative batch_size refused at startup with a clear message - [x] go vet, gofmt clean
This was referenced Aug 19, 2026
xe-nvdk
added a commit
that referenced
this pull request
Aug 19, 2026
Internal audit of the shipped edge sync subsystem (#576-#584) found four release blockers; all fixed and live-verified on a two-node rig. - Auth (B1): the spoke transport never sent an Arc API token while the hub mounted admin-level token auth on /api/v1/sync — every request against an auth-enabled hub (the default) died with 401. The hub group now gates at write level (matching ingest), and the spoke carries a new env-only credential ARC_EDGE_SYNC_HUB_TOKEN (same config-file-refusal discipline as the spoke secret). Tokenless spokes warn at startup, and the 401/403 remediation text names the variable. - Quoted identifiers (B2, general query-layer bug): MaskStringLiterals masked double-quoted tokens as string literals, so quoted database and measurement names resolved to quote-polluted read_parquet globs and returned zero rows — and hyphenated names (every spoke ID) had no working syntax at all. Double-quoted tokens now mask into a distinct, deduplicated __IDENT__ placeholder class; the table rewriters, the header-db path, and RBAC extraction all resolve placeholders to validated unquoted names, so extraction matches execution. Deep review of this fix caught that leaving an INVALID quoted identifier unrewritten would hand DuckDB a replacement scan (net-new cross-tenant read with RBAC off): ValidateSQLRequest now rejects invalid quoted identifiers in table position (generalizing the GHSA-w8x2 scanner, which also closes the pre-existing double-quoted comma-cross-join bypass), and the transform resolves invalid names to an inert sentinel as a backstop. - Reconcile paging (B3): the default batch_size=0 offered the whole backlog in one reconcile; a backlog above the hub's cap failed every pass with the same 413 forever. batch_size now defaults to 1000, and a refused page is split and retried in-pass using the hub's advertised cap (halving when the refusal is the byte limit), so no configuration can strand a backlog. - Vanished files (B4): compaction deleting a discovered-but-unsent file permanently wedged air-gap export (the entry was re-selected every time with no operator escape) and burned the network retry budget. StateSkipped is now wired: the exporter pre-checks existence and skips vanished entries (Exists must positively report the file gone — transient errors keep today's abort), the network agent skips instead of failing, and skipped rows are counted in /status and pruned. - Dead code wired: SweepStaging (hub staging DoS guard) now runs hourly (edge_sync.staging_sweep_max_age_hours, default 72); PruneSynced — also never called — and the new PruneSkipped run twice daily on the spoke (edge_sync.spoke.ledger_retention_days, default 90). - Compaction row-duplication on the hub (audit High) is documented with loud warnings in the release notes and docs; the hub-side supersede fix is tracked in #610. Remaining audit findings filed as #611-#616. Live gates on a real two-node rig, all previously-broken cells: sync through an auth-enabled cap-3 hub drains 8/8 at batch_size=0; quoted hyphenated queries return real rows on the hub; a discovered-then-deleted file lands skipped; the tokenless 401 names its remedy; export with a vanished file produces a valid bundle and completes import + ack; the replacement-scan shapes are rejected over HTTP with explicit errors.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Combines PRs 6 and 7 of nine for edge sync — see #569. Follows #570, #571, #572, #573, #574, #575.
PR 6 was already merged. Spoke namespacing, the 409-never-overwrite rule, and spoke-ID validation shipped in #574/#575 because the receive and reconcile endpoints could not be correct without them. What remained — the per-spoke secret binding — lives here, so opening a thin PR to re-test merged code would have added process without safety. Reasoning on the issue.
This makes the hub usable
Until now it rejected every request for want of a way to register a spoke.
Plus rotate, enable/disable (reversible, keeping history), delete (which deliberately keeps received files), and list.
Two design decisions
§8.1's
secret_hashcannot work. HMAC verification recomputes the MAC from the secret, so the hub needs the plaintext at request time — unlike an API token, only ever checked against a presented value. Secrets are AES-256-GCM encrypted underARC_ENCRYPTION_KEY, the same mechanism MQTT uses for broker passwords. A hub enabled without that key refuses to start. The column is namedsecret_encryptedso the storage model cannot be misread from the schema.The hub generates the secret, returned once.
Spokecarries no secret field, so metadata paths cannot leak it by construction. Generating rather than accepting one removes the weak/reused-secret failure mode.Two blockers — both in claims I made and did not properly check
My "verified, not assumed" middleware comment was false. I tested the route nesting with the sync group registered first;
main.goregisters admin first, and under that order the sync middleware does not run on admin routes. Worse, it flips on reordering two lines — after which a registration whose JSON exceeds the sync body limit gets a 413 from the wrong group before auth runs. Admin routes moved to/api/v1/sync-spokesso isolation is structural rather than order-dependent.The cipher guard did not guard.
mqtt.NewPasswordEncryptorreturns a non-nil pass-through for an empty key, which satisfies the interface — so a caller that forgot to check the key would store every secret in plaintext while construction reported success.NewRegistrynow round-trips a canary and refuses a cipher that does not encrypt.Three mutations survived because my tests used one spoke
RecordActivityupdating every row,Listdropping itsORDER BY, and a scan always reportingenabledare all invisible with a single registration. Multi-spoke tests now catch each;Listalso gained a stable tiebreaker so bulk provisioning cannot produce arbitrary order.Other findings fixed
ARC_ENCRYPTION_KEYwas otherwise invisible: admin endpoints keep answering 200 while nothing authenticates, and on an edge deployment the first failure might be a missed contact window away.Countinstead oflen(List(...)), bounded startup context, distinct logger component, generic 404 message.The auth question I put to the reviewer
With
auth.enabled=falsethe admin routes are unauthenticated — anyone reaching the port can mint write credentials. Its recommendation, which I took: keep the pattern (consistent across all 16 Arc admin handlers;mqtt.gomanages encrypted credentials identically) and make the posture visible. Startup now warns explicitly, and the docs carry a callout.Configuration matrix
enabled=false(default)enabled=true, noARC_ENCRYPTION_KEYenabled=true, key changed since registrationenabled=true+ key, auth enabledRequireAdminon admin routesenabled=true+ key, auth disabledTest plan
-racecleanORDER BYremoval, and routing the key check through a path that short-circuits on disabled spokespresent→ disable → immediate 401Review
Security pass, deep pass, and a third review. The security pass judged the auth-disabled question; the deep pass caught both blockers. A transient mutation in
GenerateSecretduring review — which would have made every secret identical — was a reviewer's own probe, not shipped code; I verified independently that generation is intact.Note
§8.1 of the design doc still specifies
secret_hash, which is architecturally impossible here. Documented in code, schema, and on the issue, but the doc itself should be corrected — a separate change from this PR.🤖 Generated with Claude Code