Skip to content

feat(edgesync): spoke registry with encrypted-at-rest secrets (#569 PRs 6+7/9) - #576

Merged
xe-nvdk merged 1 commit into
mainfrom
feat/edge-sync-registry
Aug 7, 2026
Merged

xe-nvdk merged 1 commit into
mainfrom
feat/edge-sync-registry

Conversation

@xe-nvdk

@xe-nvdk xe-nvdk commented Aug 7, 2026

Copy link
Copy Markdown
Member

Combines PRs 6 and 7 of nine for edge sync — see #569. Follows #570, #571, #572, #573, #574, #575.

PR 6 was already merged. Spoke namespacing, the 409-never-overwrite rule, and spoke-ID validation shipped in #574/#575 because the receive and reconcile endpoints could not be correct without them. What remained — the per-spoke secret binding — lives here, so opening a thin PR to re-test merged code would have added process without safety. Reasoning on the issue.

This makes the hub usable

Until now it rejected every request for want of a way to register a spoke.

curl -X POST https://hub/api/v1/sync-spokes/ -H "Authorization: Bearer $TOKEN" \
  -d '{"spoke_id": "rocket-01", "name": "Rocket 07 Telemetry"}'
# → 201 {"secret": "5db508a4…", "warning": "shown once and cannot be retrieved"}

Plus rotate, enable/disable (reversible, keeping history), delete (which deliberately keeps received files), and list.

Two design decisions

§8.1's secret_hash cannot work. HMAC verification recomputes the MAC from the secret, so the hub needs the plaintext at request time — unlike an API token, only ever checked against a presented value. Secrets are AES-256-GCM encrypted under ARC_ENCRYPTION_KEY, the same mechanism MQTT uses for broker passwords. A hub enabled without that key refuses to start. The column is named secret_encrypted so the storage model cannot be misread from the schema.

The hub generates the secret, returned once. Spoke carries no secret field, so metadata paths cannot leak it by construction. Generating rather than accepting one removes the weak/reused-secret failure mode.

Two blockers — both in claims I made and did not properly check

My "verified, not assumed" middleware comment was false. I tested the route nesting with the sync group registered first; main.go registers admin first, and under that order the sync middleware does not run on admin routes. Worse, it flips on reordering two lines — after which a registration whose JSON exceeds the sync body limit gets a 413 from the wrong group before auth runs. Admin routes moved to /api/v1/sync-spokes so isolation is structural rather than order-dependent.

The cipher guard did not guard. mqtt.NewPasswordEncryptor returns a non-nil pass-through for an empty key, which satisfies the interface — so a caller that forgot to check the key would store every secret in plaintext while construction reported success. NewRegistry now round-trips a canary and refuses a cipher that does not encrypt.

Three mutations survived because my tests used one spoke

RecordActivity updating every row, List dropping its ORDER BY, and a scan always reporting enabled are all invisible with a single registration. Multi-spoke tests now catch each; List also gained a stable tiebreaker so bulk provisioning cannot produce arbitrary order.

Other findings fixed

  • Spoke names bounded and control-character-free — a 1MB name was accepted, and an embedded newline let a registration forge log lines.
  • Decrypt failure now logs at Warn, not Debug — it is a hub misconfiguration affecting every spoke, not the scan noise an unknown ID is.
  • Startup verifies the key still decrypts what is stored. A changed ARC_ENCRYPTION_KEY was otherwise invisible: admin endpoints keep answering 200 while nothing authenticates, and on an edge deployment the first failure might be a missed contact window away.
  • Count instead of len(List(...)), bounded startup context, distinct logger component, generic 404 message.

The auth question I put to the reviewer

With auth.enabled=false the admin routes are unauthenticated — anyone reaching the port can mint write credentials. Its recommendation, which I took: keep the pattern (consistent across all 16 Arc admin handlers; mqtt.go manages encrypted credentials identically) and make the posture visible. Startup now warns explicitly, and the docs carry a callout.

Configuration matrix

Configuration Reaches new code? Preconditions established?
enabled=false (default) No — nothing constructed n/a
enabled=true, no ARC_ENCRYPTION_KEY No — refuses to start Verified live, no plaintext fallback
enabled=true, key changed since registration No — refuses to start Verified live: names the cause; original key starts cleanly
enabled=true + key, auth enabled Yes RequireAdmin on admin routes
enabled=true + key, auth disabled Yes Admin routes unauthenticated by design; startup warns. Verified live
Zero registered spokes Yes — sync 401s Startup names the endpoint to fix it. Verified live
Spoke disabled mid-flight Yes Immediate 401. Verified live
Cluster + enabled Yes — registry is Raft-independent Not exercised live

Test plan

  • Build, vet, gofmt, -race clean
  • Mutation-tested: 8 killed, including plaintext storage, disabled-still-authenticates, re-register-reissues, per-spoke activity scoping, ORDER BY removal, and routing the key check through a path that short-circuits on disabled spokes
  • Full loop verified live: register → upload signed with the issued secret → reconcile reports present → disable → immediate 401
  • Binary runs for every refusing configuration
  • Confirmed the moved admin path (old 404s, new works) and the 129-char name rejection

Review

Security pass, deep pass, and a third review. The security pass judged the auth-disabled question; the deep pass caught both blockers. A transient mutation in GenerateSecret during review — which would have made every secret identical — was a reviewer's own probe, not shipped code; I verified independently that generation is intact.

Note

§8.1 of the design doc still specifies secret_hash, which is architecturally impossible here. Documented in code, schema, and on the issue, but the doc itself should be corrected — a separate change from this PR.

🤖 Generated with Claude Code

Combines PRs 6 and 7 of nine for edge sync (#569). PR 6's spoke
namespacing and 409-never-overwrite shipped in #574 and #575 — the
receive and reconcile endpoints could not be correct without them — so
what remained was the per-spoke secret binding, which lives here.

This makes the hub usable. Until now it rejected every request for want
of a way to register a spoke.

§8.1 specifies a `secret_hash` column. That cannot work: HMAC
verification recomputes the MAC from the secret, so the hub needs the
plaintext at request time — unlike an API token, which is only ever
checked against a value the caller presents. Secrets are therefore
encrypted at rest with AES-256-GCM under ARC_ENCRYPTION_KEY, the same
mechanism MQTT uses for broker passwords, and the column is named
secret_encrypted so the storage model cannot be misread from the schema.

The hub generates the secret and returns it once, from registration or
rotation. It is never readable again — Spoke carries no secret field, so
metadata paths cannot leak it by construction. Generating rather than
accepting one removes the failure mode where an operator or an
automation picks something weak or reuses it across a fleet.

Adversarial review found two blockers, both in claims I had made and not
properly checked.

The comment asserting the admin routes inherit the sync group's
middleware was wrong. I had tested it with the sync group registered
first; main.go registers admin first, and under that order the
middleware does not run. Worse, the behavior flips on reordering two
lines, at which point a registration whose JSON exceeds the sync body
limit would get a 413 from the wrong group before auth ran. Admin routes
now sit at /api/v1/sync-spokes, so the isolation is structural rather
than a property of registration order.

The cipher guard did not guard. mqtt.NewPasswordEncryptor returns a
non-nil pass-through for an empty key, which satisfies the interface, so
a caller that forgot to check the key would have stored every secret in
plaintext while construction reported success. NewRegistry now
round-trips a canary and refuses a cipher that does not encrypt.

Three mutations survived because the tests used a single spoke: a
RecordActivity that updates every row, a List that drops its ORDER BY,
and a scan that always reports enabled. All three are invisible with one
registration. Multi-spoke tests now catch each, and List has a stable
tiebreaker so bulk provisioning cannot produce arbitrary order.

Also: spoke names are bounded and reject control characters, since they
are echoed into logs and every list response; a decrypt failure now logs
at Warn rather than Debug, because it is a hub misconfiguration
affecting every spoke rather than the scan noise an unknown spoke ID is;
and the startup path verifies the configured key still decrypts what is
stored, refusing to start on a changed key rather than letting the hub
look healthy while nothing can authenticate.

With auth.enabled=false the admin routes are unauthenticated, matching
every other Arc admin endpoint in that mode. Review's recommendation was
to keep the pattern rather than introduce a lone inconsistency, and to
make the posture visible: startup now warns explicitly that anyone
reaching the port can mint write credentials.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@xe-nvdk
xe-nvdk merged commit f98a8b3 into main Aug 7, 2026
4 checks passed
@xe-nvdk
xe-nvdk deleted the feat/edge-sync-registry branch August 7, 2026 18:13
xe-nvdk added a commit that referenced this pull request Aug 7, 2026
Completes the edge-sync story: an edge Arc now pushes its Parquet files
to a hub on demand. The hub receive side, ledger, transport interface,
HMAC scheme, reconcile index, and spoke registry landed in #570-#576;
this is the client that drives them.

A pass recovers transfers interrupted by a crash, discovers new files,
reconciles the backlog in one round-trip, then streams what the hub
lacks — newest first, so a contact window that closes mid-backlog has
already delivered the freshest telemetry. It pages until the backlog
drains, so one pass on a spoke returning from a long outage moves
everything rather than the first batch.

Operator surface (admin-only, on /api/v1/spoke-sync):
  POST /run      run one pass, return what it did
  GET  /status   pending/synced/failed counts and sync lag
  GET  /ledger   per-file state, attempts, and last error

Config lives in [edge_sync.spoke]; the secret is environment-only via
ARC_EDGE_SYNC_SPOKE_SECRET. This is the manual form and is OSS; the
scheduled agent is Enterprise and lands later.

Bugs found and fixed during review, each with a regression test verified
to fail against the pre-fix code:

- Secret guard used v.IsSet, which consults the environment under
  AutomaticEnv — so Arc refused to start in the one configuration the
  guard exists to require. Now v.InConfig, which reads only the file.
- Run fetched a single page: it sent batch_size files, reported success,
  and silently stranded the rest.
- A negative batch_size reached make()'s capacity argument and panicked
  the spoke on its first pass. Clamped in NewAgent, rejected at load.
- Fiber's Group().Use() matches by string prefix, not path segment, so
  the hub's group at /api/v1/sync also matched /api/v1/sync-spoke/*, and
  its body limit ran on the operator routes. Moved to /api/v1/spoke-sync,
  with a test pinning the property rather than the name.
- The ledger view showed only pending entries, hiding the exhausted files
  it exists to diagnose. Added Ledger.Unfinished.
- Reconcile-path conflicts stayed pending and rode along in every future
  reconcile payload forever. Added Ledger.MarkConflicted, since MarkFailed
  requires in_flight and silently matched no rows.

Also: max_concurrent is bounded (64), cancellation is checked before the
select rather than as a case (both being ready made it a coin flip),
hub_url is parsed rather than prefix-matched, and every new key has a
v.SetDefault.

Test plan:
- [x] 37 new tests across agent, spoke API, e2e, and config
- [x] e2e drives the real agent against real hub handlers over real HTTP
- [x] go test -race ./internal/edgesync/ — clean
- [x] two live processes: 20 files, batch_size=7, all byte-identical on
      the hub, idempotent re-run, no staging residue
- [x] non-default max_attempts/max_concurrent/batch_size exercised
- [x] auth boundary: forged MAC 401, unknown spoke 401, wrong secret
      writes nothing
- [x] negative batch_size refused at startup with a clear message
- [x] go vet, gofmt clean
xe-nvdk added a commit that referenced this pull request Aug 7, 2026
Completes the edge-sync story: an edge Arc now pushes its Parquet files
to a hub on demand. The hub receive side, ledger, transport interface,
HMAC scheme, reconcile index, and spoke registry landed in #570-#576;
this is the client that drives them.

A pass recovers transfers interrupted by a crash, discovers new files,
reconciles the backlog in one round-trip, then streams what the hub
lacks — newest first, so a contact window that closes mid-backlog has
already delivered the freshest telemetry. It pages until the backlog
drains, so one pass on a spoke returning from a long outage moves
everything rather than the first batch.

Operator surface (admin-only, on /api/v1/spoke-sync):
  POST /run      run one pass, return what it did
  GET  /status   pending/synced/failed counts and sync lag
  GET  /ledger   per-file state, attempts, and last error

Config lives in [edge_sync.spoke]; the secret is environment-only via
ARC_EDGE_SYNC_SPOKE_SECRET. This is the manual form and is OSS; the
scheduled agent is Enterprise and lands later.

Bugs found and fixed during review, each with a regression test verified
to fail against the pre-fix code:

- Secret guard used v.IsSet, which consults the environment under
  AutomaticEnv — so Arc refused to start in the one configuration the
  guard exists to require. Now v.InConfig, which reads only the file.
- Run fetched a single page: it sent batch_size files, reported success,
  and silently stranded the rest.
- A negative batch_size reached make()'s capacity argument and panicked
  the spoke on its first pass. Clamped in NewAgent, rejected at load.
- Fiber's Group().Use() matches by string prefix, not path segment, so
  the hub's group at /api/v1/sync also matched /api/v1/sync-spoke/*, and
  its body limit ran on the operator routes. Moved to /api/v1/spoke-sync,
  with a test pinning the property rather than the name.
- The ledger view showed only pending entries, hiding the exhausted files
  it exists to diagnose. Added Ledger.Unfinished.
- Reconcile-path conflicts stayed pending and rode along in every future
  reconcile payload forever. Added Ledger.MarkConflicted, since MarkFailed
  requires in_flight and silently matched no rows.

Also: max_concurrent is bounded (64), cancellation is checked before the
select rather than as a case (both being ready made it a coin flip),
hub_url is parsed rather than prefix-matched, and every new key has a
v.SetDefault.

Test plan:
- [x] 37 new tests across agent, spoke API, e2e, and config
- [x] e2e drives the real agent against real hub handlers over real HTTP
- [x] go test -race ./internal/edgesync/ — clean
- [x] two live processes: 20 files, batch_size=7, all byte-identical on
      the hub, idempotent re-run, no staging residue
- [x] non-default max_attempts/max_concurrent/batch_size exercised
- [x] auth boundary: forged MAC 401, unknown spoke 401, wrong secret
      writes nothing
- [x] negative batch_size refused at startup with a clear message
- [x] go vet, gofmt clean
xe-nvdk added a commit that referenced this pull request Aug 19, 2026
Internal audit of the shipped edge sync subsystem (#576-#584) found four
release blockers; all fixed and live-verified on a two-node rig.

- Auth (B1): the spoke transport never sent an Arc API token while the hub
  mounted admin-level token auth on /api/v1/sync — every request against an
  auth-enabled hub (the default) died with 401. The hub group now gates at
  write level (matching ingest), and the spoke carries a new env-only
  credential ARC_EDGE_SYNC_HUB_TOKEN (same config-file-refusal discipline as
  the spoke secret). Tokenless spokes warn at startup, and the 401/403
  remediation text names the variable.

- Quoted identifiers (B2, general query-layer bug): MaskStringLiterals
  masked double-quoted tokens as string literals, so quoted database and
  measurement names resolved to quote-polluted read_parquet globs and
  returned zero rows — and hyphenated names (every spoke ID) had no working
  syntax at all. Double-quoted tokens now mask into a distinct, deduplicated
  __IDENT__ placeholder class; the table rewriters, the header-db path, and
  RBAC extraction all resolve placeholders to validated unquoted names, so
  extraction matches execution. Deep review of this fix caught that leaving
  an INVALID quoted identifier unrewritten would hand DuckDB a replacement
  scan (net-new cross-tenant read with RBAC off): ValidateSQLRequest now
  rejects invalid quoted identifiers in table position (generalizing the
  GHSA-w8x2 scanner, which also closes the pre-existing double-quoted
  comma-cross-join bypass), and the transform resolves invalid names to an
  inert sentinel as a backstop.

- Reconcile paging (B3): the default batch_size=0 offered the whole backlog
  in one reconcile; a backlog above the hub's cap failed every pass with the
  same 413 forever. batch_size now defaults to 1000, and a refused page is
  split and retried in-pass using the hub's advertised cap (halving when the
  refusal is the byte limit), so no configuration can strand a backlog.

- Vanished files (B4): compaction deleting a discovered-but-unsent file
  permanently wedged air-gap export (the entry was re-selected every time
  with no operator escape) and burned the network retry budget. StateSkipped
  is now wired: the exporter pre-checks existence and skips vanished entries
  (Exists must positively report the file gone — transient errors keep
  today's abort), the network agent skips instead of failing, and skipped
  rows are counted in /status and pruned.

- Dead code wired: SweepStaging (hub staging DoS guard) now runs hourly
  (edge_sync.staging_sweep_max_age_hours, default 72); PruneSynced — also
  never called — and the new PruneSkipped run twice daily on the spoke
  (edge_sync.spoke.ledger_retention_days, default 90).

- Compaction row-duplication on the hub (audit High) is documented with
  loud warnings in the release notes and docs; the hub-side supersede fix
  is tracked in #610. Remaining audit findings filed as #611-#616.

Live gates on a real two-node rig, all previously-broken cells: sync
through an auth-enabled cap-3 hub drains 8/8 at batch_size=0; quoted
hyphenated queries return real rows on the hub; a discovered-then-deleted
file lands skipped; the tokenless 401 names its remedy; export with a
vanished file produces a valid bundle and completes import + ack; the
replacement-scan shapes are rejected over HTTP with explicit errors.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant