Repository navigation
feat(edgesync): air-gap bundle export (#569) - #581
Merged
Merged
Conversation
PR 9b of 4 for the air-gap bundle. A spoke with no network path — a
submarine, a classified facility, a vehicle whose data comes off on a
drive — writes its files to a signed directory bundle on removable media.
9c adds the hub-side import, 9d the acknowledgment.
Two format decisions, both argued rather than assumed:
Replay protection is a bundle ID plus a hub-side dedup ledger, NOT a
timestamp window. checkSyncFreshness works for a request in flight where
the network guarantees "recent"; a bundle legitimately sits on a drive
for weeks, so a clock is the wrong instrument. A widened window would
leave replay protection off for exactly the period bundles exist.
A directory, NOT a tar. Resume falls out for free — an interrupted copy
leaves whole files and the manifest's per-file SHA identifies which
landed — and `ls` plus `sha256sum entries.jsonl` lets a human audit what
crosses the air gap without special tooling.
bundle-<spoke>-<ulid>/
manifest.json signed header: IDs, entries digest, MAC
entries.jsonl one JSON object per file, streamed not buffered
data/ the Parquet files under their original paths
Signing adds a third HMAC family, `sync-bundle` (length 11, distinct from
sync-file's 9 and sync-reconcile's 14). canonicalSyncInput is
length-prefixed, so no arrangement of one family's field contents can
produce another's byte string — tested in both directions. No nonce and
no freshness check, deliberately, with the reasoning in the code.
Where a bundle may be written is explicit. Every other Arc write path is
confined to the storage root by its backend; a USB mount is outside that
root by definition, so allowed_dirs is required and an empty list refuses
every export. Paths resolve through symlinks before the check, compare at
a segment boundary, and the storage root is refused outright — exporting
into it would make the next discovery pass queue the copies for sync.
Found by running the binary, not by tests: an air-gap-only spoke could
never export anything. Discover lived on the Agent, which such a spoke
never constructs, so its ledger stayed empty and every export answered
"nothing to export" while files piled up. Extracted Discoverer; the
exporter runs it before selecting.
Deep-review findings, all fixed:
- Air-gap-only spoke with no secret failed as a late fatal from an
internal constructor AFTER full startup, naming a Go type rather than
the env var. Now a config-load error.
- allowed_dirs = ["/"] refused every export: appending a separator to
the root yields "//". The mid-segment fix shipping its own edge case.
- Verify only walked data/, so an autorun.inf or decoy manifest beside
it verified clean. It now covers the whole directory — "verified" has
to mean the whole thing an operator is handed.
- BundleWriter returned a look-alike of ErrNothingToExport that
errors.Is did not match; negative sizes are rejected at the digest;
Mkdir on the leaf replaces stat-then-MkdirAll.
- status and ledger were gated on the agent, hiding the exported count
from the only operator who needs it. Both are pure ledger reads and
are now served by either side.
Test plan:
- [x] 12 tamper cases: content, truncation, missing file, unsigned file
under data/ and beside it, unsigned directory, symlink, MAC,
retargeted hub, swapped digest, edited entry, absent manifest
- [x] cross-family MAC rejection proven in both directions
- [x] DestinationPolicy: traversal, symlink ancestor, sibling prefix,
storage root, filesystem root, unconfigured
- [x] live air-gap-only spoke: 7 files at max_files=3 drained over three
bundles, 7 distinct files no repeats; sha256sum entries.jsonl
matched the manifest by hand
- [x] a real on-disk bundle with a smuggled autorun.inf refused
- [x] go test -race, go vet, gofmt clean
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up to review observations on the bundle export.
BundleReader.Open did not validate its argument, so Open("../secret.txt")
escaped data/ and read an arbitrary file — verified before fixing.
Unreachable today because Verify checks every declared entry first, but
Open is exported and the importer that will call it is a separate
component landing in the next PR. A traversal primitive that happens to
be unreachable is one refactor away from being reachable, and the caller
that would trip over it does not exist yet to be careful.
The Close obligation is now stated in the doc comment as well: an
importer streams thousands of files through this, so a missed Close
exhausts descriptors on the hub.
resolveDir's walk up to the nearest existing ancestor is now bounded at
32 missing levels. It always terminated at the filesystem root, so this
is not a hang — but it accepted a 200-level path that resolved to a 2KB
string, and every level costs a stat. A destination that far below an
existing directory is a typo or a hostile input, not a drive mount, and
refusing early gives a clearer error.
Test plan:
- [x] Open refuses ../, ../../, absolute, and a/../../ forms; a declared
entry still opens
- [x] 40-level path refused; a one-level not-yet-created destination
still resolves
- [x] go test -race, go vet, gofmt clean
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A spoke with no network path — a submarine, a classified facility, a vehicle whose data comes off on a drive — writes its files to a signed directory bundle on removable media.
edge_sync.spoke.bundle.enabledis independent ofedge_sync.spoke.enabled: a fully air-gapped spoke needs nohub_urland never runs the network path.Two format decisions
Replay protection is a bundle ID + hub-side dedup ledger, not a timestamp window.
checkSyncFreshnessworks for a request in flight, where the network itself guarantees "recent". A bundle legitimately sits on a drive for weeks, so a clock is the wrong instrument — and a widened window would leave replay protection off for exactly the period bundles exist.A directory, not a tar. Resume falls out for free (an interrupted copy leaves whole files; per-file SHAs say which landed), and
ls+sha256sum entries.jsonllets a human audit what crosses the air gap without special tooling.Signing adds a third HMAC family,
sync-bundle(length 11, vssync-file9 andsync-reconcile14).canonicalSyncInputis length-prefixed, so no arrangement of one family's field contents can produce another's byte string — tested in both directions. No nonce, no freshness check, deliberately.Where a bundle may be written is explicit. Every other Arc write path is confined to the storage root by its backend; a USB mount is outside it by definition. So
allowed_dirsis required, paths resolve through symlinks before the check, containment compares at a segment boundary, and the storage root is refused outright — exporting into it would make the next discovery pass queue the copies for sync.The bug only the binary run found
An air-gap-only spoke could never export anything.
Discoverlived on theAgent, which such a spoke never constructs — so its ledger stayed empty and every export answered "nothing to export" while files piled up on disk. The feature was inert in exactly the deployment it exists for. Unit tests passed because they built the exporter directly. Fixed by extractingDiscoverer.Deep-review findings, all fixed
allowed_dirs = ["/"]//. The mid-segment fix shipping its own edge case (#534 corollary).Verifyonly walkeddata/autorun.infor decoymanifest.json.bakbeside it verified clean. Now covers the whole directory — "verified" must mean the whole thing an operator is handed.ErrNothingToExportthaterrors.Ismissed; negative sizes now rejected at the digest;Mkdiron the leaf replaces stat-then-MkdirAll; raw error no longer returned to clients.status/ledgerwere gated on the agent, hiding theexportedcount from the only operator who needs it. Both are pure ledger reads, now served by either side.Test plan
data/and beside it, unsigned directory, symlink, MAC, retargeted hub, swapped digest, edited entry line, absent manifestDestinationPolicy: traversal, symlink ancestor, sibling prefix, storage root, filesystem root, unconfiguredmax_files=3drained over three bundles — 7 distinct files, no repeats;sha256sum entries.jsonlmatched the manifest by handautorun.infrefused with a message naming the problem/runstill 503s on an air-gap spoke whilestatus/ledgerservego test -race,go vet,gofmtcleanDocs
Basekick-Labs/docs.basekick.net#21 — states plainly what is not here: no import, no ack (so ledgers do not prune yet), and bundles are signed but not encrypted.
🤖 Generated with Claude Code