Skip to content

fix(sync): persist sync state across engine invocations - #53

Merged
sknr merged 1 commit into
okf-memory:developfrom
akushonkamen:fix/51-persist-sync-state
Oct 12, 2026
Merged

sknr merged 1 commit into
okf-memory:developfrom
akushonkamen:fix/51-persist-sync-state

Conversation

@akushonkamen

Copy link
Copy Markdown
Contributor

Fixes #51

Description

okf hub push|pull|sync each spawn a fresh sync.Engine per CLI invocation, but the engine's sync position (cachedTree / cachedHead, pkg/sync/engine.go:45-46) lived only in memory. This PR persists that position to a hidden state file inside the bundle so consecutive invocations behave like one continuous session.

  • New pkg/sync/state.go: {version, vault_id, head, tree} is persisted to <bundle>/.okf/sync-state.json. ScanBundle skips hidden directories, so the state file is never mistaken for bundle content. NewEngine loads it at startup (engine.go:61); missing, corrupt, or foreign-vault state silently degrades to the previous empty-start behavior.
  • saveState() performs an atomic write (temp file + rename inside the hidden dir, pkg/sync/state.go:107-125) and is called on the three success paths — push pkg/sync/engine.go:212, pull pkg/sync/engine.go:307, sync pkg/sync/engine.go:523.
  • TestHub_Serve_E2E_Socket (pkg/sync/hub_test.go:349-373) previously locked in a pseudo-conflict: device B's local edit was flagged as a collision with no remote change. The test now makes device A edit and push note_b.md before device B syncs, so the collision is genuine; all original conflict-fork assertions are preserved.

Motivation & Context

Root cause: Engine.cachedHead/cachedTree (pkg/sync/engine.go:45-46) are pure in-memory fields; NewEngine (pkg/sync/engine.go:50) starts empty. In a new process expectedHead is always nil (pkg/sync/engine.go:182), so the hub's compare-and-swap (pkg/sync/server.go:129, expected != current) rejects the second push with 409 head conflict. Similarly Reconcile (pkg/sync/engine.go) diffs against the empty cached tree, so on a single machine every local file looks like a collision and is moved to <file>.conflict-local.md — sync appeared to "revert" local edits.

See the triage in this comment.

Changes Made

  • Add pkg/sync/state.go — versioned sync-state schema, load (with corrupt/foreign-vault fallback) and atomic save
  • Load persisted state in NewEngine (pkg/sync/engine.go:61)
  • Persist state atomically after successful push / pull / sync (pkg/sync/engine.go:212,307,523)
  • Make the E2E hub test exercise a genuine concurrent-edit conflict instead of a pseudo-conflict (pkg/sync/hub_test.go)
  • Regression tests: TestEngine_PushAcrossEngineInstances_SecondPushNotConflict, TestEngine_SyncAcrossEngineInstances_LocalEditsKept, TestEngine_SyncStateFile_*

Verification & Pre-Submission Checklist

All commands below were executed on this exact tree (db7204a, fix/51-persist-sync-state).

  • make test passes with 100% test success — timeout 600 make test (go test -v -race ./...) → exit 0, all green
  • make validate passes on knowledge/ — 23 concept(s), 0 error(s), 0 warning(s), 0 orphan(s), Conformant
  • make validate-examples passes on all sample corpora — exit 0, all governance/symlink checks passed
  • make fmt and make vet show clean code — make fmt left the tree unchanged (git status clean); make vet exit 0; gofmt -l . no output; go vet ./pkg/sync/ clean

Red/green evidence for the fix

Red (before the fix, recorded during independent acceptance of this change): timeout 300 go test ./pkg/sync/ -run 'TestEngine_PushAcrossEngineInstances|TestEngine_SyncAcrossEngineInstances|TestEngine_SyncStateFile' -v → 3 FAIL (second push 409 head conflict; single-machine sync pseudo-conflict moving edits into *.conflict-local.md; state file missing), 1 PASS (ForeignVaultIgnored, which passes even empty-to-empty). Green (after the fix, re-run on this tree): same command → 4/4 PASS. Note: a -run TestPush… pattern matches 0 tests in this package (there are no TestPush* functions), so the two cross-engine-instance regressions above serve as the red/green proof.

Regression (re-run on this tree)

  • timeout 600 go test ./pkg/sync/ -count=1 → ok … 1.130s, 26/26 top-level tests PASS (13 pre-existing engine/vault tests + 6 hub tests + new regression tests)
  • go test ./... → 8 packages ok, 0 FAIL (cmd/okf-benchmark has no test files)
  • timeout 600 make test → exit 0 (with -race)

AI Disclosure

This change was produced by an AI-assisted contributor (see disclosure in the issue claim comment) working under human direction; the fix passed independent acceptance review before submission.

The hub sync engine kept cachedHead/cachedTree purely in memory, so
every `okf hub push|pull|sync` invocation started from an empty state:

- the second push sent no expected head and was rejected by the hub
  compare-and-swap with 409 head_mismatch;
- a subsequent sync reconciled against an empty base tree, so local
  edits to previously pushed files looked like incompatible concurrent
  modifications and were moved to `*.conflict-local.md` even with no
  other writer;
- every push re-encrypted and re-uploaded the whole bundle because
  change detection had no baseline.

Persist the last synced head and its tree manifest in
<bundle>/.okf/sync-state.json (hidden directories are skipped by
ScanBundle, so the state is never pushed as content). NewEngine loads
it on startup; push/pull/sync atomically rewrite it (temp file +
rename) after each successful commit. Missing, corrupt, or
foreign-vault state is ignored, degrading to the previous empty-state
behavior.

TestHub_Serve_E2E_Socket previously locked in a collision that was an
artifact of the lost state; it now edits and pushes on device A first,
so device B's sync hits a genuine concurrent modification.

Fixes okf-memory#51
@sknr

sknr commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

Merged into develop via commit db7204a.

Thank you @akushonkamen for the clean implementation, atomic state handling, and comprehensive regression test suite! You have been credited in CONTRIBUTORS.md.

cdcore09 added a commit to cdcore09/okf-agent-memory that referenced this pull request Oct 14, 2026
Creating different concepts in the same folder on two devices changes that folder's index.md on both sides. Only log.md had a semantic merge, so the shared index collided: the canonical index dropped one side's listing and the other was forked to index.conflict-local.md. Add a 3-way merge keyed by listing link target, preserving hand-written index text and falling back to the collision failsafe only when that text changed on both sides.

The base index comes from the last-synced tree that okf-memory#53 persists in .okf/sync-state.json.

Fixes okf-memory#52

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cdcore09 added a commit to cdcore09/okf-agent-memory that referenced this pull request Oct 14, 2026
Creating different concepts in the same folder on two devices changes that folder's index.md on both sides. Only log.md had a semantic merge, so the shared index collided: the canonical index dropped one side's listing and the other was forked to index.conflict-local.md. Add a 3-way merge keyed by listing link target, preserving hand-written index text and falling back to the collision failsafe only when that text changed on both sides.

The base index comes from the last-synced tree that okf-memory#53 persists in .okf/sync-state.json.

Fixes okf-memory#52
sknr pushed a commit that referenced this pull request Oct 19, 2026
Creating different concepts in the same folder on two devices changes that folder's index.md on both sides. Only log.md had a semantic merge, so the shared index collided: the canonical index dropped one side's listing and the other was forked to index.conflict-local.md. Add a 3-way merge keyed by listing link target, preserving hand-written index text and falling back to the collision failsafe only when that text changed on both sides.

The base index comes from the last-synced tree that #53 persists in .okf/sync-state.json.

Fixes #52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: okf hub loses sync state between invocations: second push gets 409, sync reverts local edits

2 participants