Repository navigation
Sync: tracking issue and v0.14.0 release triage #1445
Description
Activity
Correction to the table above: aw-server-rust#688 is not "daemon exits on first error" — exiting is the intended failure signal, and both aw-qt and aw-tauri implement supervision for it (3 restarts with backoff, then a dialog carrying the module log).
The issue has been rewritten: the defect is that peer-scoped and bucket-scoped errors are promoted to whole-process failures, so one unreadable peer db burns the restart budget and disables sync for every peer. Still worth having in v0.14.0 for the same reason as before — ActivityWatch/aw-server-rust#685 makes the daemon the path that actually walks every peer each cycle — but it is a smaller, better-targeted change than the original framing implied.
Design review: results
Two independent reviews of aw-sync, then a cross-review where each critiqued the other's conclusions. One designed clean-slate from requirements without reading
aw-sync/; the other reviewed the existing implementation against the filed bugs. The triage in the issue body above is superseded by the plan at the end of this comment — two hard gates were found that it does not account for.They converge (treat as settled)
Reached independently, from opposite directions:
device_idis the identity; hostname is metadata. Both arrived at{device_id}/+ a per-device manifest.- sqlite is the wrong wire format — one argued torn snapshots under a file syncer, the other that peers are opened read-write and migrated. aw-sync publishes aw-server-rust's private sqlite schema as the wire format (unreadable by aw-server-python, unversioned, WAL-mutable) aw-server-rust#691 adds a third: it is aw-server-rust's private schema, unreadable by aw-server-python.
- Full CRDTs are unwarranted. One writer per bucket means no concurrent edit exists to merge; the only real conflict is path conflict on the syncer.
- Observability is a design property. A pass returns
(), so nothing can report what it did (aw-sync: return and persist a SyncReport — a pass is a side effect, so nothing can report what it did aw-server-rust#695). That is why aw-sync:daemon(the default subcommand) never pulls — two incompatible sync-folder layouts aw-server-rust#682 ran 102 passes unnoticed — not that nobody looked, but that there was nothing to look at. - First three steps: aw-sync: return and persist a SyncReport — a pass is a side effect, so nothing can report what it did aw-server-rust#695 (report) → aw-sync writes to peer databases on every pull: WAL flip + schema migration on files it does not own aw-server-rust#693 (read-only peers) → device_id + manifest. Both rankings, independently.
The decomposition that clarifies the disagreement
The clean-slate reviewer conceded the other's bug-attribution split and corrected its own framing: it had conflated two independent decisions.
- Wire format — sqlite staging vs immutable segment files.
- Import vs read-in-place — whether peers' events enter the local database.
Splitting them resolves most of the argument. Torn snapshots and migration-on-open are wire-format bugs, not import bugs. ID rewriting, laundering and late-event loss are import bugs. And read-in-place depends on the segment format — you must never open a peer's sqlite at query time, so federation cannot come first.
Also corrected, and worth stating plainly: the import column did hurt — doubled timelines in discussions#1373, and ActivityWatch/aw-server-rust#683 is real data loss. What came from the other column was the silence.
Two gates on ActivityWatch/aw-server-rust#685 — both verified, neither previously noted
Gate 1 — peer database version spread. Measured on a live 5-device folder: 58 of 63 peer databases are at
user_version4, againstNEWEST_DB_VERSION = 6.create_datastoreopens peers read-write with migrations enabled (ActivityWatch/aw-server-rust#693). Today the daemon never pulls, so this is latent; ActivityWatch/aw-server-rust#685 is what makes every daemon pull for the first time. The first release shipping v6 would have every upgraded device run a v4→v6 migration on files it does not own, inside a replicated directory, simultaneously.The obvious mitigation is not available as-is:
datastore.rs:447—db_version != NEWEST_DB_VERSION→Err. Exact equality, so disabling migration refuses all 58 peers.datastore.rs:1039,1044—INDEXED BY events_bucketrow_{endtime_starttime,starttime_endtime}_index. A v4 file hasevents_bucketrow_index/events_starttime_index/events_endtime_index; neither hinted index exists, so reads fail "no such index" even with the version check relaxed.
So ActivityWatch/aw-server-rust#693's read path needs a genuinely tolerant reader (
user_version <= NEWEST, no index hints on older files, never_create_tableson a foreign db). That is a work item, not a flag flip.Gate 2 — ActivityWatch/aw-server-rust#692, and a correction to my own earlier recommendation. I proposed sanitize-on-import. That is wrong: anyone who pulled before ActivityWatch/aw-server-rust#658 has
…-synced-from-POCO F8 Ultralocally, sanitizing constructs a different ID,get_or_create_sync_bucketmisses it, resume starts from zero — full re-import, doubled timeline. The discussions#1373 symptom, reintroduced. The destination ID must resolve as existing legacy ID if present → otherwise sanitized.Revised plan
v0.14.0 — ActivityWatch/aw-server-rust#685 · ActivityWatch/aw-server-rust#692 (with legacy-ID lookup) · minimal read-only peer opens from ActivityWatch/aw-server-rust#693 · per-peer isolation from ActivityWatch/aw-server-rust#688 (one bad peer must not abort the pass, which matters precisely when pulls start happening) · ActivityWatch/aw-server-rust#687 (needs rebase) · ActivityWatch/aw-server-rust#690.
Both reviewers independently said ActivityWatch/aw-server-rust#685 must not ship alone: a silent no-op becoming an active importer, with known laundering bugs and a fleet-wide foreign migration, is a regression rather than a repair.
Next — ActivityWatch/aw-server-rust#695
SyncReport(key it bydevice_idso the manifest doesn't change its schema) · manifest +device_ididentity · ActivityWatch/aw-server-rust#694 mark imported buckets derived and read-only.Later — derived rebuildable cache · segment wire format (ActivityWatch/aw-server-rust#691) · read-in-place. Note the ordering correction: read-in-place must follow the segment format, not precede it. The datastore routing can come earlier and is cheaper than feared — a
buckets.origincolumn and ~6 dispatching methods in the existing worker, withaw-querykeeping its concrete&Datastore.Open decisions
Layout — resolved, not deferred.
{hostname}/{device_id}/puts mutable metadata in an immutable path: renaming a device moves its directory and undoes the identity fix. Canonical should be{device_id}/+ manifest, with readers rendering a hostname tree from manifests. Practically: R2 writes a manifest into whatever layout exists, readers treat depth as a legacy detail, and the write layout changes once — together with the segment format, not twice. ActivityWatch/aw-server-rust#685 can merge without settling this, provided it keeps reading root-level 2-level databases as a legacy source (otherwise upgraded devices go blind to not-yet-upgraded daemon peers — pure-daemon meshes did see each other).Import vs read-in-place — decidable later. Both agree the first three steps are prerequisites either way, so this can be chosen after identity lands, with data in hand.
Manifest worth adopting now (inside today's sqlite staging)
Minimum fields:
v,device_id,generation(monotone),hostname,app_version,published_at,retired,seen: {peer_id: gen}, and per bucket{bucket_uid, id, type, client, hostname, created, file, published, partitions: [{p: "YYYY-MM-DD", g}]}.Three properties that pay off immediately, before any format change:
- A reader that last applied generation N imports exactly the day-ranges with
g > N— exact, late-event-proof, skip-tolerant. This supersedes the event-level watermark in aw-sync: no stored cursor — resume-from-destination silently never syncs late-arriving events aw-server-rust#696 and fixes the same bug. - Written atomically as the last step of a push, the manifest doubles as the "push complete" marker — the back-channel the folder has never had.
- The
seenmap is the only way a device can answer "did my data reach X?" — roughly ten lines, and it is what aw-android#274 needs.
A
formatslist (sqlite-datastorenow,segments-v1later) lets a device dual-publish during transition, so this identity work is not discarded when the format changes.
Full documents are available on request. Filed from this review: ActivityWatch/aw-server-rust#692 ActivityWatch/aw-server-rust#693 ActivityWatch/aw-server-rust#694 ActivityWatch/aw-server-rust#695 ActivityWatch/aw-server-rust#696, ActivityWatch/aw-webui#982, ActivityWatch/aw-android#274, ActivityWatch/aw-server-rust#691.
Execution plan — decisions, not options
@TimeToBuildBob this is the order. Each PR small, independently green, reviewed in sequence. Erik reviews in this order.
Correction to my own comment above: I framed ActivityWatch/aw-server-rust#693 as a gate that should hold ActivityWatch/aw-server-rust#685. It is not. The v5→v6 migration is additive (one
CREATE INDEX IF NOT EXISTS, no drops), so the "owner reopens a file someone else migrated and breaks" scenario does not happen for the versions that are live. The 58 v4 files are stale (July and earlier). ActivityWatch/aw-server-rust#693's v0.14.0 slice is a ten-line companion change, not a blocker. Merge ActivityWatch/aw-server-rust#685.v0.14.0 — in this order
1. ActivityWatch/aw-server-rust#687 — rebase onto master, then merge. Additive, read-only, no existing path changes shape. Gives everyone
aw-sync statusto verify the rest with.2. ActivityWatch/aw-server-rust#685 — add two things in-PR, then merge:
- read root-level
{device_id}/*.dbas a legacy source, deduped bydevice_idagainst the 3-level entry, picking by newest data not size (otherwise the old file wins on size forever) - on first run, if an own root-level db exists and no own 3-level db does, rename it into place — one metadata op, no 3.7M-event re-export, dissolves aw-sync: leftovers after #685/#686 — orphaned 2-level staging db, stale -synced-from- buckets, walker enters dot-dirs aw-server-rust#689 §1
3. ActivityWatch/aw-server-rust#693-minimal — new small PR, after ActivityWatch/aw-server-rust#685 (both touch
create_datastore):- open peer databases with
?mode=ro&immutable=1 - never call
_create_tableson a foreign db - on
user_version != NEWEST_DB_VERSION, skip that peer with a warning ("peer X is on datastore v5; will sync once it upgrades") — do not build a tolerant reader; that is throwaway work given the format change below
Consequence to know about: a v6 desktop skips a v5 peer until that peer upgrades and pushes once. aw-android v0.14.2b1 embeds
5e67ac8c, which predates ActivityWatch/aw-server-rust#676 and is therefore v5 — so the next Android build needs the submodule bumped pastfde0a48, and until then a master desktop will list the phone as skipped rather than import it. Visible, self-resolving, correct.4. ActivityWatch/aw-server-rust#692 — resolve the destination ID as existing legacy ID if present → otherwise sanitized; sanitize the hostname field and new IDs only. Rust destinations only, small population, ~20 lines.
5. ActivityWatch/aw-server-rust#688 — per-peer isolation only: one unreadable peer logs and skips, the pass continues. Leave the whole-run exit contract as-is (aw-qt/aw-tauri supervise it correctly). Full error classification can wait.
6. Bump the aw-server-rust pin, cut v0.14.0. Sync ships as "works, not advertised". The release note gets one line: first daemon cycle after upgrading backfills from every peer.
Not in v0.14.0: ActivityWatch/aw-server-rust#690 README (do it, but it should describe the post-ActivityWatch/aw-server-rust#685 behaviour, so write it after ActivityWatch/aw-server-rust#685 lands); ActivityWatch/aw-server-rust#689 orphan cleanup (the rename in ActivityWatch/aw-server-rust#685 handles the own-db case; foreign orphans wait for
statusto report them first); everything below.After v0.14.0 — the format change (decided)
We are changing the wire format. Immutable, content-addressed, time-partitioned segment files — JSON Lines + zstd, one file per UTC day, compacted by the owner into one per month — in a new
aw-sync-v2/namespace under the sync dir, with{device_id}/directories and a per-device manifest. Measured on a real 1M-event bucket: 261 MB sqlite → 14.6 MB. Coexists withtest.dbstaging; readers accept both until every device has upgraded, then the old layout is retired. No big-bang migration.What this kills by construction:
test.db, WAL sidecars in foreign directories, migration-on-open, torn snapshots, the private-schema wire format (ActivityWatch/aw-server-rust#691), aw-server-python incompatibility, resume-from-newest and its late-event loss (ActivityWatch/aw-server-rust#696). Deletes and edits come free via range-replace.Sequence, each a PR:
- aw-sync: return and persist a SyncReport — a pass is a side effect, so nothing can report what it did aw-server-rust#695
SyncReport— returned fromsync_run, persisted, keyed bydevice_id. Unblocks aw-sync: sync failures are silent — addaw-sync status, loud diagnostics, per-device manifest, and surface peers in Raw Data aw-server-rust#684, aw-android#274, and givesstatus"what the last pass did". Start now; needed under any format. - Manifest — written atomically as the last step of a push, immutable, named
<generation>-<hash>. Fields per the design comment above:v, device_id, generation, hostname, app_version, published_at, retired, seen, formats[], buckets[{…, partitions:[{p, g}]}]. Theformatslist lets a device dual-publish during transition. Start now; needed under any format. - Segment writer — owner-side, from
sync_dirtytriggers, intoaw-sync-v2/devices/{device_id}/buckets/{bucket_uid}/. - Segment reader + import — generation-diff import (
g > applied), replacing the day's range. This replacessync_one. - Imported (-synced-from-) buckets are writable and unmarked; local writes to them are silently lost aw-server-rust#694 — imported buckets marked derived + read-only, rebuildable.
- Retire
test.dbonce every peer in a folder publishes v2.
Read-in-place (peer buckets served from segments without import) is the step after 6, and only after an Android decode benchmark. Not committed yet.
Layout question: closed
{device_id}/+ manifest is canonical in v2. v1's 3-level layout is read forever as legacy. No decision needed from Erik on this.- read root-level
Acknowledged. Working this in the order given — each PR small, independently green, reviewed in sequence.
Local tracking task:
activitywatch-issue-1445. Snapshot as of this comment:Step Item State now 1 ActivityWatch/aw-server-rust#687 OPEN, CONFLICTING vs master after ActivityWatch/aw-server-rust#686 squash ( 876709e). Rebase is this session.2 ActivityWatch/aw-server-rust#685 OPEN, MERGEABLE, CI green at 23e17f9. Still needs the two in-PR additions (legacy root-level{device_id}/*.dbby newest data; first-run rename of own root-level db).3 ActivityWatch/aw-server-rust#693 OPEN. Minimal slice after #685. 4 ActivityWatch/aw-server-rust#692 OPEN, already in flight in another session. 5 ActivityWatch/aw-server-rust#688 OPEN. Per-peer isolation only. 6 pin bump / cut v0.14.0 gated on 1–5 Out of v0.14.0 as specified: ActivityWatch/aw-server-rust#690 (README after #685), ActivityWatch/aw-server-rust#689 foreign orphans, format-change sequence (ActivityWatch/aw-server-rust#695 + manifest first).
Starting the ActivityWatch/aw-server-rust#687 rebase onto
origin/masternow: drop the already-squashed #686 stack commits, keep the twostatuscommits.Step 1 is done. ActivityWatch/aw-server-rust#687 is rebased onto master (
42d6128), MERGEABLE, and CI is fully green (including coverage). I cannot self-merge — pull-only on this org — so it needs a maintainer click.Next in the sequence is the two in-PR additions on ActivityWatch/aw-server-rust#685 (legacy root-level
{device_id}/*.dbby newest data; first-run rename of own root-level db). A sibling session already holds that claim, so I am not double-editing the PR.ActivityWatch/aw-server-rust#695 (
SyncReport) and the manifest remain the "start now" items off the v0.14.0 critical path; I will pick those up once the #685 claim is free rather than racing it.Plan revision (after ActivityWatch/aw-server-rust#687 merged)
ActivityWatch/aw-server-rust#685 is parked for v0.14.0. The daemon switch is right but targets the interim 3-level layout; all its remaining work (legacy 2-level reading, first-run rename, then ActivityWatch/aw-server-rust#692 + ActivityWatch/aw-server-rust#693-minimal to make the activated pulls safe) is scaffolding for a layout v2 retires. The daemon will switch directly to
devices/{device_id}/with v2. Default-bundle users lose nothing relative to today; withstatusmerged the failure is visible rather than silent.v0.14.0 — revised
step item state 1 ActivityWatch/aw-server-rust#687 statusmerged 2 pull-only never stages in peer folders — split from #685 new small PR 3 ActivityWatch/aw-server-rust#697 (#692 + per-bucket non-fatal) needs the sanitizer to match DeviceHostname.ktbyte-for-byte, and per-peercontinueinpull_all(review on the PR)4 ActivityWatch/aw-server-rust#693 minimal — mode=ro&immutable=1, never migrate foreign, skip + warn on version mismatchnew small PR 5 bump pin, cut v0.14.0 Erik Steps 3–4 are no longer gated on #685; they matter because Android and
aw-sync syncpull today. Note ActivityWatch/aw-server-rust#687's mergedsync_runstill returnsErrwhen a peer db fails to open, so step 3's per-peercontinueis what makes step 4's "skip" actually skip.v2 format — three decisions from Erik, now fixed
Layout. Top-level
devices/{device_id}/, not anaw-sync-v2/namespace. It cannot collide with either legacy layout ({hostname}/…and{device_id}/test.db), and the format version lives in the manifest'svfield, not the path.<syncdir>/ devices/<device_id>/manifests/<gen>-<hash>.json devices/<device_id>/buckets/<bucket_uid>/<partition>-<hash>.awseg.zst <hostname>/<device_id>/test.db ← legacy 3-level: read forever, never written by v2 code <device_id>/test.db ← legacy 2-level: sameLegacy
test.dbis the legacy format, tolerated indefinitely. Rule: v2 code never deletes anything it did not write.aw-sync statusreports legacy directories and their owners, and says when every device in a folder publishes v2. Cleanup is one explicit command —aw-sync cleanup --legacy, confirmation required — that removes only this device's own legacy directories. Foreign legacy dirs are cleaned by their owners, or by hand per the README. Nothing ever wipes a folder.The format change itself. The case, in one line: the wire format is a mutable database and the transport's contract is "I copy bytes when they change" — those cannot coexist, and every mitigation in the current code is fighting the transport. Immutable content-addressed files (no path is ever written twice) are what Syncthing, Dropbox and rsync are optimised for. Compression, deletes-for-free, Python compatibility and the death of
test.dbare consequences, not reasons. One open detail to settle before the writer is built: today's partition rewritten every 5 minutes means the syncer sees a new file ~288×/day, which is unpleasant with file versioning on — prefer publishing the current day less often over the designer's hourly-partition escape hatch.Sequence unchanged: ActivityWatch/aw-server-rust#695
SyncReportand the manifest first (needed under any format), then segment writer, reader + generation-diff import, ActivityWatch/aw-server-rust#694 derived read-only buckets, then legacy retirement per device by choice.Added to the v0.14.x queue: ActivityWatch/aw-server-rust#678 (7-day reconciliation of owner-side event edits), after ActivityWatch/aw-server-rust#697. Fixes ActivityWatch/aw-android#253 on the legacy format now; deleted when v2's generation-diff import replaces
sync_one.Design note for the record, since #253 is the clearest case for v2: owner-side edits and deletes propagate in v2 via day-partition range-replace — any age, no window, no per-event identity on the wire. Non-owner edits (editing another device's events from this UI) stay unsupported in v2: derived buckets are read-only and return
409naming the owner. A request-file mechanism is sketched for later, deliberately not built until someone needs it.Acknowledged the revised plan. ActivityWatch/aw-server-rust#685 stays parked; no in-PR additions.
step item now 1 ActivityWatch/aw-server-rust#687 statusmerged ( 988e606)2 pull-only never stages in peer folders ActivityWatch/aw-server-rust#698 opened (split from #685, no daemon switch). MERGEABLE, CI in flight. 3 ActivityWatch/aw-server-rust#697 OPEN, MERGEABLE, CI green at 67eeee7. Sibling holds that review lane.4 ActivityWatch/aw-server-rust#693 minimal this session: mode=ro&immutable=1, never migrate foreign, skip+warn on version mismatch. Independent of #685.5 pin bump / cut v0.14.0 Erik + ActivityWatch/aw-server-rust#678 after #697. Rebase-check already posted on the PR: 7/7 commits replay clean vs current #697 head. Out of v0.14.0 unchanged: ActivityWatch/aw-server-rust#690 README, ActivityWatch/aw-server-rust#689, format-change sequence (ActivityWatch/aw-server-rust#695 + manifest first).
Step 4 opened: ActivityWatch/aw-server-rust#700 (
mode=ro&immutable=1, never migrate foreign, skip+warn on version mismatch). Independent of ActivityWatch/aw-server-rust#685. CI in flight.Queue reorder after ActivityWatch/aw-server-rust#698 merged: ActivityWatch/aw-server-rust#697 and ActivityWatch/aw-server-rust#699 now conflict with master (both touch the
sync_runregion #698 rewrote) and need rebasing. ActivityWatch/aw-server-rust#678 is green and mergeable but based on626af70, 10 commits behind — rebase + CI rerun requested first.Order for Erik is now: #700 → #678 (once rebased) → #697 (once rebased + sanitizer fix). #699 rebases once over all three, after the cut.
Plan insertion — bucket identity. Erik's end state is buckets unique on
(device_id, id)with the hostname out of the ID; full proposal on #302 (comment just posted). One sequencing rule falls out of it and is now part of the v2 sequence here:The datastore
(device_id, name)change must land before v2's import step — otherwise v2 imports into the old model and every imported bucket migrates twice.Revised v2 sequence: ActivityWatch/aw-server-rust#699
SyncReport→ manifest → segment writer →(device_id, name)identity (datastore +/api/0/devices, both servers, names untouched) → reader + import (writes(device_id, id)directly, no-synced-from-ever again) → ActivityWatch/aw-server-rust#694 → legacy retirement. API/query/webui follow via a compatibility resolver.This also retires ActivityWatch/aw-server-rust#692's whole class: with the hostname out of the id there is nothing to sanitize.
Acknowledged both comments.
Queue after ActivityWatch/aw-server-rust#698 merged. Order for Erik is now ActivityWatch/aw-server-rust#700 → ActivityWatch/aw-server-rust#678 (rebased) → ActivityWatch/aw-server-rust#697 (rebased + sanitizer). ActivityWatch/aw-server-rust#699 waits for the cut — not rebased this pass.
step item now 1 ActivityWatch/aw-server-rust#687 statusmerged ( 988e606)2 pull-only never stages in peer folders ActivityWatch/aw-server-rust#698 merged ( 5c12d3d)3 ActivityWatch/aw-server-rust#700 read-only peers OPEN, MERGEABLE (sibling lane; immutable=1follow-up in flight)4 ActivityWatch/aw-server-rust#678 edit reconcile rebased onto master ( 1379d77), MERGEABLE, CI green5 ActivityWatch/aw-server-rust#697 sanitizer + isolation rebased onto master ( 8037907) over ActivityWatch/aw-server-rust#698'ssync_runrewrite. MERGEABLE, CI in flight. Sanitizer already matchedDeviceHostname.kt;pull_allper-peercontinuekept.6 pin bump / cut v0.14.0 Erik after cut ActivityWatch/aw-server-rust#699 SyncReportstill CONFLICTING; rebase once over ActivityWatch/aw-server-rust#700, ActivityWatch/aw-server-rust#678, and ActivityWatch/aw-server-rust#697 as specified Bucket identity (#302): recorded. v2 sequence is now:
ActivityWatch/aw-server-rust#699
SyncReport→ manifest → segment writer →(device_id, name)identity (datastore +/api/0/devices, both servers, names untouched) → reader + import → ActivityWatch/aw-server-rust#694 → legacy retirement.The datastore change lands before v2 import so imported buckets are never suffix-migrated twice. First concrete PR after the v0.14.0 cut:
buckets.device_id+UNIQUE(device_id, name)+/api/0/devicesin both servers. That also retires the ActivityWatch/aw-server-rust#692 class — hostname out of the id, nothing to sanitize.Queue status: all three rebased/updated and green, each now needs exactly one more push (reviews on the PRs). ActivityWatch/aw-server-rust#700 — gate backslash rewrite on platform. ActivityWatch/aw-server-rust#678 — no
unwrap()inreconcile_updated_events. ActivityWatch/aw-server-rust#697 — sanitize unconditionally (case-only fork,PIXEL8→pixel8) and make total failure returnErr. Order unchanged: #700 → #678 → #697. Also: maintainer@greptileai reviewconfirmed working on #697; Bob's account cannot trigger it.The three one-more-pushes are in. Order unchanged: ActivityWatch/aw-server-rust#700 → ActivityWatch/aw-server-rust#678 → ActivityWatch/aw-server-rust#697.
step item now 3 ActivityWatch/aw-server-rust#700 sibling pushed 3c1eeb1— Windows normalisation gated oncfg!(windows)4 ActivityWatch/aw-server-rust#678 eb7e751— reconcile returns Result, no unwrap5 ActivityWatch/aw-server-rust#697 fcd851c— sanitize unconditionally; total failure is Err6 pin bump / cut v0.14.0 Erik CI in flight on the last two. I cannot self-merge (pull-only on this org). ActivityWatch/aw-server-rust#699 still waits for the cut.
All three verified on their new heads and handed to Erik: ActivityWatch/aw-server-rust#700 (
3c1eeb1, all green) → ActivityWatch/aw-server-rust#678 (eb7e751) → ActivityWatch/aw-server-rust#697 (fcd851c). Also mergeable now and independent of that order: ActivityWatch/aw-server-rust#701 (README), ActivityWatch/aw-server-rust#702 (dot-dirs), ActivityWatch/aw-webui#969, ActivityWatch/aw-webui#983. Once these are in, the v0.14.0 sync slice is complete; the pin bump is the only remaining step before the cut.4 remaining items
v0.14.0 sync slice complete — every PR in the sequence is merged (ActivityWatch/aw-server-rust#686, #687, #698, #700, #699, #678, #697, #701, #702, #703, #706; ActivityWatch/aw-webui#969, #983; ActivityWatch/aw-android#275). ActivityWatch/aw-android#285 is green and merges on Android's own cadence.
Erik is doing the pin bump himself (steps 2–3: aw-tauri
Cargo.lockrelock of the six git-sourcedaw-*crates, then the bundle'saw-server-rust+aw-taurisubmodule pins, validated bycheck_tauri_server.py). @TimeToBuildBob — no bump PRs from your side, please; anything else in the queue (#685 parked, #699 follow-ups, the identity work on #302, the v2 format) waits for the cut.Submodule-bump friction, two PRs:
- feat(scripts): bump-submodules.sh — confirmed per-commit submodule bump with Tauri lock alignment #1447 —
scripts/bump-submodules.sh: one Enter per commit, walks nested submodules → direct submodules → aw-tauriCargo.lockrelock → bundle, pushing children before parents so CI never checks out an unpushed SHA. Encodes the policy that pure pointer bumps go straight to master (no PR). - ci(release): nightly build against the latest master of every submodule #1448 — nightly build against the latest master of every submodule: pull only, nothing committed; skips when nothing moved;
build-taurirelocks to the moved aw-server-rust;preflight/create-tagexcluded so it can never mint a prerelease.
Both Greptile-triggered. Erik is doing the v0.14.0 pin bump by hand this time; #1447 is the tool for the next one.
- feat(scripts): bump-submodules.sh — confirmed per-commit submodule bump with Tauri lock alignment #1447 —
b8 bar and the v2 restart (2026-09-18)
State: ActivityWatch/aw-server-rust#710 merged — the daemon can now see 3-level (Android) peers (ActivityWatch/aw-server-rust#709 was the original #1445 symptom surviving every fix because each was verified through
pull_all, not the daemon). Verified end to end on erb-m2 by one-shotaw-sync sync: 6 buckets, 1,436,166 events imported, incremental resume exact for window/unlock.Decision (Erik): b8 ships pull opt-in, default off. 0.14 is push-only everywhere unless opted in — Android by design (ActivityWatch/aw-android#291, re-scoped), desktop because ActivityWatch/aw-server-rust#711 is unfixed: the resume cursor re-imports the boundary event every pass (the phone's staging already holds 285 copies of one stopwatch event; a desktop web bucket gained an exact duplicate on its second pull). Turning pull on by default would spread that to every desktop at 5-minute cadence.
b8 = #710 (merged) + ActivityWatch/aw-server-rust#712 (own-device root-db test, doctor
imported locallyviasanitize_hostname) + ActivityWatch/aw-server-rust#714 (config.tomlpull = false, one-shotsyncunaffected). Then bump aw-server-rust in the bundle.After b8, in order: ActivityWatch/aw-server-rust#711 (idempotent boundary insert + one-off dedupe) → v2 restart. There is no v2 PR anywhere yet; the sequence from the 09-16 plan stopped after #695. First slice: manifest + segment writer (immutable JSONL + zstd under top-level
devices/{device_id}/, per-device manifest with generations andseenmap) behind a flag, never touching what the legacytest.dbpath writes. Then(device_id, name)identity (#302), then import. Once v2 reads from staging directly, "read from sync folder" replaces pull — on desktop and on Android alike.ActivityWatch/aw-server-rust#685 closed as superseded (pull half → #710, pull-only guard → #698, push layout → v2).
@TimeToBuildBob please post your task id and the planned first v2 PR here.
Task filed:
aw-sync-v2-manifest-segment-writer(tracks ActivityWatch/aw-server-rust#691, depends on this umbrella).Planned first v2 PR — additive-only, no interaction with the existing sync write path:
- Segment writer: immutable JSONL + zstd segment files under a top-level
devices/{device_id}/tree - Per-device manifest tracking segment generations + a seen-event-id map, so a reader knows what it's already imported without re-reading full segments
- Behind a feature flag, off by default
- Satisfies the [Linux Mint 20.1] Port issue - Failed to reopen ActivityWatch #691 constraints record: versioned in the folder, self-describing filenames, safe to publish into a directory an external syncer watches (write-once, never mutated in place)
- v2 never deletes or mutates anything the legacy
test.dbpath writes — the two coexist until the full sequence lands
Order after this:
(device_id, name)identity (#302) → v2 reader/import → #694 (refuse writes to imported buckets) →test.dbretirement. No(device_id, name)work in this first PR.Won't start implementation until #712 → #711 land per your order — will post the PR link here when it's up.
- Segment writer: immutable JSONL + zstd segment files under a top-level
Format spec drafted (workspace
knowledge/technical-designs/aw-sync-v2-segment-format.md). Summary for review before I write code:Directory layout — coexists with legacy test.db:
<sync_root>/ devices/ {device_id}/ manifest.json {bucket_id}.{gen:08d}.jsonl.zstSegment file — zstd-compressed JSONL:
- Line 1: header
{"v":1, "device_id":…, "bucket_id":…, "bucket_type":…, "generation":N, "written_at":…} - Lines 2+: events in the aw-core REST API format (
{"id":…, "timestamp":…, "duration":…, "data":{…}}), chronological order - Write-once: written to
.{name}.tmp, fsync, then atomic rename. Never mutated.
Manifest — JSON, atomic write+rename, lists all sealed segments with generation, filename, n_events, timestamp range, sha256 of compressed bytes. Version field
venables explicit version negotiation — an importer seeingv > MAX_UNDERSTOOD_VERSIONrefuses rather than opening blind.Importer protocol — reads manifest, compares segment generations against locally-tracked imports, downloads/verifies (sha256) only new generations, inserts events idempotently, records generation as done. Pure filesystem reads; no new HTTP/daemon protocol.
First PR scope —
SegmentWriter+Manifeststructs behind a Rust feature flag; three tests (immutability, manifest correctness, flag isolation). No call sites in existing push/pull/daemon flows yet.Open questions deferred to later slices: segment rollover policy, seen-event dedup equality key, partial-write resumption.
Raise anything that conflicts with your design intent before I start the implementation — will hold until ActivityWatch/aw-server-rust#712 and #713 merge per the stated sequencing.
- Line 1: header
@TimeToBuildBob The spec matches the intent (top-level
devices/{device_id}/, immutable segments via tmp+fsync+rename, manifest with generations + sha256 + explicitv, importer is filesystem-only, coexists withtest.db). Five things to settle in the format before the first PR, because each changes what a v1 segment/manifest means and would otherwise cost a format version bump later:-
Idempotency key = the source event id. Don't defer "seen-event dedup equality key":
(device_id, bucket_id, source_event_id)is stable (autoincrement in one datastore, never reused), so the importer only needs "highest source id imported per (device, bucket)" — no fingerprinting, no-1 mswindows. Today's aw-sync: resume cursor re-imports the boundary event every pass (inclusive start, ties at the newest timestamp) — duplicates grow on both push and pull aw-server-rust#711 is exactly what fingerprint-based idempotency costs. Keepidin every event line (you have it) and state in the spec that it is the identity. -
Edits and deletes need a first-class shape from v1. The stopwatch
running:true → running:falsecase, aw-android#253 (user edits propagating), and New Visualization #678's reconcile all need it. Decided design: range-replace — a segment header may carry"replaces": {"from": ts, "to": ts}meaning "for this bucket, the importer deletes what it previously imported in[from, to)and inserts this segment's events" (an empty segment withreplacesis a delete). Put the field in the header schema now; the writer need not emit it in the first PR. -
Bucket metadata in the manifest, not just
bucket_typein the header. The importer must be able to create the bucket:id,type,client,hostname,created, plus the future(device_id, name)fields from Change how hostname/devices work #302. Otherwise the reader has to invent them. -
Filenames must not carry raw bucket ids.
{bucket_id}.{gen}.jsonl.zstbreaks on ids with spaces/:/unicode (…-synced-from-POCO F8 Ultraexists in the wild) and on Windows reserved characters. Use a filesystem-safe slug or a short hash in the filename; the manifest maps filename → real bucket id. -
Rollover is a storage decision, not a later slice. One immutable segment per bucket per pass = 288 files per bucket per day at the 5-minute cadence; that is the Android-storage blow-up Erik wants to avoid (aw-android#291) and it makes Syncthing scan cost grow without bound. Decide now: a segment is sealed by size/time (e.g. ≥ 1 MB or a day), the open tail is rewritten under a new generation each pass, and compaction — a later generation that supersedes earlier ones, listed as such in the manifest — is the one case where v2 may delete files, and only its own. State the deletion rule explicitly in the spec: v2 never deletes or modifies anything outside
devices/{own_device_id}/, and inside it only superseded generations of its own.
Plus one line: timestamps in segments are RFC 3339 with full nanosecond precision, not the REST ms form — the ms-vs-ns mismatch is part of what made #713 hard to get right.
Hold order unchanged: ActivityWatch/aw-server-rust#712 → #713 → #716 land first, then this.
-
All five incorporated into the spec doc (
knowledge/technical-designs/aw-sync-v2-segment-format.md,537204f93d):- Idempotency key = source
id. Importer Protocol now tracks highest importedidper(device_id, bucket_id)— no fingerprinting. - Range-replace. Segment header gets an optional
"replaces": {"from": ts, "to": ts}field now; writer doesn't need to emit a non-null value yet, reader must handle it when a later slice needs it. - Bucket metadata in the manifest. Already had
type/client/hostname/createdthere; added a note that#302'sdevice_id/nameland as additive fields on the same manifest entry. - Filenames. Switched from sanitized-bucket-id to
sha256(bucket_id)[:16]slugs; manifest maps slug → real bucket id. - Rollover/deletion. New "Rollover and Deletion" section: seal on size/time threshold, open tail rewritten under the same generation between seals, compaction is the only case v2 deletes files (its own superseded generations only), explicit rule that v2 never touches anything outside
devices/{own_device_id}/.
Plus timestamps switched to RFC 3339 with nanosecond precision throughout (was ambiguous 6-digit micros before).
Holding on #712 → #713 → #716 before starting the writer PR, per your order.
- Idempotency key = source
@TimeToBuildBob Good — one consequence of "open tail rewritten under the same generation between seals" to pin in the spec, otherwise the first importer built to the letter will refuse the tail on its second pass:
- An unsealed generation is mutable by design (its sha256 changes every pass), so the immutability / content-address guarantee is a property of sealed segments only. Manifest entries carry
"sealed": true|false; a sealed entry's sha256 and byte length never change afterwards. - Importers may import from an unsealed tail (safe, because idempotency is by source
id) but must re-verify its sha256 against the manifest on every pass and re-read it when it differs — a mismatch on an unsealed entry is normal, a mismatch on a sealed entry is corruption and is refused. - Rewrite of the tail stays tmp + fsync + rename over the same name, so a reader never sees a torn file; manifest is written after the segment, so a manifest entry never references bytes that are not there yet.
With that, no objections to the format. Order unchanged.
- An unsealed generation is mutable by design (its sha256 changes every pass), so the immutability / content-address guarantee is a property of sealed segments only. Manifest entries carry
Pinned in the spec (
4667bb7f56):- Manifest segment entries now carry
"sealed": true|false. Sealed → sha256/length permanent (the immutability guarantee applies only to sealed entries). Unsealed → the current open tail, mutable by design. - Importer protocol updated: importing from an unsealed tail is fine (idempotency is by source
id, not segment hash), but the importer re-verifies sha256 every pass and re-reads on a mismatch. Mismatch on unsealed = normal (tail rewritten); mismatch on sealed = corruption, refused. - Rollover section now states the ordering guarantee explicitly: tail rewrite uses the same tmp+fsync+rename as sealing (no torn reads), and the manifest is written after the segment so it never references bytes that aren't on disk yet.
Doc:
knowledge/technical-designs/aw-sync-v2-segment-format.md, status flipped to Approved. Starting on the writer PR now.- Manifest segment entries now carry
@TimeToBuildBob Merged!
Confirmed — ActivityWatch/aw-server-rust#712, #713, and #716 are all merged. Gate is clear.
Task
aw-sync-v2-manifest-segment-writerreleased fromwaitingtotodo— will pick up implementation of the segment writer + manifest (per the approved spec) next, and post the PR link here when it's up.PR opened: ActivityWatch/aw-server-rust#719 — feat(aw-sync): v2 segment writer + manifest behind
sync-v2feature flagThis is the first concrete slice of aw-sync v2, implementing the segment writer and manifest module per the approved format spec (designed in session 11c2 earlier today).
What's in the PR:
aw-sync/src/v2/manifest.rs— manifest load/save (atomic rename), bucket slug derivation (sha256), version guard refusingv > MAX_Vaw-sync/src/v2/segment.rs—SegmentWriter: zstd-compressed JSONL segments, sealed/unsealed open-tail logic, SHA-256 over compressed bytesaw-sync/src/v2/mod.rs— feature module root- Feature flag
sync-v2inaw-sync/Cargo.toml(off by default — zero change to existing sync paths) - 9 tests covering slug stability, manifest round-trip, version guard, segment immutability after sealing, and two-segment rollover
No existing code paths modified. Awaiting maintainer review of ActivityWatch/aw-server-rust#719.
Local task:
aw-sync-v2-manifest-segment-writer- added a commit that references this issue
on Sep 18, 2026 Loop closure on the first v2 slice: ActivityWatch/aw-server-rust#719 (segment writer + manifest behind
sync-v2) merged 19:25Z, and ActivityWatch/aw-server-rust#718 (dedupesubcommand) merged 19:24Z, both after your review points landed (#719 head7e7c366, four review points; #718 paged fetch, progress and README caveat in4c40f27).Step 1 of your sequence is done. Task
aw-sync-v2-manifest-segment-writeris closed.What I am not starting: step 2,
(device_id, name)identity (#302). Your gate is no upstream PRs before the desktop v0.14.0 tag, so that lane stayswaitingon the stable tag. The v2 reader/import (step 3) depends on step 2, so I am not skipping ahead to it.Latest release is
v0.14.0b8(prerelease); no stablev0.14.0yet. I'll pick up #302 once that tag exists.
Tracking issue for the sync work opened over the last two days, with a recommendation on what should and should not land in the imminent v0.14.0 desktop release.
Context: v0.14.0b6 is cut as a draft; aw-android v0.14.1 is already Play Store
Latest. The desktop bundle ships aw-sync via the aw-server-rust submodule, so whatever is merged there before the final pin bump ships in v0.14.0.Everything found
aw-sync daemon(the default) never pulls — two incompatible folder layoutsdevice_idsilently truncate historystatusdoctor, manifest, peers in Raw Data-synced-from-buckets, dot-dir walkingID: undefinedoncedata.device_idis populated; device-ID check is dead code!localhas zero callers, 0/32 buckets carry itRecommended for v0.14.0
All bug fixes against already-shipped behaviour, none of them migrations:
daemon(the default subcommand) never pulls — two incompatible sync-folder layouts aw-server-rust#682) — without it desktop sync is push-only into a directory nothing reads. This is the release-blocking one.isSyncEnabled()defaultsfalse) and needs an explicit SAF directory pick, so the blast radius is users who deliberately turned sync on — which is exactly the population that would hit aw-sync: duplicate folders for one device_id silently truncate history on pull aw-server-rust#683 hardest.Merge order matters: ActivityWatch/aw-server-rust#686 → ActivityWatch/aw-server-rust#687 → ActivityWatch/aw-server-rust#685 rebased. ActivityWatch/aw-server-rust#685 conflicts with both others in
util.rs; ActivityWatch/aw-server-rust#686 and ActivityWatch/aw-server-rust#687 compose cleanly. See the review comments on each PR.Land if ready, otherwise defer
statusdoctor command and fail-loud empty-pull warnings aw-server-rust#687 — additive: a new read-only subcommand plus extra warnings, no existing path changes shape. High support value for exactly the users who will hit sync trouble in this release. Not worth holding the release for.Defer past v0.14.0
Everything here either rewrites data or moves files inside a Syncthing mesh. That is not rollback-safe — downgrading the app does not un-replicate a rename to every peer — so it wants a beta cycle, not a final release:
statusshowing what will happen first, which argues for doing it after feat(aw-sync): addstatusdoctor command and fail-loud empty-pull warnings aw-server-rust#687 has shipped and been used.device_idauthoritative for provenance. Worth doing, definitely not two days before a release.aw-sync status, loud diagnostics, per-device manifest, and surface peers in Raw Data aw-server-rust#684 items 3–6 —manifest.json, Raw Data peer surfacing, walker unification,test.dbrename.data.device_idis populated, which is deferred anyway. Cheap to fix whenever.Release-note item
After ActivityWatch/aw-server-rust#685, the first daemon cycle on an existing install performs a full backfill from every peer that has ever written to the sync folder. On my own mesh that is ~1M events / 273 MB from a single Android peer, plus several desktops. Expect sustained CPU and disk on first run after upgrading, and a large database growth. Worth a line in the release notes so it does not read as a regression — and a decent argument for shipping ActivityWatch/aw-server-rust#687 alongside, so users can run
aw-sync statusand see what is about to be imported.cc @TimeToBuildBob — ActivityWatch/aw-server-rust#688, ActivityWatch/aw-server-rust#689, ActivityWatch/aw-server-rust#690 and aw-webui#982 are unclaimed; ActivityWatch/aw-server-rust#690 is the smallest and aw-android#272 is the one nothing currently addresses.