feat(categories): local manual event overrides outside watcher data

[TimeToBuildBob]

Summary

Implement the backend contract for single-event manual categorization without editing watcher data or changing heartbeat equality.

• SQLite v7 sidecar with deletion cleanup; upserts preserve annotations and reject cross-bucket ID collisions.
• Bucket-scoped GET/PUT/DELETE event category API with validated paths and commit-before-success acknowledgments.
• Query-only sidecar overlay wins over rules, including explicit Uncategorized; preaggregation and legacy chunking preserve category boundaries.
• Query-cache invalidation, CORS PUT, and manual_event_category capability flag in /api/0/info.

Boundaries

Local database only. No UI editor, bucket defaults, Python backend parity, or annotation transport through export/import/aw-sync. README documents retention and sync reconciliation limits. Category paths remain historical until explicitly reassigned or cleared.

Verification

• Regression tests reproduced preaggregation and legacy-chunk category loss before fixes.
• Full make test passed using a fresh isolated Cargo target, including doctests.
• cargo check --workspace, cargo clippy --workspace -- -D warnings, cargo fmt --all -- --check passed; commit hooks enabled.
• New tests cover heartbeat continuity, raw-data preservation, ownership, validation, durable writes, restart, deletion, query precedence, totals, AFK/union transforms, cache invalidation, capability flag and CORS.

Includes a separate minimal lint prerequisite commit removing pre-existing duplicated parser allowances rejected by current Clippy.

This is the backend prerequisite for the user-requested manual category editor, not a claim that the user-facing feature is delivered.

[greptile-apps]

Confidence Score: 4/5

[High risk] Adds database schema version and new event annotation storage.

The PR does not appear safe to merge until queries that rely on watcher-stored $category retain a usable path to that data.

Findings

1.  Stored categories disappear from queries ▶
2.  Automatic time gets miscategorized ▶

Summary

The PR adds local, bucket-scoped manual category overrides, overlays them during queries, and preserves category boundaries through aggregation. The latest revision removes watcher-stored $category from query_bucket results to prevent a pre-categorization merge from mistaking it for a computed category.

• That correction also removes the field from queries that do not categorize, breaking direct reads and grouping or filtering on stored categories.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Stored event data] --> B[query_bucket]
  C[Local category sidecar] --> B
  B --> D[Optional categorize]
  D --> E[Category-aware transforms]
  B --> F[Direct query result or key-based transform]

Loading

_{Reviews (6) · Last reviewed commit: "fix(categories): strip stored $category ..."}

[codecov]

Codecov Report

❌ Patch coverage is 99.13169% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.34%. Comparing base (656f3c9) to head (235fe35).
⚠️ Report is 195 commits behind head on master.

Files with missing lines	Patch %	Lines
aw-datastore/tests/manual_category.rs	98.31%	3 Missing ⚠️
aw-datastore/src/worker.rs	93.10%	2 Missing ⚠️
aw-transform/tests/manual_category.rs	99.35%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master     #782       +/-   ##
===========================================
+ Coverage   70.81%   85.34%   +14.52%     
===========================================
  Files          51       86       +35     
  Lines        2916    11947     +9031     
===========================================
+ Hits         2065    10196     +8131     
- Misses        851     1751      +900     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[TimeToBuildBob]

@greptileai review

[TimeToBuildBob]

Addressed both Greptile findings in 1fad8f4:

• P1 (category totals split) — merge_events_by_keys now folds the manual marker into the merge key only before classification. Once an event carries a final $category, a manual and an automatic event in the same category sum into one row; pre-categorization aggregation still keeps the groups separate. Regression test added.
• P2 (sidecar reads scan entire intervals) — get_events_with_categories now looks up overrides only for the returned event ids (chunked indexed lookup), so limited reads are proportional to their result set and honour limit. Regression test added.

Verified locally: cargo test --workspace green, cargo clippy --workspace -- -D warnings clean, cargo fmt --check clean. Re-triggered review.

[TimeToBuildBob]

@greptileai review

[TimeToBuildBob]

@greptileai review

[TimeToBuildBob]

The red windows-latest check on 1a8bb07 isn't from this PR. It's aw-client-rust test_wait_for_start_retries_without_blocking_executor, a crate this PR doesn't touch, and the same flake hit #780, #775 and #772 in the last day. Root cause and fix (test-only) are in #783. A rerun or rebase onto master after #783 should turn it green.

[TimeToBuildBob]

Status check on this PR: CI is green on the current head (eea7682) and the branch is CLEAN/MERGEABLE.

Two inline threads remain open, both on aw-transform/src/merge.rs. They are the same design question seen from two angles, not two independent bugs:

1. Merging by final $category collapses manual and automatic events into one row — the row keeps a single $manual_category, so a later re-categorize cannot re-derive the automatic portion.
2. A stored $category on watcher data is indistinguishable from one inserted by this query's categorize, so a merge-before-categorize query can merge events carrying different manual overrides.

A sound fix needs an unambiguous, query-scoped classification marker (rather than reusing $category), or manual provenance kept out-of-band from the merged row. That is a call on reserved-field semantics for aw-server-rust, so I left both threads open rather than adding another merge-key heuristic. If a simpler direction is preferred, sanitizing stored $category at query time in get_events_with_categories (mirroring the existing $manual_category handling) is a ~2-line change plus a datastore/query test.

No further Bob action here until that decision is made.

[TimeToBuildBob]

🤖 AI code review

Adds a SQLite v7 sidecar table for per-event manual category overrides, with a new bucket-scoped GET/PUT/DELETE API, a query-only overlay that strips watcher-supplied reserved keys and applies sidecars, and changes to merge/chunk transforms to keep manual categories separate during pre-aggregation. Also bumps the advertised DB version, adds a manual_event_category flag to /api/0/info, and adds CORS PUT support.

Safe to merge — no P0/P1 findings

Confidence 5/5

✅ No thread-worthy findings. Advisory notes follow; they are retained without opening review threads.

1 advisory finding (summary-only, not scored)

These P2 guard, heuristic, trade-off, or documentation claims are retained for judgment without opening review threads.

⚠️ P2 medium — aw-datastore/src/datastore.rs:917

In insert_events, the new ownership validation queries SELECT bucketrow FROM events WHERE id = ?1 for each event with an explicit ID. If the event ID does not exist, owner is None and the insert proceeds. If the event ID exists in another bucket, it returns NoSuchEvent. However, the validation is done inside the same transaction as the insert, but the worker's transaction is shared across multiple commands. The validation reads the current transaction's view. If a previous command in the same transaction inserted an event with that ID into another bucket, the validation would see it and reject. That is correct. But there is a race: the worker serializes all commands, so no race. The issue is that the validation loop uses events.iter().filter_map(|event| event.id) and for each id runs a separate query. If the same ID appears twice in the batch, the second occurrence is validated again, but the first insert has not happened yet (validation is before any insert), so both see the same state. Fine. The real problem: the ON CONFLICT(id) DO UPDATE sets bucketrow = excluded.bucketrow. If an event with the same ID exists in the same bucket, it updates. If it exists in another bucket, the validation rejects. But what if the batch contains two events with the same ID, both targeting the same bucket? The first insert creates/updates, the second updates again. That is fine. However, the validation only checks events with explicit IDs. Events with id=None are inserted as new rows. If a watcher sends an event with id=None but the data contains a $manual_category key, that key is stored in the events.data. Later, get_events_with_categories strips it, so it is not exposed. But raw get_events returns it. The PR says stored copies cannot spoof annotations because query_bucket strips them. That is true. But the sidecar table is keyed by event_id, and if an event is deleted and re-inserted with a new ID, the sidecar is gone. That is documented. I think the merge key change is the most concrete bug.

_{How this was verified: Traced the insert_events path and the worker's serialized transaction. No race is possible because the worker processes commands sequentially.}

_{Consensus: 2/3 passes agreed — pass 2 looked and disagreed}
Distinct keys: 1 (general)

Files changed (18) — the diff as I read it

• README.md — Documents the new local manual event category API, its semantics, and its non-portability across export/sync.
• aw-client-rust/src/lib.rs — Increases a test sleep from 30ms to 100ms to reduce flakiness in the wait-for-start retry test.
• aw-datastore/src/datastore.rs — Adds v7 migration creating event_category_overrides and a delete trigger, plus get/set/delete category methods and a query-only overlay that strips reserved keys.
• aw-datastore/src/lib.rs — Adds the InvalidCategory variant to DatastoreError.
• aw-datastore/src/worker.rs — Adds worker commands and responses for category get/set/delete and get_events_with_categories, with commit-before-ack for writes.
• aw-datastore/tests/datastore.rs — Updates a migration test to assert against NEWEST_DB_VERSION instead of hardcoded 6.
• aw-datastore/tests/manual_category.rs — Adds integration tests for category lifecycle, durability, read-only compatibility, cross-bucket collision, and limited reads.
• aw-query/src/functions.rs — Switches query_bucket to use get_events_with_categories so manual overrides are overlaid before transforms.
• aw-query/tests/manual_category.rs — Adds query-level tests for sidecar precedence, pre-aggregation separation, and clearing restoring rule behavior.
• aw-server/src/endpoints/bucket.rs — Adds GET/PUT/DELETE routes for /events/<event_id>/category with write-lock and query-cache invalidation.
• aw-server/src/endpoints/cors.rs — Adds PUT to the allowed CORS methods.
• aw-server/src/endpoints/mod.rs — Mounts the new category routes and adds manual_event_category: true to /api/0/info.
• aw-server/src/endpoints/util.rs — Maps DatastoreError::InvalidCategory to HTTP 400.
• aw-server/tests/manual_category.rs — Adds HTTP tests for bucket scoping, validation, cache invalidation, CORS PUT, and the info flag.
• aw-transform/src/chunk.rs — Makes chunk_events_by_key treat differing manual categories as a boundary so they are not merged.
• aw-transform/src/classify.rs — Adds manual_category() and gives it precedence over rules in categorize.
• aw-transform/src/merge.rs — Adds an effective-category component to the merge key so manual and automatic events stay separate unless they share the final category.
• aw-transform/tests/manual_category.rs — Adds transform tests for manual category precedence, pre-aggregation separation, and category-totals merging.

Previous review passes

commit	score	findings	engine	when
f0c12276de3f	5/5	0	llm	2026-10-07 13:37 UTC
f0c12276de3f	4/5	1	llm	2026-10-09 04:05 UTC
235fe35c6c3c	4/5	1	llm	2026-10-10 14:38 UTC

_{Reviewed 235fe35c6c3c · openrouter/deepseek/deepseek-v4-flash-0731 · llm engine · 101s · about this reviewer}

_{Maintainer commands}

_{@TimeToBuildBob review (own line) — fresh review · @TimeToBuildBob fix — a worker acts on the findings. Once per comment; 👀 = received.}

[TimeToBuildBob]

Rebased onto master (e277a8f) to clear the conflict.

I dropped dd73976 (fix(lint): remove duplicate parser lint allowances). #771 already fixed the same clippy::duplicated_attributes problem on master by moving block_scrutinee into parser.rs, so that commit was superseded and was the only source of the conflict. The other five commits applied without changes.

Locally: cargo clippy --workspace -- -D warnings is clean, cargo fmt --check passes, and the aw-datastore, aw-query, aw-transform and aw-server tests pass.

[TimeToBuildBob]

@greptileai review

[TimeToBuildBob]

@greptileai review

[TimeToBuildBob]

Status after today's rebase and latest Greptile review round (commit f0c1227):

Greptile P1 findings — both addressed:

1. Automatic time gets miscategorized (#discussion_r4200463337) — fixed in f0c1227: get_events_with_categories now strips a watcher-stored $category so it can only come from categorize; regression test added.

2. Stored categories disappear from queries (#discussion_r4200669335) — Greptile acknowledged the design: stripping stored $category in query context is intentional to prevent pre-categorize merges from conflating stored vs. computed categories. Greptile withdrew the finding.

Current state: CI is running on f0c1227. Greptile inline threads are resolved. The Greptile summary still shows 4/5 from the last review pass; a maintainer merge or a fresh re-review on the next CI green run will update it. All prior P1/P2 findings from earlier rounds are closed.

[TimeToBuildBob]

Merge recommendation (convergence adjudication)

Head verified: f0c12276de3f. Freshness gate passed: the newest AI-review marker and the latest Greptile reviews (2026-10-09 04:11) cover this head, and no commit postdates them.

Fixed (earlier rounds, all threads resolved): category totals split by manual marker; desktop/app summaries misattributing time; duplicate app totals; merged time losing category provenance; raw stored $category hiding manual overrides (stripped in query_bucket at f0c1227); sidecar reads scanning whole intervals (limit now honoured); stored categories disappearing from queries (intentional, documented in get_events_with_categories).

Remaining, non-blocking:

• P2 merge-key change (merge.rs:86, finding 5181ebb2bb39): dismissed as intentional. Split is only added when the grouping lacks $category. Category-total merges are unchanged, and app/title merges after categorize no longer collapse across categories. Replied on the thread and resolved. No code change.
• Open open-ended design call from the earlier thread (4171615413) on folding same-category manual/automatic rows stays recorded for maintainers.

Our open findings: none (ai-review-dispose --list). Re-render reports "Safe to merge — no P0/P1 findings on latest review".

CI: all checks pass on f0c12276 (ubuntu, macOS, windows, Android, clippy, format, coverage, Greptile Review).

Convergence: detector verdict converged with 0 unresolved P1/P2. round_convergence shows 0 stable rounds, status new_blocking, because the latest Greptile round still carried P1s from 2026-10-06. Those are all resolved now. Per the cap, no further @greptileai review triggered.

Domain risk: the query/merge path (aw-transform/src/merge.rs, aw-datastore get_events_with_categories) is fragile. Maintainer manual test: queries that merge by app/title after categorize with a mix of manual and automatic events, plus the desktop and Android category reports.

Decision for maintainer: whether $category should be visible inside query scope under another name ($watcher_category), per the reply on the stored-categories thread. Not a merge blocker.

Maintainer judgment: merge is not automatic. Ready for review.

[TimeToBuildBob]

Rebased onto master (04e8d66) and force-pushed with an explicit lease; new head is 235fe35. GitHub now reports the branch mergeable with no conflict.

The textual conflict was the datastore response enum: preserved both master's FilteredEvents and this PR's EventCategory variants.

Tests caught a semantic integration issue with #776: schema inference recognized only v6, so a restored v7 database would try to recreate the category sidecar. Fixed in 235fe35 by recognizing the sidecar as v7. The existing current-schema test failed before the fix; added coverage that v6 without a sidecar still infers v6. Restored-database tests pass as well.

Verified locally: full cargo test --workspace, cargo clippy --workspace -- -D warnings, and cargo fmt --all -- --check pass. CI on the new head is separate from the previous green run; the category/provenance design decisions discussed above are unchanged.

[TimeToBuildBob]

Merge recommendation (convergence adjudication)

Verified against: 235fe35c6c3c6d7fece56b7e40270e550a9319f8. The newest AI-review marker covers this exact head (2026-10-10 14:38 UTC); no newer commit was present. Greptile's summary footer still names f0c12276de3f, so its 4/5 summary is historical, not a fresh review of the rebased head.

Fixed this session: none; no code changes. Earlier fixes and the rebase/schema-inference integration are recorded in the preceding comments.

Remaining findings:

• AI P2 c0e401dd27a9 (duplicate-table migration after deleting the cleanup trigger): false positive, replied inline and resolved. _infer_db_version returns v7 from the sidecar table's existence, independently of the trigger. Dropping only the trigger cannot select the v6→v7 migration. Verified with the exact v7 migration SQL in in-memory SQLite, plus inspection of the inference and migration guards. No Rust tests rerun in this adjudication.
• Historical Greptile findings: all resolved; no new Greptile finding classified against the rebased head. The same-category manual/automatic merge loses provenance for a subsequent re-categorization; that previously recorded representation tradeoff remains for maintainer judgment, not another heuristic fix.

Our unresolved findings: none, confirmed with ai-review-dispose --list after disposition.

CI on this head: Windows and Android passed; Ubuntu, macOS, coverage, format and Clippy are queued. This is not a claim that current-head CI is green.

Domain risk: maintainer smoke-test mixed manual/automatic category reports and merge-before/after-categorize queries; confirm the intended reserved-field and re-categorization semantics. Automatic repair of a manually removed SQLite trigger is not provided or verified by this change.

Convergence: detector reports converged with zero unresolved Greptile P1/P2; round_convergence.stable_rounds = 0, status new_blocking (historical rounds). Thread disposition is not evidence of two stable review rounds. No review re-trigger and no auto-merge.

Recommendation: maintainer judgment after current-head CI completes; the remaining AI finding is not a blocker.