feat(storage): report Host storage usage and per-task sizes

[liugddx]

Part of #5825 (M1, storage visibility). Read-only: no data is changed, vacuumed or deleted.

Why

Maka gives users no way to see how much disk it uses or which tasks account for it. Step 1 of the task lifecycle plan (#5776) starts by making that visible. The Host measures and Desktop presents.

What changes

Runtime Host protocol (epoch 198 → 199)

• storage.usage.query
  • A query that requires Ready and takes {}.
  • It returns { measuredAt, totals, reclaimableBytes, worktreeCount }, with totals of kind database, artifacts, context_offload and memory.
  • Every figure is a file stat, a PRAGMA, or a small metadata sum. Nothing scans the large per-session tables.
  • Concurrent requests share one measurement. There is no TTL, so Refresh always re-measures.
• storage.usage.sessions.query
  • Takes 1–25 unique session IDs and returns transcript, runtime, artifact and (logical) context bytes, plus a worktree count, for each session the Host knows.
  • Unknown IDs are omitted rather than reported as 0.
  • Each source is summed with a range scan on its existing session_id-leading index, and the query yields to the event loop after every statement.
• Access. Both queries are granted to remote owners, like host.resources.query. They return numbers only, never paths.

Storage

• storage-footprint.ts is a read-only reader on the shared runtime.sqlite lease. It adds no triggers or write-path counters (tracking(perf): bound storage and protocol read/write amplification #5038).
• Shared SQLite file set. The SQLite file set (-wal/-shm/-journal) is now a shared helper, and the long-term memory store reuses it. The Subagent worktree directory name is exported from its owner rather than copied.

Desktop

• Data settings gets a Storage group with the breakdown, the reclaimable space and Refresh.
• Tasks settings shows a size on each archived task. Rows are measured in sequential batches of 25, with successes cached and failures cooled down.
• Per-Host failures stay local. One unavailable Host leaves only its own rows without a size.
• formatBytes in @maka/ui gains GB/TB and an optional locale. Output below 1 GB is unchanged.
• Copy notes the limits. Context sizes are logical, since shared content is deduplicated. Usage history is kept after tasks are deleted.

Why not a full per-kind breakdown of runtime.sqlite

The first version summed transcript, runtime and usage bytes across whole tables. Benchmarks showed this is O(table): inline rows put every leaf page on the read path, so a ~375 MB table took about 1 s in a single synchronous call on the Host event loop. The breakdown was dropped rather than moved to counters (write amplification, #5038) or to a worker thread (a new pattern here). Per-task sizes stay, bounded by the 25-ID cap.

Verification

• Builds: storage, runtime-host, ui and desktop build and typecheck.
• Tests:
  • storage footprint, writer composition, git worktree, long-term memory and public entrypoints: 77 pass;
  • runtime-host storage-usage protocol and coordinator, protocol (88), operation dispatcher, host composition and session collaboration authority;
  • ui: 693 pass;
  • desktop loader, preload, Storage group and tasks page: 39 pass.
• Checks: Biome, ASF headers, protocol-epoch-check (198 → 199), check-renderer-architecture --strict-base, locale hygiene and the Astryx surface inventory.
• Real data: against a copy of a real 60 MB runtime.sqlite, totals take 13 ms and a 25-ID session query takes 90 ms, with the unknown ID omitted.

Running app: I checked the running dev app (dev:worktree, zh-CN) against a disposable copy of a real workspace. The Data page shows the breakdown, and archived tasks show their sizes. The dev log had no errors.

Data → Storage	Archived tasks

Not verified: per-session timing on a multi-GB database.

Open questions (also on #5825)

1. Remote owners. Should both queries be available to them, as they are now?
2. Worktree count depth. A task's worktree count covers the task and its direct Subagent children only, not deeper.

AI use

Implemented and reviewed with Claude Code. Commits carry a Co-Authored-By trailer.

🤖 Generated with Claude Code

[hqhq1025]

Reviewed the storage-footprint reader, Host protocol/coordinator, Desktop routing and presentation at bc635bc. The read-only split and bounded per-session requests are useful, but I found one issue in the total footprint. This is not an approval to merge.

The focused storage/Host/Desktop tests pass (12/12), as do build:test, ASF headers, app-shell hooks, focused Biome, diff-check, and a merge-tree against current main. The current-head hosted test check is red in an ACP Plan paging test outside the changed files; its cause is not established. I did not exercise a multi-GB workspace or packaged Desktop.

Automated review notice: This comment was posted by an automated review agent operated by hqhq1025. It is not an independent human review and does not replace one.

[hqhq1025]

[P2] Do not add inline context bytes to the SQLite file size. Normal tool_result_archive blobs use inline storage (sqlite-context-offload-store.ts:1389-1390), and their bytes are already in context-offload.sqlite/WAL while also counted in context_store_usage.physical_bytes (:625-638). Adding that counter to the SQLite file set double-counts every inline blob, so the Storage page's "Total" overstates disk usage, not merely by rounding. A real 4,000,000-byte inline blob produced physicalBytes=4,000,000 and context_offload.bytes=12,265,376; the measured SQLite file set was 8,265,376 bytes. Please count only managed external file bytes in addition to the SQLite file set (or otherwise remove the overlap), and cover an inline blob in the footprint test.

[Astro-Han]

Reviewed exact head bc635bc492d9dc471f33bff9607324e4c1951661. This adds to the existing inline-BLOB double-count P2.

Trust boundary: looks sound. No client input becomes a filesystem path; the Host measures only fixed names inside its own State Root. Session ids are validated, capped at 25 per request and bound as SQL parameters. Remote owners can call the new operations and collaboration guests cannot. Worktrees are counted, never walked. There is no migration, and the per-task queries use session_id-leading indexes.

P2 (performance, inline): each per-task query sums up to 25 Sessions in one synchronous SQLite statement. The archived-tasks page measures every archived task, so it can block the Host event loop for about 1.3 s per statement on large tasks (reproduced locally with synthetic data). I haven't checked real-world task sizes.

P3:

• Epoch 198 → 199 collides with other open PRs.
• The renderer's size cache never evicts.
• The tests insert no runtime_partial_snapshots/core_agent_run_events rows, so a wrong table or column name there would still pass.

Checks run:

• Pass: the ASF header check, git diff --check, the app-shell hooks and renderer-architecture checks, locale hygiene, the desktop typecheck, and the protocol epoch check.
• Up to date: the Windows test inventory.
• Tests: new 13/13, related runtime-host/desktop 139/139, storage 71/71.
• Not run: the full npm test.

Automated review notice: This comment was posted by an automated review agent (Claude) operating on behalf of @Astro-Han. It is not an independent human review and does not replace one.

[Astro-Han]

P2: each statement sums payload bytes for up to 25 Sessions in one synchronous .all(). octet_length only skips overflow pages, so the cost grows with the byte volume of those Sessions' rows. Locally, 25 Sessions × 20k runtime events × ~1.5 KB blocked the Host event loop for about 1.3 s in a single statement (~67 ms at 4k events). The archived-tasks settings page then requests every archived task in 25-id pages, which in effect streams whole tables through the Host. Could this measure one Session per statement with a yield in between, or use a smaller bounded chunk? The UI could also measure only visible rows, or wait for a user action. A test asserting that the Host still yields during a multi-Session measurement would lock this in.

[Astro-Han]

P3 (coordination): several other open PRs also claim epoch 199/200, so whichever merges second has to renumber. Both new operations are additive; protocol-compatible-changes/recall-query-operation.json shows the exemption route for that case. Keeping the bump is fine, but it will need re-pinning if another PR lands first.

[Astro-Han]

P3 (nit): nothing ever deletes from entries, so it grows with every task shown for the lifetime of the window. Consider dropping expired entries.

[liugddx]

Thanks @hqhq1025 and @Astro-Han, both P2s were real. Everything below is addressed in 5af33e5, and the branch still merges cleanly with current main.

P2: inline context blobs counted twice (@hqhq1025). context_offload is now the context-offload.sqlite file set plus the managed value files under context-offload-values/; the Store's physical_bytes is no longer added. The Store keeps no running total of external bytes, so the value files are sized by a bounded async stat of that directory. The figure is now exact. storage-footprint.test.ts writes a real 1,000,000-byte inline blob and a 300,000-byte managed blob through SqliteContextOffloadStore and asserts total = SQLite file set + 300,000.

P2: per-task measurement blocking the Host (@Astro-Han).

• One Session per statement. Each source runs per Session, with a yield to the event loop after every statement, so one blocking step is at most a single Session's index range scan. With your shape (25 Sessions × 20k events × ~1.5 KB), the longest single step dropped from ~1.5 s to ~70 ms locally. A single very large Session can still take a noticeable step; that is the remaining bound.
• Only visible rows are measured. The archived-tasks page measures a row only once it scrolls into view (IntersectionObserver), so rows that are never seen are never measured. The 25-id cap per request is unchanged.
• Tests: a footprint test counts setImmediate turns during a multi-Session measurement, and a section test checks that nothing is requested before a row intersects.

P3s

• Epoch. The bump is gone and the epoch stays at 198. The two new operation keys are declared in protocol-compatible-changes/storage-usage-operations.json, following recall-query-operation.json, and protocol-epoch-check accepts it.
  • An older same-epoch Host that doesn't know these operations behaves as it does for recall.query: Storage shows its "could not measure" state and rows show no size.
  • Desktop skips that Host for 60 s after a failed query, so scrolling doesn't retry on every row.
• Renderer cache. Expired entries are dropped on each flush, and the cache is capped at 1,000 entries (tested).
• Footprint tests. They now insert runtime_partial_snapshots and core_agent_run_events rows, so a wrong table or column name fails.

CI note: the earlier red test run was two unrelated failures:

• the project-management-service ordering race, fixed in test(desktop): make the exact-selection Project test deterministic #5835;
• the ACP Plan paging timeout, tracked in test: acp-goal-plan-child-process times out on main @ 0fd754083 (AskUserQuestion never resolves) #5777.

The rerun is green.

Checks on this head:

• Builds: storage and runtime-host build; desktop typecheck and build.
• Tests: footprint 4/4; runtime-host storage-usage, protocol (88), dispatcher and host composition; desktop 41.
• Checks: Biome, ASF headers, protocol-epoch-check, renderer architecture (strict base), locale hygiene and the Astryx inventory.
• Not done: the observer is covered by a fake-observer test only; I didn't do another manual scroll pass in the app.

[hqhq1025]

This revision fixes the duplicate counting of inline context blobs by adding only the SQLite file set and managed value files. It also splits per-session SQL aggregation into yielding statements, adds visibility-gated task measurement, and bounds the renderer cache. The focused storage/UI tests, local build, and current-head hosted test pass. I found one remaining compatibility issue below; this is not a merge approval. I did not measure a multi-GB session or exercise an older remote Host end to end.

Automated review notice: This comment was posted by an automated review agent operated by hqhq1025. It is not an independent human review and does not replace one.

[hqhq1025]

P2: The compatibility rationale covers an old client talking to a new Host, but not a new Desktop talking to an old Host at the same epoch 198. Current main advertises epoch 198 without either storage operation; opening Data Settings sends storage.usage.query unconditionally (storage-usage-section.tsx:55-71, preload.ts:3716). That Host rejects the unknown key in requireOperationKey, and connection-session.ts closes the live connection. Visible task rows also send storage.usage.sessions.query; their 60-second failure cooldown only limits repeat disconnects, while Settings has no such guard. Please negotiate support before sending these optional queries, or make the pair fail compatibility admission rather than dropping a live connection.

[Astro-Han]

Re-reviewed exact head 5af33e59d9004d1de9054fa4a197a1877e453a62 against my earlier review on bc635bc4.

Resolved

• Event-loop blocking (my earlier P2). Each Session now gets its own statements, with a yield between them. With my earlier synthetic load (25 Sessions × 20k events × ~1.5 KB), the longest single stall dropped from ~1.28 s to ~132 ms. What remains grows only with one Session's rows: ~245 ms for a 300 MB Session. The code documents that, and exact per-Session sums can't avoid it. Archived-task sizes are now measured only when rows scroll into view, and a test covers this.
• Inline-blob double count. The SQLite file set plus the context-offload-values/ files are now counted once each. The new real-store test checks this.
• Earlier P3s. The renderer cache now expires entries and is capped at 1,000. The tests now insert runtime_partial_snapshots and core_agent_run_events rows. The epoch stays at 198, with a compatible-change declaration.

Old-Host compatibility. This adds to the P2 already raised on this head. storage-usage-section.tsx:73 re-queries on every mount and every Retry with no backoff. So a new Desktop connected to an older Host at epoch 198 drops that Host's connection every time Data Settings is opened or retried. The per-task path at least waits 60 s after a failure. The reason in storage-usage-operations.json:7 covers only an older Client, not a new Client talking to an older Host. Either capability negotiation, or keeping the epoch bump, would close this.

Nit (inline): the 200 px prefetch rootMargin in task-storage-size.tsx:61 has no effect if the list scrolls inside a nested container, because the observer's root is the viewport.

Checks run

• check:asf-headers, git diff --check, the protocol epoch check, app-shell-hooks, renderer-architecture, locale hygiene and the Desktop typecheck all pass.
• The Windows test inventory is current.
• Targeted tests pass: storage 28/28, runtime-host 147/147, desktop 7/7.
• Not run: the full npm test, or anything against a real older Host.

Automated review notice: This comment was posted by an automated review agent (Claude) operating on behalf of @Astro-Han. It is not an independent human review and does not replace one.

[Astro-Han]

This issues storage.usage.query on every mount and every Retry, with no backoff or capability check. An older Host at epoch 198 doesn't know the operation, so it rejects it in requireOperationKey and closes the connection each time Data Settings is opened or retried. The per-task path at least has a 60 s failure cooldown. Could this be gated on the Host advertising the operation, or back off after the first rejection?

[Astro-Han]

This reason holds for an older Client, which never sends these keys. It doesn't cover a new Client talking to an older Host at epoch 198: that Host rejects the unknown key and drops the connection (see the comment on storage-usage-section.tsx:73).

[Astro-Han]

Nit: with the default viewport root, this 200 px rootMargin has no effect when the list scrolls inside a nested container. Rows are still measured when they become visible, just without the prefetch.

[liugddx]

Thanks @hqhq1025 and @Astro-Han, you're right: the compatible-extension declaration only covered an old Client talking to a new Host. The protocol has no way for a Host to advertise optional operations, and adding one would be new surface for a read-only feature. So 2264703 takes the other route you offered: the pair fails admission.

• Epoch. The epoch goes back to 198 → 199, with a note that an epoch-198 Host would drop the connection on the unknown key, and storage-usage-operations.json is removed. A new Desktop and an older Host are now refused at the handshake, so Data Settings can't close a live connection. protocol-epoch-check reports the move, and the branch still merges cleanly with main. If another PR takes 199 first, I'll re-pin.
• Cooldown. The per-Host cooldown in the session reader stays as general failure backoff for draining or failing Hosts. Its comment no longer claims to handle unknown operations.
• rootMargin nit. Rows are now observed against their nearest scrolling ancestor, so the 200 px prefetch applies inside the Settings scroll container.

Re-ran on this head:

• runtime-host build and desktop typecheck and build;
• desktop storage/loader/tasks tests: 41/41;
• runtime-host storage-usage protocol, coordinator and protocol: 94/94;
• renderer architecture (strict base) and Biome.

[hqhq1025]

Reviewed commit 95cc9be. This merge brings current main into the storage-usage branch and resolves the Astryx inventory conflict. The prior compatibility finding is addressed: the new storage operations use compatibility epoch 199, while epoch-198 Hosts are rejected during handshake before domain commands are admitted (protocol/index.ts:107-110, client/connection.ts:1372-1395, server/host-kernel.ts:553-565). The lazy task-size observer also now roots its prefetch margin at the nearest scrolling ancestor. I found no new P0–P3 issue on this head.

Clean Node 24 installation and build:test, 18 focused storage/handshake/UI tests, Astryx inventory coverage, Biome, diff-check, and a merge-tree against current main pass locally. Hosted Windows recovery passes; hosted test was still running at review time, so merge readiness remains contingent on that result. I did not run a real epoch-198 remote Host, a packaged Desktop, or a multi-GB storage workload.

Automated review notice: This comment was posted by an automated review agent operated by hqhq1025. It is not an independent human review and does not replace one.

[Astro-Han]

Re-reviewed exact head 95cc9be827d3edc3e8b316d742aa9fa2266ac488 (incremental from 5af33e59). No P0–P3 findings.

• Old-Host disconnect resolved. The unknown-operation rejection only happens after admission, and admission now requires equal epochs on both sides (Host refuses a mismatched hello at host-kernel.ts:553-577; Client throws on a mismatched accepted handshake at connection.ts:1580). New Desktop ↔ epoch-198 remote Host gets the terminal RUNTIME_HOST_REMOTE_INCOMPATIBLE error, which reconnect treats as final (reconnect-lifecycle.ts:200,375), so no storage query is sent and there is no retry loop; old Desktop ↔ new Host is symmetric. This is the usual cost of an epoch bump (the whole remote Host is unavailable from a mismatched Desktop), and it makes the earlier no-backoff re-query concern moot.
• Epoch moves 198→199 with the compatible-change declaration removed; the epoch check passes. FYI only: several other open PRs also claim 199, so whichever merges later will need to bump again.
• Since 5af33e59: one commit plus a main merge. The Astryx inventory totals conflict was resolved correctly; task-storage-size.tsx now observes rows against their nearest scroll container (fixes last round's prefetch nit); storage-footprint.ts is unchanged, so the event-loop blocking fix still holds.

Checks run locally: git diff --check, ASF headers, renderer-architecture, Windows and Astryx inventories (current), locale hygiene, app-shell hooks, merge-tree vs main, full build and Desktop typecheck; storage 28/28, runtime-host 288/288 plus handshake/compatibility 63/63, Desktop 7/7. Not run: a real epoch-198 Host binary, Electron UI, full npm test.

Automated review notice: This comment was posted by an automated review agent. It is not an independent human review and does not replace one.

[Astro-Han]

Approved at @Astro-Han's explicit request: two independent automated reviews of this head found no blocking issues, the earlier old-Host disconnect is resolved by the epoch 199 handshake gate, and CI is green.