Scope: watch the workspace so externally-added files appear in the Files tree #2

Activity

1. Francois3d

   added

   enhancementNew feature or requestNew feature or request

   needs-triageMaintainer needs to evaluate this issueMaintainer needs to evaluate this issue

   on Aug 29, 2026
2. Francois3d

   marked this as blocked on Aug 29, 2026

   • Files tree does not update live when files are added or deleted #1
3. Francois3d commented on Aug 29, 2026

   

   Francois3d

   on Aug 29, 2026

   OwnerAuthor

   More actions

   This was generated by AI during triage.

   Triage findings: the workspace watcher already exists

   This ticket's framing assumed the gap needs "a real filesystem watcher on the workspace root, which is where the cost and the cross-platform risk live." Verification against the codebase suggests that watcher is already running, supplied by a dependency.

   The entry index is not hand-rolled. It is FileFinder from @ff-labs/fff-node (v0.9.4, Rust), which ships "a real-time file watcher" as a documented feature. The watcher is on by default — disabling it is the opt-in disableWatch option — and the finder is constructed without that option. So every indexed workspace root already has a recursive, gitignore-aware native watcher attached.

   The corroborating detail is what the existing refresh path actually does. WorkspaceEntries.refresh delegates to the search index's refresh, which calls FileFinder.scanFiles(). The library documents scanFiles() as "useful after major file system changes that the background watcher might have missed." In other words, the explicit refresh is a belt-and-braces rescan layered on top of a watcher that already exists.

   Effect on the original five questions

   • Q1 (recursive watching across platforms): largely moot. Recursion and cross-platform behaviour are the library's concern and already solved in native code. (Note also that the installed effect release does expose a recursive option on FileSystem.watch; the vendored .repos/effect-smol copy is stale on this point and shows a single-argument signature. Worth a vpr sync:repos regardless.)
   • Q2 (ignore rules): already handled. Gitignore exclusion is provided by FileFinder, which is why gitignored files are excluded from entry search with no ignore list anywhere in repo code. There is no list to design or place.
   • Q3 (refresh cost and coalescing): reframed. The worry was a full 25,000-entry rescan per filesystem event; the native watcher updates incrementally instead. Coalescing may still apply to the notification, not to the scan.
   • Q4 (watcher lifetime): already answered by construction. The watcher's lifetime is the index's lifetime, which follows the existing 15-minute idle TTL.
   • Q5 (remote environments): still open, and still a real judgment call.

   The actual gap

   FileFinder exposes no change callback. It offers a readiness flag, a scanning-in-progress check and an explicit rescan, but nothing to subscribe to. The index updates itself and never tells the server. So the real question is not "how do we build a watcher" but:

   How does the server learn that the index changed, given the library exposes no change event?

   Narrowed scope for this ticket

   Run this experiment first — it may close the ticket outright. In a real repo (populated node_modules, active .git; not an empty fixture), create a file externally with touch, then query projects.listEntries fresh, with no refresh call in between.

   • If the file appears: the server's data is already live, the remaining gap is purely client-side cache behaviour, and Files tree does not update live when files are added or deleted #1's push channel plus an invalidation trigger is the whole fix. Record that and close this ticket with a much smaller follow-up.
   • If it does not appear: the watcher is not feeding the query path as expected. Establish why, then answer the notification question below.

   Then answer, with evidence:

   1. Notification mechanism. Given no callback API, how does the server detect an index change to feed Files tree does not update live when files are added or deleted #1's signal? Compare at minimum: a cheap periodic poll of index state; requesting an event API upstream from ff-labs; a thin notification-only watcher that does no scanning of its own. State the cost of each on a busy repo.
   2. Churn. Confirm on a real repo that .git and node_modules activity does not produce continuous notification traffic. If it does, decide the debounce window and coalescing key — KeyedCoalescingWorker in @t3tools/shared already exists for this.
   3. Remote environments. Does this behave acceptably over SSH-managed and relay-connected environments, or does it need to be opt-out there? This one is unchanged by the findings above and remains a genuine judgment call — flag it for a maintainer rather than deciding it unilaterally.

   Deliverable: a comment recording the decisions, plus either a small implementation ticket or a note that #1 already covers it. Then close this one.

   Out of scope:

   • Building a filesystem watcher from scratch. Establish why the existing one is insufficient before proposing any new watching.
   • Duplicating the push channel from Files tree does not update live when files are added or deleted #1. Any notification found here feeds that channel.
   • Changing the index's entry cap, scan timeout, or idle TTL.
   • Auditing the finder's enableFsRootScanning / enableHomeDirScanning settings. Both are enabled while the library documents them as off-by-default for watcher-churn reasons; that is a real question but belongs in its own ticket, not here.

   Conventions: read docs/internals/code-conventions.md first, and the Effect service conventions review-agent prompt under .macroscope/check-run-agents/ before touching server services.
4. Francois3d

   added

   ready-for-agentFully specified, ready for an AFK agentFully specified, ready for an AFK agent

   and removed

   needs-triageMaintainer needs to evaluate this issueMaintainer needs to evaluate this issue

   on Aug 29, 2026
5. Francois3d commented on Aug 29, 2026

   

   Francois3d

   on Aug 29, 2026

   OwnerAuthor

   More actions

   Scoping outcome: the experiment closes most of this ticket

   Ran the experiment this ticket asked for first, against this repo itself — 17,701 indexed entries, populated node_modules, active .git. Not a fixture. Probe driving FileFinder directly with the exact options WorkspaceSearchIndex.createFinder uses for the paths variant.

   Result: externally-created files already reach the query path with no refresh call.

   [A] external CREATE visible without scanFiles: true  after ~103ms
   [B] external DELETE reflected without scanFiles: true after ~104ms
   

   So the server's data is live already. The library's built-in watcher handles it, exactly as the previous comment predicted. Nothing needs to be watched, and no ignore list needs designing — .git and node_modules are already invisible to the index:

   60 writes into node_modules/ and .git/ over 1.5s -> index changed on 0/30 samples
   git status (read-only git):                      -> no index change
   idle real repo, 20s:                             -> 0 index changes
   

   That answers Q2 (ignore rules) and Q3 (churn) outright, with evidence: there is no churn to coalesce. Q1 (recursive watching) and Q4 (watcher lifetime) stay moot — the watcher is the library's, and its lifetime is the index's, which already follows the 15-minute idle TTL.

   The remaining gap is exactly the one the previous comment identified: FileFinder exposes no change callback, so the server never learns the index moved and never publishes #1's signal.

   ---

   Q1 — Notification mechanism

   Measured the three candidates.

   Rejected: poll getScanProgress(). Looks like the obvious cheap signal and is not one. scannedFilesCount is frozen at the initial-scan value and never moves when the watcher updates the index:

   before burst of 200 external file creates: {"scannedFilesCount":16084,...}
   after  (all 200 visible via search):       {"scannedFilesCount":16084,...}
   

   Rejected: a second, notification-only filesystem watcher. It would reintroduce every problem this ticket was opened to solve — recursion on Linux, an ignore list, .git churn — to duplicate a watcher the library is already running successfully. Building a watcher to notify us about the watcher we already have is the wrong shape.

   Recommended: poll the index's own entry count. mixedSearch("", { pageSize: 1 }) returns totalMatched without materialising the page, and it tracks the watcher exactly:

   count before=17701  afterCreate=17702  afterDelete=17701   (tracks add/delete: true)
   
   cost, pageSize 1:      0.9, 0.8, 0.5, 0.5, 0.4, 0.5, 0.6 ms
   cost, pageSize 25002:  338, 350, 359 ms      <- the full list, for comparison
   

   ~0.5ms against a 17.7k-entry repo, ~700x cheaper than a full list. At a 1s interval that is 0.05% of one core per subscribed workspace, and it does no scanning — it reads state the watcher already maintains. For reference, the current scanFiles() + waitForIndexReady() refresh costs 219ms, so this is also far cheaper than the refresh #1 already triggers.

   No KeyedCoalescingWorker needed: the poll interval is the coalescing, and #1's PubSub is already keyed by normalised workspace root. Poll, compare to the last value, publish the existing coarse signal on change. Client re-reads through projects.listEntries as it does today.

   Honest gaps, and why they are acceptable:

   • A rename, or a balanced add+delete, leaves the count unchanged. The client's existing 30s staleness window still catches those, and in-app renames go through refresh → Files tree does not update live when files are added or deleted #1's signal. This is a narrowing of the gap, not a closing of it.
   • Content-only edits do not move the count. Correct — the tree does not care, and subscribeProjectFileChanges already covers open-file contents.

   Upstream: the clean fix is a change callback on FileFinder. Worth raising with ff-labs. The poll is the bridge, not the destination.

   Q2 — Churn

   Answered above: none. 0/30 samples on deliberate node_modules and .git write churn, 0 changes on a 20s idle real repo. Nothing to debounce beyond the poll interval itself.

   Q3 — Remote environments

   No opt-out needed, and I do not think this one needs a maintainer's judgment after all. The poll is server-side and never crosses the wire. What crosses is #1's coarse signal, and only on an actual change. Compared with today — every client refetching the full entry list every 30s — this strictly reduces relay and SSH traffic. The cost profile is identical local, relayed, or tunnelled.

   ---

   Two things this turned up that belong in their own tickets

   1. Watcher-readiness race (real bug). waitForIndexReady() returns while the watcher is still starting:

   waitForIndexReady returned at   52ms; isWatcherReady=false
   isWatcherReady became true at 1094ms   => ~1.0s blind window
   

   WorkspaceSearchIndex.make treats waitForIndexReady as fully ready and starts serving. Any external change in that ~1s window is missed permanently, until something forces a rescan. My first probe run hit this and produced a false negative before I noticed. The fix looks like also waiting on isWatcherReady in make.

   2. enableFsRootScanning / enableHomeDirScanning are both true in createFinder, while the library documents them as off by default because root "floods the watcher with churn-prone events" and makes the caller "responsible for the resulting fs-event volume". This ticket put that out of scope and I have left it there, but the library's own wording is stronger than the previous comment assumed.

   Deliverable

   Decisions recorded above. Follow-up implementation ticket filed. Closing this one.

   Model: Claude Opus 5, harness: Claude Code
6. Francois3d commented on Aug 29, 2026

   

   Francois3d

   on Aug 29, 2026

   OwnerAuthor

   More actions

   Scoped. Follow-ups filed:

   • Files tree does not update for changes made outside the app #4 — the implementation: publish Files tree does not update live when files are added or deleted #1's existing signal from a cheap entry-count poll. No new watcher.
   • Workspace index serves before its file watcher is ready, losing ~1s of external changes #5 — watcher-readiness race found while probing (~1s of external changes lost after index creation).
   • Workspace index opts into filesystem-root and home-dir scanning, which the library documents as off by default #6 — enableFsRootScanning / enableHomeDirScanning, which this ticket put out of scope and asked to file separately.

   No code changed for this ticket; its deliverable was the decision above.
7. Francois3d

   closed this as completedon Aug 29, 2026