Load a gitignored directory's contents when it is expanded - #2092
Conversation
`fs.listIgnored` runs `git ls-files --ignored --directory`, which collapses a wholly-ignored directory to a single trailing-slash entry so `node_modules/` costs one row instead of thousands. Pierre renders that entry as a directory row with a working chevron, but the collapse means no child path was ever sent — so expanding `blog/out/` opened onto nothing while `ls` showed six files in it. Keep the collapse and read the level on demand instead: - `fs.listDirectory` (padi surface 4.6, additive) reads ONE level of a directory, subdirectories carrying git's trailing slash so they arrive collapsed and expandable in their own turn. One level is what bounds the cost — expanding `node_modules/` is a readdir, not a 100k-path listing. No ignore filtering is needed or missing: git collapses a directory only when everything beneath it is ignored. - `<FileTree>` gains `lazyDirectories` + `onExpandLazyDirectory`. Pierre exposes no expansion callback and maps expand/collapse to no mutation event, so the transition is read off its store subscription, probing only the host's declared keys. Loading a level destroys and rebuilds the directory node, so the open ones are re-expanded — otherwise the folder snapped shut just as its contents arrived. - `mergeBrowseInventory` grows rule 4: a loaded directory yields to its children, which join the dimmed overlay; children are dropped unless their directory survived rules 1-2. Reopening a folder refetches it, which is the only honest refresh available: the working-tree watcher's ignore set is built from `listIgnored`, so an ignored path emits no pulse by construction.
…its slot Three defects the architecture-first-principles checks surfaced on #2092. C4/C6 (P4, illegal state) — a directory that reads back EMPTY took the "loaded" branch, so the collapsed key yielded to nothing: it left `paths`, Pierre had no child prefix left to infer the folder from, and the row the user had just clicked vanished. Worse than the bug it replaced — there was no longer a folder to click again. Reachable with no race at all: git lists a permanently-empty ignored directory in the overlay directly (verified against git). An empty read now keeps its own row. C2/C3/C6 (P3, one authority) — `loadLazyDirectory` captured `repoPath()` at call time but never re-checked it, so a read still in flight when the user switched repos resolved afterwards and wrote the PREVIOUS repo's listing into the new repo's map, under a repo-relative key that collides by construction (`out/`, `dist/`, `node_modules/`). That is precisely the collision the slotKey effect clears the map to prevent, which it cannot do for a request already in flight. Both callbacks now drop a response that outlived its slot — the toast too, since a failure belongs to the repo the user has already left. C6 (P3, merges must survive reorder) — a rapid expand → collapse → expand issues two reads for one directory, and promises resolve in completion order, not issue order, so a slow first response could overwrite a fresh second one. A per-directory generation admits only the newest. Also: record why Pierre's own `beginChildLoad`/`applyChildPatch` path is not used (public event names, private driver at 1.0.0-beta.6), delete a stale comment that contradicted the one below it, and soften browse.ts's "ignored by construction" to the snapshot claim it actually supports. Docs: drop the twice-drifted hardcoded contract version from the padi README in favour of the source of truth it already names, and refresh the show-ignored tip, which described a folder you could not yet open.
…own tag - hickey #1 / lowy #7 — `readdir({withFileTypes:true})` has lstat semantics, so under pnpm most of `node_modules` rendered as clickable FILE leaves and a click answered EISDIR; stat the symlink entries so the row's shape matches what a click reads, and a broken link stays a leaf. - lowy #5 — a `readdir` ENOENT was filed as `GIT_FAILED`, a tag meaning "a git subprocess failed" for a call that spawns none, and the documented typed `NOT_FOUND` survived only by regex on a twice-re-wrapped errno string. Add a `FILE_GONE` member, return it from `listDirectory` and `readFile`, map it in `unwrapGit`, and delete the now-redundant `servePadi` wrapper. - hickey #12 — cross-reference `isDirectoryPath` as the consumer of the folder-key format `listDirectory` mints. Raised by the lowy ∥ hickey lens review. Not pushed or merged.
- hickey #2 — rule 2's `seen` membership test now covers every entry reaching `overlay`, not just the top-level filter: a duplicated path makes `pathDiffOperations` emit two adds for one row, Pierre throws, and the recovery discards every hand-expanded folder. - hickey #6 — the rule-4 queue walk becomes one self-contained recursive `emit`. Fixes `lazyDirs.push` sitting above the visited check (a directory reached twice was listed twice) and `queue.unshift(...children)`, a `RangeError` on the six-figure flat level this feature targets. The loaded-but-empty directory keeps its own row exactly as before. - hickey #5 — `ignored` now names a loaded directory's own key as well. Pierre still paints that row from its children's prefixes, so dropping it un-dimmed the folder at the moment the user opened it, children dimmed below. - hickey #10 — `diffInventory` is the one constructor for the no-overlay case, so a new `BrowseInventory` field can't be forgotten at CodeTab's literal. Raised by the lowy ∥ hickey lens review. Not pushed or merged.
- hickey #3 — `openLazyDirs` carried three facts and two broke. A row that has no node right now (a search projection hid it) no longer retires the record, so a filter keystroke stops erasing the user's expansion; and a key the host no longer declares lazy is pruned, so an eye-toggle round trip reports afresh instead of showing an arbitrarily old cached level with no refetch path. - lowy #2 — a `lazyEpoch` prop clears the record when the host's loaded levels stop describing this tree (a repo / host switch), the wrapper's half of the invalidation the host already performs on its children cache. - lowy #6 — `onExpandLazyDirectory` may return a promise; on rejection the wrapper forgets the expansion, so a transient read failure no longer wedges the folder open-and-empty for the mount. - hickey #7 — the `tree.subscribe` callback runs under `safeApply`, so a throw can't escape into Pierre's emit loop and take other subscribers with it. - hickey #9 — "which directories should be open" is spelled once (`desiredExpandedPaths`) and used at both the constructor and `toOpen`. - hickey #8 — `expandPaths`' JSDoc no longer asserts an invariant `openLazyDirs` has made false. Raised by the lowy ∥ hickey lens review. Not pushed or merged.
- hickey #4 — one `AbortController` per directory replaces the hand-rolled `loadGeneration` + captured-`issuedSlot` pair, so "is this response still wanted" is ONE fact both callbacks read rather than two conditions each has to repeat. Fixes the real asymmetry where `.catch` checked only the slot and could toast a failure over a folder showing correct contents, and stops the superseded readdir server-side instead of racing it. - lowy #2 — pass `lazyEpoch={slotKey()}` beside the existing clear, so the tree's record of open lazy directories is invalidated by the same signal. - lowy #6 — the handler returns its promise and re-throws after toasting, so the tree drops its record and a retry costs one re-expand. Raised by the lowy ∥ hickey lens review. Not pushed or merged.
Codex review round 1 — five findings, all fixed, none disputed. - F1 (major) `browse.ts` — the symlink `stat` swallowed EVERY failure and returned the entry as a slash-free leaf with the listing reported as a SUCCESS. EACCES/EIO/ELOOP are not evidence that a target is a file: it hid the fault AND put back the wrong-row/EISDIR behaviour the follow-the-link branch exists to remove. Only a gone target is absorbed now; everything else fails the listing loudly. Pinned by a symlink-cycle test (ELOOP being the one such failure reachable without root), verified red first. - F2 (major) `errors.ts` — `isFileGoneError` read code and message as ALTERNATIVES, so an EACCES on `/repo/ENOENT-artifacts/out` answered true. Since this PR made that predicate mint a typed FILE_GONE -> NOT_FOUND, and the Code tab deliberately swallows NOT_FOUND as an expected deletion, a real permission fault would have vanished silently. A native errno is now authoritative and consulted alone; the message is a fallback only when the code was stripped crossing a boundary, matched on the errno shape rather than a bare substring. New `errors.test.ts`, two cases verified red first. - F3 (minor) `CodeTab.tsx` + `browse.ts` — the comment claimed the abort "stops the superseded readdir server-side". It does not: no signal is threaded through servePadi -> TerminalEndpointFs -> listDirectory. Corrected to state the abort is client-side only. The `Promise.all` over a level is now a sequential loop, so a six-figure flat cache directory no longer schedules one stat per symlink at once. - F4 (minor) `surface.test.ts` — the case was named "is version 4.5" while asserting 4.6, and still pinned the 4.4<->4.5 edge, so this PR's own skew claim went unexercised. Now asserts 4.5 vs 4.6 with the listDirectory reason, keeping the older edge alongside it. - F5 (nit) `kolu-git/README.md` — FILE_GONE added to the documented GitError union, with its NOT_FOUND mapping stated. Raised by the Codex ⇄ Claude agent debate. Not pushed or merged.
Codex review round 2 — F1-F5 verified resolved; one regression found and fixed.
F6 (major) — a regression MY round-1 F2 fix introduced. Narrowing
`isFileGoneError` to treat a present native `code` as authoritative was right,
but it broke the binary-preview leg of delete-while-viewing, which had been
relying on the errno text surviving onto the wire:
filePreviewTag returns GIT_FAILED for a gone file
-> unwrapGit throws ORPCError(code = "INTERNAL_SERVER_ERROR",
message = "Failed to hash file: ENOENT: no such file or directory, ...")
-> servePadi's fileGoneAsNotFound asks isFileGoneError
-> a code IS present and it is not ENOENT, so the preserved message is
never read -> false
The client swallows only NOT_FOUND, so deleting an open image/PDF/video
surfaced a visible "File content stream" error and dropped the last preview.
Confirmed empirically that ORPCError.code is the string
"INTERNAL_SERVER_ERROR", which is what short-circuits the predicate.
Fix: filePreviewTag now returns the structural FILE_GONE member when the error
is a gone file — the same thing readFile and listDirectory already do — so
unwrapGit emits NOT_FOUND before any wire wrapper can obscure the errno. The
narrowed predicate is untouched; the classification just happens at the layer
that still holds the native error.
Tests: a padi-level chain test driving the REAL endpoint against a REAL temp
repo asserts the ORPCError code for all three reads sharing this race
(readFile, filePreviewTag, listDirectory). Verified red first, failing with
exactly `expected 'INTERNAL_SERVER_ERROR' to be 'NOT_FOUND'` on the
filePreviewTag case while its two siblings passed — which also confirms the
defect was isolated to that read. A predicate unit test cannot see this; only
the assembled chain can.
Raised by the Codex ⇄ Claude agent debate. Not pushed or merged.
The /simplify pass — four cleanup agents (reuse, simplification, efficiency, altitude). Three independently reached the dead-code finding below, as did the Codex review. - servePadi's `fileGoneAsNotFound` and its two try/catch wrappers are DEAD. All three kolu-git reads that share the delete-while-viewing race now return the structural FILE_GONE member, which `unwrapGit` maps to a typed NOT_FOUND before servePadi ever sees it — so the helper only ever re-returned an already-correct error unchanged. Worse, it sat directly beneath the new `listDirectory` handler whose comment explains why no wrapper is needed, giving a reader no way to tell the sibling was vestigial rather than load-bearing. Deleted, along with the orphaned import, and `isFileGoneError`'s docstring no longer claims a sharing relationship that ended with it. - `listDirectory` now resolves symlinks with BOUNDED concurrency instead of a fully sequential loop. Both unbounded shapes are wrong on the input this feature exists for: `Promise.all` over a six-figure level is a promise/libuv spike (which is why it became sequential), but sequential pays a round trip per link — tens of seconds of dead UI on one click. A small pool overlaps the I/O with a fixed ceiling. Rows are filled by index so the emitted order still matches `readdir` regardless of settle order. - `inFlight` entries are now retired when their own read settles, guarded on controller identity so a newer read for the same directory keeps its controller. The map previously only shrank on a repo switch, so browsing many ignored folders in one repo accumulated a settled AbortController per directory for the session. - The two FileTree suites that drive the REAL Pierre under happy-dom shared ~60 lines of copy-pasted DOM scaffolding (shadow-root lookup, repaint flush, painted-row read, disposer bookkeeping). Extracted to `FileTree.testlib.ts` per the `*.testlib` convention. `mountTree` stays per-suite: each drives different props, and that is what each suite tests. Reported clean and left alone: the rule-4 walk's accumulators, FileTree's two invalidation effects (they watch genuinely different domains), and `reportLazyExpansions`' per-tick cost (bounded to the declared lazy set, not the visible rows).
… log its expected FILE_GONE race at debug Two findings from the code-police rule pass on PR #2092 (fs.listDirectory, the lazy-load-one-level-of-a-gitignored-directory feature): - prefer-focused-library: `listDirectory`'s bounded-concurrency symlink `stat` fan-out hand-rolled a worker-pool loop (shared mutable counter + `while` + `Promise.all`) instead of reaching for a focused concurrency-limiter. Swapped for `p-limit`, added as a new dependency to kolu-git (and allow-listed as a stable leaf in padi's daemon-closure guard, since it carries no daemon wire/behaviour). - no-untyped-escape-hatches: the hand-rolled loop cast away the `| undefined` `noUncheckedIndexedAccess` puts on `links[next++]` and `entries[i]` with `as` casts. The `p-limit`-based rewrite only needs one index read (`entries[i]`), narrowed with a genuine `if (!entry) return` guard instead of a cast. - errors-must-log-at-error: `listDirectory`'s catch logged UNCONDITIONALLY at `error`, including for the FILE_GONE case — the function's own headline scenario (a build directory cleaned between listing and click), which its own doc comment calls an expected race. Its sibling `filePreviewTag` in the same file already downgrades the identical case to `debug`; `listDirectory` now matches.
…ed tick Fact-check finding on PR #2092's `FileTree.tsx` (lazy-loaded gitignored directories, #2091): `reportLazyExpansions`'s rejection handler deleted the key from `openLazyDirs` bookkeeping but never collapsed the row itself. Pierre's own expansion state (`item.isExpanded()`) stayed true, so the very next store tick — caused by ANY unrelated interaction (clicking another file, an unconnected path mutation), not a deliberate collapse-and-reopen — read "expanded, not recorded" and mistook it for a fresh user expansion, re-firing the same failing `onExpandLazyDirectory` call forever. Fix: on rejection, also call the row's `collapse()` so its on-screen state agrees with the bookkeeping. The user sees the folder visibly close (real feedback that the expand failed) instead of a silent, ever-repeating retry storm, and only a deliberate re-open re-arms the probe. Added a regression test (`FileTree.lazyDir.test.tsx`) that mounts a real Pierre tree with a rejecting `onExpandLazyDirectory`, clicks an unrelated row, and asserts the callback was not called a second time. Verified red against the pre-fix code (delete-key-only, no collapse) before restoring the fix.
…nction is gone no-dead-code finding on PR #2092: this PR deletes servePadi.ts's `fileGoneAsNotFound` wire wrapper (replaced by kolu-git's structural `FILE_GONE` tag + `unwrapGit`'s mapping), but several present-tense comments elsewhere still asserted it as a live mechanism — a dangling reference that misleads a future grep for the name into believing the function still exists: - errors.test.ts's new suite doc-comment claimed "servePadi's `fileGoneAsNotFound` wire mapping" reads `isFileGoneError` — that mapping is now `unwrapGit`'s `.with({code: "FILE_GONE"}, ...)` arm. - browse.test.ts's new `filePreviewTag` FILE_GONE test explained the regression via `fileGoneAsNotFound` in present tense; reworded to mark it clearly historical (the wrapper it describes no longer exists). - fsGitDeps.ts and servePadi.ts (untouched by this PR, but now stale because of it) still named `fileGoneAsNotFound` as a live precedent/mapping. - servePadi.recycleKaval.test.ts's suite doc-comment cited it the same way. Reworded all five to describe the current structural FILE_GONE → NOT_FOUND mapping instead of a deleted function name. (browse.ts's own two references — one present-tense, one historical-narrative — were fixed alongside the concurrency/log-level changes in the prior commit, since they sit in the same hunks.)
# Conflicts: # packages/padi/src/daemonBoot/buildId.closure.test.ts
⚖️ Lowy ∥ Hickey lens reviewTwo structural lenses read this diff independently against Both lenses landed on the same major defect from opposite directions — a Status:
Findings
The four commits
Rejected, with the reasons
Recorded as a follow-up
Recorded as observationslowy #8 and hickey #13 are the same fact from two lenses: |
Codex ⇄ Claude debate✅ Consensus in 3 round(s) — base
Legend — findings Codex raised:
|
👮 Code-policePass 1 (rule checklist) + Pass 2 (fact-check), scoped to 4 real violations found, all fixed. Every finding was verified against the code rather than taken on faith.
The retry bug (4f1a5c6) is the substantive one — it was introduced by an earlier gauntlet fix in this same PR, and its regression test was verified red against the pre-fix code before the fix was restored ( Adding Verified green: |
CI e2e metrics —
|
| platform | host | workers | cores | cap | e2e duration |
|---|---|---|---|---|---|
x86_64-linux |
kolu-ci-3 | 8 | 32 | 8 | 181052 ms |
aarch64-darwin |
ci@petit | 4 | 10 | 6 | 244791 ms |
Two-platform run, settled green in 387855 ms with no reruns. Durations are the whole ci::e2e recipe wall from the durable ledger (.ci/92182b7/runs/1.json), not Cucumber's internal timer; the worker counts are the last e2e: workers= line each lane recorded.
EvidenceWhat the clip shows: in Code → All files with the eye toggle flipped on, a wholly-gitignored End state — two levels walked, ▶ HD (full 1280×720 frame, terminal + panel): https://juspay.github.io/video-evidence/evidence.html?repo=juspay/kolu&v=code-tab-expand-ignored-dir.mp4 How it was capturedRecorded off-machine on an ephemeral The fixture the scenario builds:
Scenario — Scenario: Expanding a gitignored directory loads its contents
...repo fixture via the terminal...
And I click the Code tab
And I click the Code tab mode "browse"
Then the file browser should show a file "README.md"
And the Code tab should not show a directory node "build"
When I click the Code tab show-gitignored toggle
Then the file browser should show a directory "build"
And the file browser should not show a file "build/bundle.js"
When I click the directory "build" in the file browser
Then the file browser should show a file "build/bundle.js"
And the file browser should show a directory "build/nested"
And the file browser should not show a file "build/nested/chunk.js"
When I click the directory "build/nested" in the file browser
Then the file browser should show a file "build/nested/chunk.js"
And there should be no page errorsThe two Every step is an existing one except Mobile / coarse-pointer: considered, no second capture needed. The two axes are orthogonal — viewport size ( |


The bug
In the Code tab's browse tree with "show gitignored files" on,
blog/outrenders as a folder with a chevron, expands — and shows nothing. The terminal right below it proves the directory isn't empty:ls blog/out/lists000.html 001.html 002.html hl.js index.html style.css.Nothing was lost in transit.
fs.listIgnoredrunsgit ls-files --others --ignored --exclude-standard --directory, and--directorycollapses a wholly-ignored directory to a single trailing-slash entry — that's what keepsnode_modules/at one row instead of thousands. Pierre builds its hierarchy from that one flat snapshot and has no lazy-children path, so the row gets a working chevron over children that were never sent, and no amount of clicking will produce them.The fix
Keep the collapse — it's load-bearing — and read the level on demand.
fs.listDirectory(padi surface 4.6, purely additive) reads ONE level of a directory, with subdirectories carrying git's trailing slash so they arrive collapsed and expandable in their own turn. One level is what bounds the cost: expandingnode_modules/becomes a single readdir handing back package names, where the recursive listinggit ls-fileswould have to produce is ~100k paths for that same click. No gitignore filtering is applied and none is missing — git collapses a directory only when everything beneath it is ignored, so inside a collapsed row a plain readdir is the truthful listing.browse.test.tschecks that against git itself rather than trusting the argument.<FileTree>gainslazyDirectories+onExpandLazyDirectory+lazyEpoch. Pierre exposes no expansion callback and maps expand/collapse to no mutation event (they'reFileTreeStoreIgnoredSemanticEvents, whichtoTreesMutationEventdrops), so the transition is read off its store subscription — probing only the host's declared keys, never the whole visible projection, since that listener also fires for selection and focus.One thing that surfaced while building it: loading a level replaces the collapsed key with its children, and Pierre's remove destroys the directory node carrying the expansion — so the folder snapped shut at the exact moment its contents arrived, reading as if the click had done nothing. Open lazy directories are now re-expanded alongside the path delta.
mergeBrowseInventorygrows rule 4 — a loaded directory yields to its children, which join the dimmed overlay, and children are dropped unless their directory survived rules 1–2 (so a.gitignoreedit can't leave a ghost subtree beside the tracked files that now own the prefix).What the review gauntlet changed
The first draft worked but was wrong in ways worth naming. Four reviewers ran over it — an architecture-first-principles pass, a lowy ∥ hickey lens review, a Codex debate (3 rounds to consensus), and a simplify + code-police pass. Between them:
readdir({withFileTypes:true})haslstatsemantics, and under pnpm most ofnode_modulesis a symlinked package directory — 20 of 30 entries inpackages/client/node_moduleshere. So this PR's own documented promise, "walk down intonode_modules/solid-js/a click at a time", was false in the repo it ships in, and clicking one gave an EISDIR toast. Now the link's target decides the row.[]is notundefined, so an empty read took the "loaded" branch and the collapsed key yielded to nothing — Pierre had no child prefix left to infer the folder from, and the row the user just clicked disappeared. Worse than the original bug: there was no folder left to click again. Reachable with no race, since git lists a permanently-empty ignored directory in the overlay directly.loadedChildrenis keyed by a repo-relative path, and those collide by construction (out/,dist/). The slot clear couldn't reach a request already in flight. Reads are now superseded by anAbortControllerand scoped to the slot that issued them.isFileGoneErrorto trust a native errno (it previously matchedENOENTanywhere in a message, so anEACCESon/repo/ENOENT-artifacts/answered true) broke the binary-preview path, which had been relying on that string surviving onto the wire.filePreviewTagnow returns the structuralFILE_GONEtag like its siblings, sounwrapGitmaps it toNOT_FOUNDbefore any wrapper can obscure the errno.RangeErroron a huge level; and the now-deadfileGoneAsNotFoundwrapper is gone.Staleness
Reopening a folder refetches it, and that's the only honest refresh available: the working-tree watcher's ignore set is built from
listIgnored, so an ignored path emits no pulse by construction. Watching them would mean watching exactly the churn (node_modules, build output) the ignore set exists to keep out. So an already-open folder can go stale until it's collapsed and reopened.Tests
Written first, and confirmed red for the right reasons before any implementation — including every fix above, each verified red against the pre-fix code:
FileTree.lazyDir.test.tsxdrives the real library rather than a mock, because the subject is precisely Pierre's own behaviour on a childless directory row — that it gets a chevron at all, and that expanding it ticks the store. Assertions are on the callback and on painted DOM rows, since "model right, screen empty" is the original defect.browseInventory.test.ts— the four merge rules as one table test.browse.test.ts/errors.test.ts— the one-level listing, the traversal guard, symlink resolution, and the gone-path classification.terminalWorkspace/endpoint.test.ts— the assembledkolu-git → unwrapGit → ORPCErrorchain, proving all three reads that share the delete-while-viewing race surface asNOT_FOUND.🤖 Generated with Claude Code