Skip to content

perf(cli): resolve the dependency graph once per command - #743

Merged
Kamirus merged 1 commit into
v3from
kamil-claude/v3-resolve-memo
Aug 13, 2026
Merged

perf(cli): resolve the dependency graph once per command#743
Kamirus merged 1 commit into
v3from
kamil-claude/v3-resolve-memo

Conversation

@Kamirus

@Kamirus Kamirus commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

A single command resolves the dependency graph at several points — local cache sync, lockfile write, requirements check, mops sources — and each one re-walked the whole graph from scratch, re-reading mops.toml and mops.lock every time. On a cold or stale-lock walk that also means re-downloading packages purely to read their manifests.

The walk is now memoized in-process.

Why the key is content, not paths

Commands rewrite their own inputs mid-run: checkIntegrity writes mops.lock, and add/remove/update write mops.toml. A path-keyed memo would happily serve the pre-write graph afterwards. So the key is the content of both files, plus the root dir, the conflict policy and MOPS_ENV.

Local path dependencies are the interesting case: they are live directories, and their manifests are in neither file. Every manifest the walk reads is therefore recorded and re-hashed on a hit — including ones that did not exist, recorded as "" before the existence check, so creating a manifest where there was none also invalidates.

Registry package manifests are deliberately outside the key. Published versions are immutable, so their content cannot change under us — the same assumption the lockfile's graph carry-over already depends on.

How the invalidation was actually verified

A stale memo hit means a silently wrong dependency graph, which would be far worse than the redundant work it removes. So the tests don't assert on timing or call counts — they use a canary: a cached registry package's manifest is outside the key by design, so arming it to declare an extra dependency is invisible to a memo hit and shows up in deps only if a walk genuinely reran. "canary" in deps is a direct hit/miss probe.

Every case below was then mutation-tested — the relevant line was deliberately broken to confirm the test fails, so nothing passes vacuously:

invalidates on proof
root mops.toml edited canary appears after appending a dep
mops.lock written mid-command canary appears after writing even an unusable {} lock
local dep manifest edited, at depth edited a manifest two levels down, reached only through another local dep
manifest created where none existed canary appears; moving the record inside the existence check fails this test only
MOPS_ENV changed ./local-dep./staging-dep
conflict policy changed canary appears after setConflictPolicy("ignore")
the walk threw corrupt a nested manifest, then restore it so the successful key matches exactly — a retained rejection would be handed straight back

Two properties are structural rather than tested: when the lock short-circuits, the result is a pure function of mops.lock, which the key hashes, so an empty localInputs is correct rather than a gap. And checkLockFileLight() — hence the key — already depends on local manifests through localDepsHash, giving local edits a second, independent invalidation path.

One bounded caveat: while a walk is in flight, a concurrent caller shares the promise without re-checking local manifests. Nothing in the CLI resolves concurrently today, so it is unreachable, but it is the one place an edit landing mid-walk would be missed. Worth knowing before anyone parallelises resolution.

Cleanup, since this file was open anyway

The ~85-line cross-major conflict report moved out of the walk into its own function, which leaves the walk doing one thing. It moved character-for-character — every message string, the dependent ordering, the de-duplication and the --conflicts error exit — and resolve.test.ts passes unchanged, including its committed stdout/stderr snapshot.

copyResult now deep-copies graph's inner edge maps. It was shallow, so a caller mutating graph[pkgId][dep] would have written into the memo. No current caller does; the copy is cheap and closes the class.

reportedConflicts survives with a corrected comment. It no longer guards against repeated identical resolves — the memo does that now — but against the narrower case where inputs legitimately change mid-command. Leaving its old comment, which described the 3–5 passes this PR eliminates, would have actively misled the next reader. Whether conflict-reporting semantics should change at all is item 3, deliberately untouched here.

Verification

resolve, cache-resilience, local-path-lock, local-dep-manifest-lock and the new resolve-memo suites: 36/36, snapshot matched. Broader sweep across locked, path-dep-cycle, requirements, legacy-lock-flag, sync-plan, remove, remove-dry-run and outdated also passes.

github-dep-lock.test.ts fails locally with socket hang up fetching github.com/.../archive/<sha>.zip. That is the outage tracked as item 39 — github.com redirects to codeload.github.com, and today the redirect host is failing while codeload returns 200. This PR never touches GitHub archives.

Item 6 in #723.

🤖 Generated with Claude Code

A command resolves at several points — local cache sync, lockfile write,
requirements check, `mops sources` — and each re-walked the whole graph from
scratch, re-reading mops.toml and mops.lock every time. On a cold or stale-lock
walk that also re-downloads packages purely to read their manifests.

The walk is now memoized in-process, keyed on the *content* of mops.toml and
mops.lock plus the root dir, conflict policy and MOPS_ENV. Content rather than
path matters because commands rewrite their own inputs mid-run: checkIntegrity
writes mops.lock, add/remove/update write mops.toml. Local `path` dependency
manifests are live directories and are not covered by either file, so every one
the walk reads is recorded and re-hashed on a hit — including ones that did not
exist, so creating a manifest invalidates too.

Registry package manifests are deliberately outside the key: published versions
are immutable, which is the same assumption the lockfile's `graph` carry-over
already relies on.

Also extracts the ~85-line cross-major conflict report out of the walk, which
leaves resolveUncached doing one thing, and deep-copies the graph on the way out
so a caller mutating an inner edge map cannot write into the memo.

reportedConflicts survives, with a corrected comment. It no longer guards
against repeated identical resolves — the memo does that — but against the
narrower case where inputs legitimately change mid-command, producing a new key
and a second walk.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Kamirus Kamirus added the cli label Aug 12, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Cursor AI review

👍 APPROVE — looks safe to merge

Category Assessment Details
Summary In-process memoization of resolveDepsAndGraph keyed on mops.toml/mops.lock content, usesLock, conflict policy, MOPS_ENV, and re-checked local path manifests; conflict reporting extracted without semantic change.
Code Quality Reuses @noble/hashes like integrity.ts; extracts reportCrossMajorConflicts without new speculative API surface; copyResult deep-copies graph edges to protect the memo.
Consistency Matches existing module-level CLI state (conflictPolicy, reportedConflicts); changelog under ## Next / Performance; no command/flag/mops.toml schema change requiring docs or skills updates.
Security Traced install-allsyncLocalCachecheckIntegrity/computeLockFile({skipLock:true}) and checkLockFileLight/getLocalDepsHash dual invalidation; failed walks clear resolveCache; registry immutability assumption matches lock graph carry-over.
Tests resolve-memo.test.ts canary probes hit/miss for toml/lock/local/create/MOPS_ENV/policy/throw/copy isolation; no snapshot regenerations in this PR; conflict extract leaves prior resolve snapshot behavior intact.
Maintainability Cache key/invalidation rationale and concurrent in-flight caveat documented at the memo; conflict reporter comment updated for the narrower mid-command case.

Verdict

Decision: APPROVE
Risk: Low
Reason: High-risk path (cli/resolve-packages.ts) but invalidation is content-keyed with local-manifest rechecks, covered by canary tests, and install/lock call sites share walks only when usesLock semantics match. No correctness defects found versus Base.


Generated for commit 65b0439

@automation-sa-sre automation-sa-sre left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated approval: the AI review verdict for 65b0439 is APPROVE. See the "Cursor AI review" comment for details.

@Kamirus
Kamirus merged commit ca53390 into v3 Aug 13, 2026
24 of 34 checks passed
@Kamirus
Kamirus deleted the kamil-claude/v3-resolve-memo branch August 13, 2026 09:31
Kamirus added a commit that referenced this pull request Aug 13, 2026
…746)

[#743](#743) shipped the
resolve memo with one bounded caveat, stated in its PR body: a caller
arriving while a walk is in flight shares the pending promise without
re-checking local `path` dep manifests — the one input the content key
cannot see. That was unreachable while nothing resolved concurrently,
but [#745](#745) landed real
install concurrency in the same codebase, so the assumption is now one
refactor away from breaking silently. This closes it.

Every memo hit now takes the same path: settle the cached walk first,
verify the local manifests it read after. A mismatch — an edit that
landed after the walk read a manifest — falls through to a fresh walk
instead of serving the stale read. Sharing an in-flight walk is thereby
safe by construction, and the `localInputs: Map | null` in-flight
sentinel disappears: the map is complete exactly when the promise
settles, which is exactly when anyone reads it.

## How the test pins the interleaving

No timers, no tick counting. Two facts make it deterministic:

- the walk reads the **first** local dep manifest synchronously, before
its first suspension point, so an edit issued right after the un-awaited
call is always post-read;
- the second call is made in the same tick, so it always finds the walk
still in flight and shares its promise.

The first caller must resolve without the edit (`deps.c` undefined) and
the sharer must detect it on settle and re-walk (`deps.c` present).
Mutation-checked: against the pre-fix memo, exactly this test fails.

## What is unchanged

Rejection semantics: a failed walk still clears the slot and is never
served to a later caller, and a concurrent sharer still sees the
rejection. The pre-existing single-walk TOCTOU — an edit landing
mid-walk gives *that* walk a torn view — is inherent to any one-pass
resolve and out of scope; what this fixes is the memo *re-serving* such
a read to another caller. `resolve`, `resolve-memo`, `cache-resilience`,
`locked`, `local-path-lock`, `local-dep-manifest-lock`, `sync-plan` and
`requirements` suites: 83/83.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Kamirus added a commit that referenced this pull request Aug 13, 2026
Establishes the manual tracking convention for the install benchmark
from [#744](#744): the most
recent meaningful run lives in `perf/install-bench/RESULTS.md`, replaced
wholesale on each re-run, with a one-line history of prior headline
numbers. Deliberately not a CI gate — runs are only comparable within
one host and network, so a committed run is a point of reference.

## The run itself

v2.22.0 vs v3 at f8c619c — the first measurement with the whole
performance stack landed (#741 sorted lock, #742 update alignment, #743
resolve memo, #745 parallel installs, #746 in-flight re-check). Medians
of 3 against the live registry:

| scenario | v2 | v3 | v3 vs v2 |
| --- | --- | --- | --- |
| install-cold-nolock | 139.6s | 107.4s | 0.77x |
| install-cold-validlock | 106.6s | 61.1s | 0.57x |
| install-cold-stalelock | 128.0s | 95.7s | 0.75x |
| install-warm-nolock | 3.20s | 5.62s | 1.76x † |
| install-warm-validlock | 6.20s | 2.99s | 0.48x † |
| install-warm-stalelock | 3.31s | 2.45s | 0.74x |
| add-two | 4.00s | 3.84s | 0.96x |
| update-few | 8.02s | 5.42s | 0.68x |
| update-all | 14.9s | 2.87s | 0.19x |

† transient registry-latency window, not code: v2's own warm numbers
tripled against every prior run of the same released binary. A warm-only
re-check half an hour later returned to line — v3 warm-nolock 2.33s
(0.75x), warm-validlock 1.08s (0.42x) — and is recorded in RESULTS.md as
the representative warm numbers, with the main table kept unedited.

**v3 now beats released v2 in every scenario.** The cold no-lock install
— v3's one regression before #745 (1.23–1.25x in both pre-parallel
baselines) — is 0.77x, and CI-shaped installs (cold + committed lock)
are 43% faster.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants