Skip to content

perf(cli): carry lock hashes over instead of refetching the whole graph - #714

Merged
Kamirus merged 1 commit into
mainfrom
kamil-claude/v3-lock-hash-carryover
Aug 12, 2026
Merged

perf(cli): carry lock hashes over instead of refetching the whole graph#714
Kamirus merged 1 commit into
mainfrom
kamil-claude/v3-lock-hash-carryover

Conversation

@Kamirus

@Kamirus Kamirus commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Follow-up to #713, stacked on its branch.

Every lock regeneration asked the registry for the file hashes of all resolved packages (getFileHashesByPackageIds over the full id list). It is a single batched canister call, but its payload scales with the whole dependency graph — after #713 removed manifest re-downloads, this was the dominant per-add network cost, paid on every mops add, remove, update and sync.

The same immutability argument as #713 applies: a published version's file hashes can never change, so hashes of packages already present in the lock are carried over verbatim, and the registry is queried only for packages new to the lock. Two consequences worth naming:

  • mops add x queries hashes for x and its newly-locked transitives only — network is now proportional to the change, not the graph.
  • mops remove regenerates the lock with zero registry queries.

Recovery semantics are preserved deliberately

mops install --lock update (any explicit --lock update) bypasses the carry-over and refetches every hash, so it remains the one command that repairs a lock with corrupt hashes. This split also has a diagnostic consequence: after an implicit regeneration a hash mismatch can be either a locally edited .mops/ file or a corrupt carried-over hash, so the post-regeneration error no longer claims a local edit with certainty on that path — it offers both fixes:

Mismatched hash for core@1.0.0/src/Array.mo
.mops/core@1.0.0/src/Array.mo does not match the lock.
If you have not modified files under .mops/, a hash carried over from the previous lock may be corrupt.
Run `mops install --lock update` to refresh hashes from the registry, or delete the `.mops/core@1.0.0` directory and run `mops install` to restore the package.

The confident "your local copy has been modified" message still appears when the lock was fully refetched (--lock update), where it is provably true.

One behavior change: a lock whose hashes were corrupted no longer gets silently repaired as a side effect of the next mops add — the corruption is carried, detected, and reported with the fix (asserted by the new test). Silent rewriting erased the evidence; pnpm errors on integrity mismatch the same way.

Also folds away a redundant resolve: updateLockFile derived package ids via a second resolvePackages() walk; they now come from the resolution it already ran.

What is unchanged

--lock check, lock format, and hash verification (every .mops/ file is still hashed and compared on mutating commands) are untouched — this PR changes where the lock's expected hashes come from, not what gets checked. Locks from older CLIs and v1/v2 locks get no carry-over (full refetch), same as today.

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Cursor AI review

👍 APPROVE — looks safe to merge

Category Assessment Details
Summary Lock regeneration carries immutable per-package file hashes from the prior v3 lock and queries the registry only for newly locked packages; explicit --lock update still full-refetches. Docs, changelog, and a regression test cover the intentional no-silent-heal behavior.
Code Quality Reuses resolvedDeps from the existing resolveDepsAndGraph walk instead of a second resolvePackages(); shares tolerant lock parsing via readLockFileTolerant for graph + hash carry-over.
Consistency Matches existing force = !!lock / defaultLock: "update" patterns in add/remove/update/sync; changelog under ## Next and docs/docs/10-mops.lock.md updated; skill’s --lock update recovery guidance remains accurate.
Security Traced checkIntegrityupdateLockFile (force gates carry-over) → checkLockFile (still hashes every locked .mops/ file). Fail-closed on corrupt carried hashes; --lock update recovery preserved; v1/v2/missing locks still full-refetch via readLockFileTolerant.
Tests New case in cache-resilience.test.ts asserts carry-over of a tampered hash on add, loud failure, and recovery via install --lock update. Existing cli.test.ts covers --lock update corrupt-hash rewrite and local-edit messaging.
Maintainability Small, localized change with clear invariants in comments; docs explain recovery. Stale top-of-checkLockFile comment is leftover wording only, not a behavior defect.

Verdict

Decision: APPROVE
Risk: Low
Reason: Integrity-path optimization is sound under published-version immutability: verification still runs, corrupt carry-over fails loudly, and explicit --lock update remains the full refetch recovery path with test coverage.


Generated for commit c2b0860

@automation-sa-sre automation-sa-sre left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated approval: the AI review verdict for e88c7a6 is APPROVE. See the "Cursor AI review" comment for details.

@Kamirus
Kamirus force-pushed the kamil-claude/v3-lock-hash-carryover branch from e88c7a6 to 96aacdb Compare August 12, 2026 06:36
@Kamirus
Kamirus force-pushed the kamil-claude/v3-lock-graph branch from 1ef7f92 to ddc8247 Compare August 12, 2026 06:57
@Kamirus
Kamirus force-pushed the kamil-claude/v3-lock-hash-carryover branch from 96aacdb to c475c1e Compare August 12, 2026 06:57
Base automatically changed from kamil-claude/v3-lock-graph to main August 12, 2026 07:07
Every lock regeneration fetched file hashes for all resolved packages
from the registry, making the payload of a mops add/remove/update/sync
proportional to the whole graph. Published versions are immutable, so
hashes of packages already in the lock are now carried over and the
registry is queried only for packages new to the lock; mops remove
updates the lock with no registry queries at all.

Explicit `mops install --lock update` still refetches everything, so it
remains the recovery command for a lock with corrupt hashes, and the
post-regeneration mismatch diagnostic only claims a local edit with
certainty on that full-refetch path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Kamirus
Kamirus force-pushed the kamil-claude/v3-lock-hash-carryover branch from c475c1e to c2b0860 Compare August 12, 2026 07:07

@automation-sa-sre automation-sa-sre left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated approval: the AI review verdict for c2b0860 is APPROVE. See the "Cursor AI review" comment for details.

@Kamirus
Kamirus enabled auto-merge (squash) August 12, 2026 07:10
@Kamirus
Kamirus merged commit 000ec35 into main Aug 12, 2026
27 checks passed
@Kamirus
Kamirus deleted the kamil-claude/v3-lock-hash-carryover branch August 12, 2026 07:17
Kamirus added a commit that referenced this pull request Aug 12, 2026
#717)

Ports the main-line lock stack —
[#712](#712),
[#713](#713),
[#714](#714) — into v3's
restructured code. The invariants transfer; the code expressing them is
re-fitted to v3's lock policy model (`maintain`/`locked`/`skip`) rather
than merged textually, since v3 rewrote `integrity.ts` and
`resolve-packages.ts`.

The crash class this closes exists identically on v3:
`computeLockFile`'s own comment documents that a stale-lock re-walk
"would throw ENOENT" on conflict losers a lock-driven install never
downloads — and on v3 the exposure is wider than on 2.x, because the
`maintain` flow regenerates the lock on every stale state with no
`--lock ignore` opt-out.

What lands, adapted to v3:

- **Lock graph**: `mops.lock` gains the optional `graph` section
(declared deps of every registry version, losers included). The walk
reads dependency lists lock-graph → cache → registry, so regeneration is
a local computation. v3's `hasValidShape` ignores unknown fields, so
pre-graph locks and older CLIs interoperate; a malformed `graph` is
ignored (it is an optimization, never a gate) and `inspectLockFile`
deliberately does not validate it, so `--locked` keeps passing on
pre-graph locks.
- **Hash carry-over**: `computeLockFile` reuses hashes of already-locked
packages (published versions are immutable) and queries the registry
only for packages new to the lock; `mops remove` regenerates with zero
registry calls. There is no force flag to port: every self-heal case
(missing/unparseable/legacy lock) yields nothing to carry, so v3's
documented recovery — `RESTORE_HINT`'s "delete it and run `mops
install`" — remains a guaranteed full refetch structurally. A
tampered-but-parseable lock is carried, not silently repaired; `mops
verify` reports it and the test pins that flow.
- **Atomic lock write**: v3 had the same non-atomic `writeFileSync` that
made parallel `mops install` crash on a torn lock in main's CI.
- **Cache resilience**: `isDepCached` requires a complete entry (empty
interrupted-run leftovers are deleted and re-downloaded — this also
fixes the unguarded manifest read after a cache hit in
`install-mops-dep.ts`); `syncLocalCache` restores packages missing from
the global cache before copying (and no longer throws `undefined` when
the failure isn't ncp's array-of-errors shape); the advisory
requirements check falls back to the global cache instead of crashing.
- **Comment corrections**: `computeLockFile` is no longer
only-safe-when-stale, and `checkLockConsistency`'s walk-free rationale
is now "offline and cheap", not "the walk would crash".

Not ported, deliberately: the github fetch-on-miss from #712 (v3 dropped
nested-config reads for github deps entirely, so there is nothing to
read), the `--lock update` force plumbing from #714, and its hedged
mismatch diagnostic (v3's maintain flow has no post-regeneration local
verification to hedge; `--locked`'s existing hints are already correct).

Fetch-on-miss downloads on v3 additionally go through
`verifyDownloadedPackageFiles`, so repaired cache entries are
integrity-checked against the registry before they land — stronger than
the 2.x port.

Tests are the four main-line regression scenarios re-expressed in v3
semantics (no `--lock` flags; carry-over is observed via `mops add` +
`mops verify` + delete-lock recovery). All four fail on unfixed v3 code
by construction — the graph test with the documented ENOENT, the
carry-over test because v3 refetched all hashes. Suites:
cache-resilience 4/4, cli.test.ts + local-path-lock + requirements
36/36, typecheck/eslint/prettier clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Kamirus added a commit that referenced this pull request Aug 14, 2026
Readers of the 2.x docs at the docs.mops.one site root never saw the
features shipped in CLI 2.21.0–2.23.0: `grep -rn "check-deploy"
docs/versioned_docs/version-2.x/` had zero hits even though `mops build
--check-deploy` shipped in 2.21.0. The versioned tree was treated as a
frozen snapshot while every feature doc landed only in `docs/docs/` —
per the amended "Keep docs in sync" rule, features shipping in a 2.x
release belong in both trees.

This replays the `docs/docs/` diffs of each 2.21.0–2.23.0 feature into
`docs/versioned_docs/version-2.x/`, wording identical:

- [#696](#696) — `mops self
update` major-version confirmation and `--major`
- [#644](#644) — `mops build
--check-wasm` / `--check-deploy`, `[build].check-wasm` / `check-deploy`,
and `wasmMemoryLimit` in the config reference
- [#713](#713) /
[#714](#714) — `mops.lock`
`graph` section and hash carry-over
- [#624](#624) —
check/check-stable/migrate diagnostics on moc 1.12.0+
- [#754](#754) — `pocket-ic
15.0.0` recommendation (6 pages)

After the replay, `diff -r docs/docs docs/versioned_docs/version-2.x` is
empty, so the sync state is trivially checkable. `docusaurus build`
passes (links and anchors resolve in both trees).

## What is unchanged

- `MOPS_POCKET_IC_URL` on the environment-variables page — already
mirrored by [#761](#761).
- 2.21.0–2.23.0 changelog entries with no `docs/docs/` counterpart (the
unzipper security fix, atomic lock writes, the ENOENT cache-state fixes,
the executable-bit and moc-wrapper fixes) — nothing to port; if any
deserve docs, that's a change to make in `docs/docs/` first.

## Follow-up worth flagging

The docs site deploys from the `v3` branch only, so this has no effect
on the live site until it is synced there. `v3`'s
`docs/versioned_docs/version-2.x/` is currently identical to
pre-backfill `main`, so the sync is exactly this patch — follow-up PR to
`v3` incoming.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants