Stop publishing npm-shrinkwrap.json: ignored by npm 12 (Node 27), and it has broken consumer npm ci twice

[Ethan-Arrowood]

Proposal

Stop publishing npm-shrinkwrap.json in the harper package, and move application reproducibility to locked artifacts (see Options). This is the decision #2172 asks for: its "drop the shrinkwrap" option.

Why

• npm 12 ignores it, and Node 27 bundles npm 12. deps: upgrade npm to 12.1.0 nodejs/node#66212 (npm 12.1.0) is merged as semver-major and marked dont-land on v22, v24 and v26. So from Node 27 on, the shrinkwrap does nothing for anyone on the default npm. feat!: drop npm-shrinkwrap.json support npm/cli#9262 removed support entirely, including from dependency tarballs, and it fails silently.
• It has broken consumers' npm ci twice on today's default npm (npm 11 on Node 24 and 26):
  • npm-shrinkwrap.json (5.1.18+) breaks consumers' npm ci on npm 11 — EBADPLATFORM @esbuild/aix-ppc64 #1780: dev entries in 5.1.18/5.1.19, EBADPLATFORM.
  • #1959's shrinkwrap prune breaks npm 11 consumers' npm ci (5.2.0+): alasql → react-native-fs left unresolved #2941: the fix(build): strip the react-native tree from the published shrinkwrap #1959 react-native prune, EUSAGE, in every 5.2.x and 5.3.0 release.
  • Both came from us editing a file whose only purpose is shaping consumer installs, with no consumer-side test.
• The pinning that matters no longer depends on it. Pin the load-bearing native and encoder dependencies #2179 pins the load-bearing native and encoder deps exactly in package.json (@harperfast/rocksdb-js 2.9.1, lmdb 3.5.6, msgpackr 2.0.6), and that works on every npm. Only transitive pinning is lost, and npm 12 consumers already live without it.
• It doesn't reduce install size in any lasting way. The react-native tree has to be removed at alasql (#1959's shrinkwrap prune breaks npm 11 consumers' npm ci (5.2.0+): alasql → react-native-fs left unresolved #2941 step 2, Make react-native-fs an optional peer dependency AlaSQL/alasql#2456), which works for every npm version. Removing the tree from the shrinkwrap only ever helped npm ≤ 11.
• It distorts consumers' lockfiles. npm 11 nests harper's shrinkwrapped tree (~500 packages) under node_modules/harper instead of deduplicating it, so a consumer's lockfile changes shape depending on which npm last touched it.

Scope

• Remove shrinkwrap generation from the build (build-tools/build.sh), plus prune-shrinkwrap-dev.mjs and prune-shrinkwrap-react-native.mjs.
• Turn check-shrinkwrap-pins.mjs into a check that the load-bearing deps are exact-pinned in package.json.
• Docker: fix(docker): install harper from the packed tarball's own shrinkwrap #2042 installs the image from the packed tarball's own shrinkwrap. That needs a replacement that keeps the image reproducible, for example building from the repo's package-lock.json with npm ci (see Docker image dependency tree is resolved fresh at build time, not from the published shrinkwrap #1960 and Docker image still ships the react-native-fs subtree unpinned (residual from #1960/#2042) #2043).
• Update DESIGN.md and dependencies.md, which describe the shrinkwrap as authoritative.

Why harper ships a shrinkwrap at all

harper is both a library and an application. For applications (CLIs, servers) that publish to the registry, the long-standing npm guidance has been to ship an npm-shrinkwrap.json. package-lock.json is never published, so a shrinkwrap is the only way for an installer to get the exact dependency tree the package was tested with. That is what #1622 turned on. The #1780 and #2941 breakages came from the library side of harper: installing it as a dependency and running npm ci.

What npm 12 does instead

npm 12 treats harper like any library. It reads the ranges in harper's package.json, resolves every transitive dependency fresh from the registry at install time, and hoists and dedupes the result into the consumer's tree. Specifically:

• Only exact-pinned direct deps stay fixed. That includes rocksdb-js, lmdb, msgpackr and argon2 (Pin the load-bearing native and encoder dependencies #2179). Everything below them floats within ranges.
• Consumers' other dependencies and overrides now shape harper's tree.
• Only the host platform's native prebuilds install, instead of all of them.
• The react-native tree returns via alasql (alasql's optional react-native-fs dep pulls ~140M of react-native/hermes/metro into installs and published Docker images #1937).
• Nothing warns that the shrinkwrap in the tarball was ignored.

How much reproducibility is lost depends on how harper is installed:

• Installed as a dependency: reproducible per consumer. Their own package-lock.json pins whatever was resolved on the first install, though that is not what harper tested.
• Installed as an application (npm i -g harper, npx harper, or an image built without a lockfile): not reproducible at all. Every install resolves fresh. This is exactly the case the shrinkwrap existed for.

So removing the shrinkwrap is only half a proposal. The other half is what replaces it for application installs.

npm's recommended replacement: bundleDependencies

npm/cli#9262, the change that dropped shrinkwrap support, names the replacement:

use bundleDependencies if you need to ship a locked dependency tree

bundleDependencies copies the named packages, and their dependency subtrees, from the packing machine's node_modules into the published tarball. The installer then uses those files as they are, with no resolution. That behaves the same on every npm version and for both library and application installs.

For harper, it can't be applied wholesale (bundleDependencies: true):

• Native prebuilds are chosen per platform at install time. A bundle captures only the packing host's. On a linux-arm64 pack of 5.3.0, 15 native or platform packages would ship for that platform only: @harperfast/rocksdb-js-*, @lmdb/lmdb-*, @harperfast/hnsw-*, @harperfast/fulltext-*, @cbor-extract/*, @msgpackr-extract/*, argon2, bufferutil, utf-8-validate, systeminformation, and their loaders. Every other platform would break.
• These packages would have to stay regular, exact-pinned dependencies. Only the pure-JS set would be bundled.
• The bundle list has to be closed: no native package can be reachable through a bundled one, otherwise the host's binary gets bundled anyway. The build would need a check that the packed tarball contains no os/cpu/gypfile package.

Measured cost, npm 12 production install of harper@5.3.0 (600 packages, 389 MiB unpacked):

	gzipped
Bundle every dependency	78 MiB
…of which the react-native tree	31 MiB
Bundle after the alasql fix (#2941 step 2)	~47 MiB

The published tarball is 48 MiB unpacked today.

Costs that apply to library consumers specifically:

• Bundled packages are not deduped against the consumer's tree.
• Consumers can't npm update or overrides a bundled package, so a CVE in a bundled transitive dependency needs a harper release to fix.
• Every harper install downloads the full bundle, including for consumers who already have those packages.

Options

1. Drop the shrinkwrap, and lock the application artifacts separately (recommended).
   • The published package becomes a normal library: exact pins for native and load-bearing deps (Pin the load-bearing native and encoder dependencies #2179), ranges for the rest.
   • Application reproducibility moves to artifacts built from harper's own package-lock.json with npm ci, starting with the Docker image (Docker image dependency tree is resolved fresh at build time, not from the published shrinkwrap #1960, Docker image still ships the react-native-fs subtree unpinned (residual from #1960/#2042) #2043).
   • Remaining gap: npm i -g harper / npx harper resolve fresh, which is what already happens for everyone on npm 12 today. If that isn't acceptable, add option 2 for the CLI entry point.
2. Replace it with a partial bundleDependencies.
   • Bundle the pure-JS closure, keep native packages as exact-pinned deps, and enforce "no native package in the tarball" at build time.
   • This is the only option that gives a locked tree to npm i -g on every npm version.
   • Cost: roughly 47 MiB gzipped per install once alasql is fixed, no dedupe, and harper releases gate transitive CVE fixes for all consumers.
3. Keep the shrinkwrap. Not viable. It stops doing anything from Node 27, and until then it is a recurring source of consumer npm ci failures.

I'd go with option 1. The failures we've actually hit came from harper being installed as a library, and bundling makes that case worse (no dedupe, CVE fixes need a harper release) to protect application installs, which already have a better home: a locked image.

Trade-off (option 1)

npm 10 and 11 consumers go back to resolving transitive deps fresh within declared ranges, as npm 12 consumers already do. The react-native tree also comes back for them until alasql is fixed. Both are consistent across npm versions, and neither breaks npm ci.

Measurement: npx npm@12 install --omit=dev --ignore-scripts harper@5.3.0 in node:24 on linux/arm64. The bundle size is node_modules (excluding harper) as a gzipped tar. Native packages were identified by os, cpu, gypfile or an install script in their package.json.

Related: #2172, #2941, #1937, #2179, #2042.

_{sent with Claude Opus 5.5}