[BUG] npm link removes previously linked module

[mshima]

What / Why

Developing coupled packages linked with npm link is not optimal.

1. npm link breaks the linked package.
2. npm link removes others links. (regression from npm 6)

When

Developing/testing linked packages together

Where

• n/a

How

Current Behavior

• n/a

Steps to Reproduce

Preparing the environment

git clone https://github.com/yeoman/environment.git
cd environment
npm ci
npm link
cd ..
git clone https://github.com/yeoman/generator.git
cd generator
npm ci
npm link
cd ..
git clone https://github.com/yeoman/yeoman-test.git
cd yeoman-test
npm ci
npm link
cd ..

Reset the environment

cd environment
npm ci
cd ..
cd generator
npm ci
cd ..
cd yeoman-test
npm ci
cd ..

1. npm link breaks the linked package.

cd yeoman-test
npm link yeoman-generator
npm test

output:

Error: Cannot find module 'semver'
Require stack:
- /Users/mshima/git/reproduce/generator/lib/index.js
- /Users/mshima/git/reproduce/yeoman-test/test/helpers.js

Workaround go to generator folder and execute npm ci

2. npm link removes others links. (regression from npm 6)

cd yeoman-test
npm link yeoman-generator
ls -l node_modules/yeoman-*

output:

lrwxr-xr-x  1 mshima  staff  15  6 Mar 12:14 node_modules/yeoman-generator -> ../../generator

node_modules/yeoman-environment:
total 24
drwxr-xr-x   4 mshima  staff   128  6 Mar 12:14 cli
.
.
.

npm link yeoman-environment
ls -l node_modules/yeoman-*

output:

lrwxr-xr-x  1 mshima  staff  17  6 Mar 12:20 node_modules/yeoman-environment -> ../../environment

node_modules/yeoman-generator:
total 32
-rw-r--r--   1 mshima  staff  1292 26 Out  1985 LICENSE
drwxr-xr-x   5 mshima  staff   160  6 Mar 12:20 lib
.
.
.

Workaround: link both together npm link yeoman-generator yeoman-environment

ls -l node_modules/yeoman-*
lrwxr-xr-x  1 mshima  staff  17  6 Mar 12:20 node_modules/yeoman-environment -> ../../environment
lrwxr-xr-x  1 mshima  staff  15  6 Mar 12:22 node_modules/yeoman-generator -> ../../generator

Expected Behavior

1. should not break node_modules tree.
2. should not remove others manually linked modules.

Who

• n/a

References

• n/a

[ljharb]

This repo is for arborist; this sounds more like something that belongs on https://github.com/npm/rfcs or https://github.com/npm/cli

[isaacs]

@ljharb Well, to be fair, if there's a bug in the workflows described, it almost certainly is a bug in Arborist, so this isn't an entirely inappropriate place to post it.

@mshima We just published a new npm version with a fix that might be relevant here. Can you npm i npm -g and retry? If it still occurs, please let us know. In the meantime, I'll try to reproduce and see if anything jumps out at me. I think I can figure out what you're going for, but a description of the expected behavior would be good, too.

[mshima]

@isaacs I've tried the npm 7.6.2 and the problem persisted.
I've updated the expected behavior but it's really generic, since I don't know the consequence of the real fix.

IMO the real fix is to don't touch manually linked leafs.
Manually linked modules should be considered a valid (npm ci was executed) and independent tree and should not touched.

First problem:

It happens when a dependency of a linked module is deduped and moved down the tree.
The linked module have a dependency on semver, which became unreachable to this module since it deduped from:
node_modules/yeoman-environment/node_modules/semver
to
node_modules/semver

The consequence is that yeoman-environment cannot find semver because it's not in it's node_modules tree anymore.

Second problem:

A manually linked module should not be overridden, in this case a second npm link removed the first link.

The first problem happens in npm 6, while the second is a regression.

[Hornwitser]

I ran the steps to reproduce this in my dev environment (Arch Linux, npm v8.3.0, node v17.3.0) and the part 1 problem of npm breaking packages on npm link did not happen on my system. npm test from yeoman-test reported 217 passing tests.

I encountered the part 2 problem in my own package setup when npm link package-a followed by npm link package-b resulted in only package-b being linked which is rather annoying. Possible workaround as pointed out by this StackOverflow answer is to use the completely undocumented feature of passing multiple packages to link with by using npm link package-a package-b.

Edit: Another thing that makes working with linked packages annoying is that npm i <pkg> also clears out linked packages, which means having to redo the link(s) every time you add, remove or update a dependency. It seems logical to me that once you've established a link it should stick until you explicitly remove it, or do something like npm ci.

[rendmath]

If my tests are correct, this bug has the worst effect in the context of workspaces, because it prevents you from linking different external packages to different workspaces. I tried working around it using --install-strategy but without success so far.

[stouch]

Related to npm/npm#17287 .

Please...: npm/npm#17287 (comment)