Setup multiple local modules with npm link + dedupe

[emmenko]

Hi everybody,

I'm trying to use npm link to develop multiple modules locally but it's not working as expected.
I'm not sure if it's actually possible with this setup or if I'm doing something wrong.
Hopefully someone can shred some light here ;)

Let's say I have this kind of structure.

.
├── app
│   └── main.js
├── modules
│   ├── a
│   │   ├── main.js
│   │   └── package.json
│   ├── b
│   │   ├── main.js
│   │   └── package.json
│   └── c
│       ├── main.js
│       └── package.json
└── package.json

The main app package.json has dependencies of all three modules

{
  "name": "app",
  "version": "0.0.0",
  "main": "./app/main.js",
  "dependencies": {
    "a": "file:./modules/a",
    "b": "file:./modules/b",
    "c": "file:./modules/c"
  }
}

The module a has underscore as only dependencies.

The module b has module a as dependency.

The module c has also module a as dependency.

// module `a`
{
  "name": "a",
  "version": "0.0.0",
  "main": "./main.js",
  "dependencies": {
    "underscore": "1.8.x"
  }
}

// module `b`
{
  "name": "b",
  "version": "0.0.0",
  "main": "./main.js",
  "dependencies": {
    "a": "file:../a"
  }
}

// module `c`
{
  "name": "c",
  "version": "0.0.0",
  "main": "./main.js",
  "dependencies": {
    "a": "file:../a"
  }
}

Scenarios

The easiest way

So if I install the app dependencies I get this

$ npm i
c@0.0.0 node_modules/c

b@0.0.0 node_modules/b

a@0.0.0 node_modules/a
└── underscore@1.8.2

$ tree
.
├── app
│   └── main.js
├── modules
│   ├── a
│   │   ├── main.js
│   │   └── package.json
│   ├── b
│   │   ├── main.js
│   │   └── package.json
│   └── c
│       ├── main.js
│       └── package.json
├── node_modules
│   ├── a
│   │   ├── main.js
│   │   ├── node_modules
│   │   │   └── underscore
│   │   │       ├── LICENSE
│   │   │       ├── README.md
│   │   │       ├── package.json
│   │   │       ├── underscore-min.js
│   │   │       ├── underscore-min.map
│   │   │       └── underscore.js
│   │   └── package.json
│   ├── b
│   │   ├── main.js
│   │   └── package.json
│   └── c
│       ├── main.js
│       └── package.json
└── package.json

Now if I change something in one of the modules I have to re-install them. Running npm i doesn't do anything of course since npm doesn't see any change in the modules definition (e.g.: version bump).

So one thing we can do is rm -rf node_modules && npm i. This will always work but it may be slow depending on your dependency tree.

Funny thing, running npm i -f has a different output.

$ npm i -f
npm WARN using --force I sure hope you know what you are doing.

a@0.0.0 node_modules/a
└── underscore@1.8.2

b@0.0.0 node_modules/b
└── a@0.0.0 (underscore@1.8.2)

c@0.0.0 node_modules/c
└── a@0.0.0 (underscore@1.8.2)
emmenko: ~/dev/src/npm-link-test $ tree
.
├── app
│   └── main.js
├── modules
│   ├── a
│   │   ├── main.js
│   │   └── package.json
│   ├── b
│   │   ├── main.js
│   │   └── package.json
│   └── c
│       ├── main.js
│       └── package.json
├── node_modules
│   ├── a
│   │   ├── main.js
│   │   ├── node_modules
│   │   │   └── underscore
│   │   │       ├── LICENSE
│   │   │       ├── README.md
│   │   │       ├── package.json
│   │   │       ├── underscore-min.js
│   │   │       ├── underscore-min.map
│   │   │       └── underscore.js
│   │   └── package.json
│   ├── b
│   │   ├── main.js
│   │   ├── node_modules
│   │   │   └── a
│   │   │       ├── main.js
│   │   │       ├── node_modules
│   │   │       │   └── underscore
│   │   │       │       ├── LICENSE
│   │   │       │       ├── README.md
│   │   │       │       ├── package.json
│   │   │       │       ├── underscore-min.js
│   │   │       │       ├── underscore-min.map
│   │   │       │       └── underscore.js
│   │   │       └── package.json
│   │   └── package.json
│   └── c
│       ├── main.js
│       ├── node_modules
│       │   └── a
│       │       ├── main.js
│       │       ├── node_modules
│       │       │   └── underscore
│       │       │       ├── LICENSE
│       │       │       ├── README.md
│       │       │       ├── package.json
│       │       │       ├── underscore-min.js
│       │       │       ├── underscore-min.map
│       │       │       └── underscore.js
│       │       └── package.json
│       └── package.json
└── package.json

You can see the nested (duplicate) dependencies in each module.

The symlink way

So npm has the ability to symlink a package folder which is particularly convenient for developing modules locally.
Let's try it then (I will remove first node_modules to have a clean start).

$ cd modules/a
$ npm link
underscore@1.8.2 node_modules/underscore
/usr/local/lib/node_modules/a -> /Users/emmenko/dev/src/npm-link-test/modules/a
$ cd ..
$ cd b
$ npm link
a@0.0.0 node_modules/a
└── underscore@1.8.2
/usr/local/lib/node_modules/b -> /Users/emmenko/dev/src/npm-link-test/modules/b
$ cd ..
$ cd c
$ npm link
a@0.0.0 node_modules/a
└── underscore@1.8.2
/usr/local/lib/node_modules/c -> /Users/emmenko/dev/src/npm-link-test/modules/c
$ cd ..
$ cd ..
$ npm link a
/Users/emmenko/dev/src/npm-link-test/node_modules/a -> /usr/local/lib/node_modules/a -> /Users/emmenko/dev/src/npm-link-test/modules/a
$ npm link b
/Users/emmenko/dev/src/npm-link-test/node_modules/b -> /usr/local/lib/node_modules/b -> /Users/emmenko/dev/src/npm-link-test/modules/b
$ npm link c
/Users/emmenko/dev/src/npm-link-test/node_modules/c -> /usr/local/lib/node_modules/c -> /Users/emmenko/dev/src/npm-link-test/modules/c
$ tree
.
├── app
│   └── main.js
├── modules
│   ├── a
│   │   ├── main.js
│   │   ├── node_modules
│   │   │   └── underscore
│   │   │       ├── LICENSE
│   │   │       ├── README.md
│   │   │       ├── package.json
│   │   │       ├── underscore-min.js
│   │   │       ├── underscore-min.map
│   │   │       └── underscore.js
│   │   └── package.json
│   ├── b
│   │   ├── main.js
│   │   ├── node_modules
│   │   │   └── a
│   │   │       ├── main.js
│   │   │       ├── node_modules
│   │   │       │   └── underscore
│   │   │       │       ├── LICENSE
│   │   │       │       ├── README.md
│   │   │       │       ├── package.json
│   │   │       │       ├── underscore-min.js
│   │   │       │       ├── underscore-min.map
│   │   │       │       └── underscore.js
│   │   │       └── package.json
│   │   └── package.json
│   └── c
│       ├── main.js
│       ├── node_modules
│       │   └── a
│       │       ├── main.js
│       │       ├── node_modules
│       │       │   └── underscore
│       │       │       ├── LICENSE
│       │       │       ├── README.md
│       │       │       ├── package.json
│       │       │       ├── underscore-min.js
│       │       │       ├── underscore-min.map
│       │       │       └── underscore.js
│       │       └── package.json
│       └── package.json
├── node_modules
│   ├── a -> /usr/local/lib/node_modules/a
│   ├── b -> /usr/local/lib/node_modules/b
│   └── c -> /usr/local/lib/node_modules/c
└── package.json

So we can see the symlinks in node_modules, but we also see that each module has duplicated dependencies.

What we actually want is a to be a top level dependency (since is shared between modules) and most importantly that underscore is not duplicated between the other modules.

Again, npm (should) come to the rescue as it provides a way to reduce duplication via the dedupe command.

Searches the local package tree and attempts to simplify the overall structure by moving dependencies further up the tree, where they can be more effectively shared by multiple dependent packages.

Let's run it then and see what happens:

$ npm dedupe
$ tree
.
├── app
│   └── main.js
├── modules
│   ├── a
│   │   ├── main.js
│   │   ├── node_modules
│   │   │   └── underscore
│   │   │       ├── LICENSE
│   │   │       ├── README.md
│   │   │       ├── package.json
│   │   │       ├── underscore-min.js
│   │   │       ├── underscore-min.map
│   │   │       └── underscore.js
│   │   └── package.json
│   ├── b
│   │   ├── main.js
│   │   ├── node_modules
│   │   │   └── a
│   │   │       ├── main.js
│   │   │       ├── node_modules
│   │   │       │   └── underscore
│   │   │       │       ├── LICENSE
│   │   │       │       ├── README.md
│   │   │       │       ├── package.json
│   │   │       │       ├── underscore-min.js
│   │   │       │       ├── underscore-min.map
│   │   │       │       └── underscore.js
│   │   │       └── package.json
│   │   └── package.json
│   └── c
│       ├── main.js
│       ├── node_modules
│       │   └── a
│       │       ├── main.js
│       │       ├── node_modules
│       │       │   └── underscore
│       │       │       ├── LICENSE
│       │       │       ├── README.md
│       │       │       ├── package.json
│       │       │       ├── underscore-min.js
│       │       │       ├── underscore-min.map
│       │       │       └── underscore.js
│       │       └── package.json
│       └── package.json
├── node_modules
│   ├── a -> /usr/local/lib/node_modules/a
│   ├── b -> /usr/local/lib/node_modules/b
│   └── c -> /usr/local/lib/node_modules/c
└── package.json

Hmm, nothing changed.

So is this how it is suppose to work? Does it actually work with local paths and symlinks?

$ npm -v
2.7.3

[smikes]

Since I only use npm link during development, I would tend to approach the problem like this:

Assuming a directory with the checked-out repositories like this:

$ ls 
a      app      b      c

First set up global linked versions of a,b,c

$ (cd a && npm link)
$ (cd b && npm link)
$ (cd c && npm link)

link a to c,b

$ (cd b && npm link a)
$ (cd c && npm link a)

Then add a,b,c dependencies as links

$ (cd app && npm link a b c)

Now - don't look at the tree. It doesn't matter how often you see underscore in the linked tree, because there's really only one version, the one installed under a\node_modules.

If you need to prove it to yourself, then go to a scratch directory, and npm install app, and see that there's only one a and one underscore.

Of course there is another approach, which is just to bump all your packages to version 1.0.0 and then increment your versions when republishing. This way you make the version dependencies explicit.

[ehsalazar]

We haven’t heard back and we’re trying to clean up some old issues.  If this is still a problem, can you please reply and let us know?  We’ll be happy to reopen if necessary.

[hon2a]

I just came across this problem as well and with a scenario that is not easily remedied by the suggested workarounds.

I have 2 modules: lib and app. They both have React as dependency and module app also depends on lib. If I install lib into app using npm install, it works all right, because the top-most React is used in both app and lib files on build (building from app, the lib is just sources). If I don't want to even have the two React installations in my tree, I can just run npm dedupe.

However, since I'm developing both modules side by side, it'd really help me to use npm link instead of hard-installing the modules and then needing to version-bump + update or uninstall + reinstall them on every small change. But when I use npm link, the lib module isn't really inside app folder, so npm dedupe doesn't unify the React dependencies and when building the app using browserify, the app and lib files use each their own local React. Since having two copies of React breaks the app, this makes the npm link approach unusable for me.

[immortal-tofu]

Same issue here. npm dedupe doesn't work with npm link. I didn't find a workaround yet.

[Reinmar]

We've got the same problem in CKEditor. We have multiple plugins requiring each other and the core package where all this is glued together. In this main package I would like to be able to npm link some of the plugins and call npm dedupe in order to flatten the dependency tree. In my case this step is necessary, because I need to build this as this is a client-side app (hence, I need to transpile JS, concatenate and minify what's needed, etc.). Having a non-deduped dependency tree means that I need to include the deduping logic inside my builder. Of course this is doable, but being able to use npm for that would be awesome.

On the other hand, I understand that by default npm cannot "hoist" dependencies from symlinked modules to their parent, because they may have multiple parents. This isn't a thing in my case though, and I think that this is a pretty common case where you work on a certain set of your modules. Perhaps an option could be added to npm dedupe/install so it also works with symlinks?

[patrickheeney]

Has anyone figured out an easy solution for this?

@Reinmar From reading several GH issues, it sounds like you are using gulp to handle some of these scenarios? How did you handle a shared common dependency like React for example?

[gaelollivier]

Same problem here. We're building a webapp that we split in multiple modules. We use webpack to bundle our app.
In development we npm-link our modules to be able to run our webapp from its "main" module. Most of our sub-modules share common dependencies such as AngularJS. Because Angular is installed in each of our sub-dependencies, it ends up being loaded multiple times in our final bundle and that's obviously very bad... Some dependencies just break if we load them multiple times.

I thought npm dedupe could solve our problem but as this issue stated, it doesn't support cleaning "npm-linked" trees.

Here is my webpack-related issue:
webpack/webpack#2134

I actually can't believe that nobody faced this problem before... I'm new to webpack and managing frontend dependencies with npm in general but I was sure it was a common practice, and using npm-link to work on multiple modules at the same time seems a pretty obvious pattern.

Any insights on this please ? :)

[FunkMonkey]

I also stumbled across this issue (especially about the problem that npm dedupe ignores symlinks) and thus created a npm package for it: npm-dedupe-symlinks.

It basically removes the symlinks and moves the according node_modules folders from the symlink targets into the directory structure of the package you want to dedupe. It then simply calls npm dedupe, which can work on a directory structure without any symlinks. Afterwards the node_modules folders (if they still exist after deduping) will be moved back to the original symlink targets and the symlinks will be recreated.

It only operates on one level of symlinks (so only symlinks in the node_modules of the package itself or in a scope) at the moment and will probably blow up if you symlinked the same module multiple times. Still beta, so use it with caution (make a backup of your project first)!

[vjpr]

How about creating a node_modules dir above all your dev dirs. Npm's search alg will recurse up looking for node_modules until your root dir. Then to prevent duplicate requires, you just delete the modules from each of your linked dependencies. You could have a script that would orchestrate this.

A common node_modules means that you will not break other modules that are npm linked to the same module, but don't have it as a peer dependency.

For me I am seeing babel-runtime, lodash, core-js, bluebird required about 5x each on server startup causing 10s server startups. I like to have tons of modules npm linked when working so I can quickly make changes.

Or...has anyone tried override the npm search algorithm...providing a module alias?

I'm going to try it now.