Yarn add attempts to resolve packages we didn't ask it to install

[steelbrain]

Do you want to request a feature or report a bug?
This is a bug report
What is the current behavior?
The current behavior is, if we do lerna add randomcolor and have a package.json file where we have a few modules that are non-existent in the NPM registry, and are local and to be linked by lerna a few seconds later, It tries to resolve those non-existent modules and fails.

We didn't ask lerna to install those modules so it shouldn't care about them, it should only throw the error when it couldn't find one of the packages we asked it to install
If the current behavior is a bug, please provide the steps to reproduce.
Described above
What is the expected behavior?
Described above
Please mention your node.js, yarn and operating system version.
This is not specific to the Node.js version or OS. I'm using Yarn@0.15.1

[wycats]

I don't think this is quite right, because ultimately yarn wants to understand the complete dependency graph, and the not-yet-linked packages may affect it.

Is there a good place I can learn about the strategy lerna is using to automate the linking process?

[wycats]

cc @tomdale I think you use lerna?

[steelbrain]

@wycats while I do not quite understand deep workings of Yarn, but let's imagine a dependency to be linked x has 10 more dependencies. The thing in this particular scenario is, none of them are a problem because the user did not ask yarn to install any of them, and in cases of links, the linked packages will have their separate dependencies in a different node_modules anyway :)

So I personally think that if Yarn could just ignore it, that would be wonderful and more in line with the NPM behavior

[tomdale]

@steelbrain I think the problem here is that Yarn, by design, treats the package.json as the source of truth. In package.json, specifying a version range implicitly means "the npm registry is the source of truth" (as opposed to a GitHub URL or file path, for example). Putting the packages in package.json with a version tells Yarn "go find these packages on npm", and it rightfully fails when it can't find them.

As a Lerna user myself, my recommendation would be to just publish the packages to npm. If they're not ready for public consumption, you can always just publish a stub package at version 0.0.1. What's the reason they aren't in npm now?

[steelbrain]

@tomdale Now that I'm thinking about it. Instead of publishing meta packages to npm, if we change the order of execution in lerna that is currently

1. Npm install dependencies
2. Link locals

to

1. Link locals
2. Npm install dependencies

Will it work? Because while the stuff will not be on the registry, but will still be available in node_modules/$name/package.json?

[tomdale]

Let me try to make another analogy that I hope helps.

In React, the model is to construct a tree of virtual DOM from component state. This tree of virtual DOM is then diffed against the previous version, and a sequence of update operations is applied to the real DOM. But in this sense, the DOM is just dumb output—the side effect of computing virtual DOM diffs.

In Yarn, the package.json is the source of truth, like component state in React. As you go through package.jsons, you build up a graph of dependencies, which in this analogy would be the virtual DOM. Only at the very end do you persist that graph of dependencies to disk in the form of the node_modules directory.

In this case, asking Yarn to look at the file system as a secondary source of truth is like asking React to sometimes pull data out of the DOM in case a jQuery plugin had changed it. Not only is it an awkward inversion of the model, it is likely to have consequences that prevent many of the performance and correctness benefits of Yarn.

(Which is not to say that the problem isn't worth solving, just that I think we need some way of expressing the desired semantics in the package.json; I don't think falling back to examining the file system is going to pay dividends long term.)

[tomdale]

@steelbrain I made my above comment before reading yours, sorry about that. Your suggestion to link first, then install dependencies, was my first thought too. But I'm not sure it works because I think Yarn would still check the npm registry. Maybe someone who is more familiar with the implementation can chime in.

[jamiebuilds]

The problem here is actually that you can't install individual packages without updating the dependency in package.json. Lerna currently does this:

npm install external-package-1 external-package-2 external-package-3
ln -s ...
ln -s ...
ln -s ...

In yarn you can only either do:

yarn install

Which installs everything or

yarn add/upgrade package-1 package-2 ...

If we could run installs on a limited set of packages that would fix the issue but at the same time I don't want to add that to yarn

[superherointj]

@thejameskyle What is your concern on adding custom multiple modules install to yarn?

[jamiebuilds]

That's not the concern, the concern is allowing an install which adds dependencies to your node_modules that doesn't update package.json and yarn.lock

[jamiebuilds]

Maybe we can add a yarn add <package...> --no-update-package-implicit-deps-something-flag command

[ccorcos]

As I understand it, the reason Yarn will look in the registry first is so that it can generate a proper lock file, and that makes sense.

First, let me describe a typical situation when you're developing two packages together than have a dependency on each other: Suppose you have a package X that depends on Y, and you want to develop a new version of Y. You might clone Y, yarn link it to your local filesystem, and then yarn link Y inside of X. Then you can make changes to both X and Y. But what happens when you add a new dependency to Y? Those changes don't propagate to the X lockfile. But that's fine because X only depends on the old version of Y and we haven't published the new version of Y yet. So we publish a new version of Y and then update X's dependency on Y and then X's lockfile is update appropriately.

That workflow all makes sense to me and its reasonable that there's this limbo state of mis-matched lockfiles because we're amidst making changes.

Now, let me describe the same situation but you're just getting started with building these package. That is, suppose X and Y don't yet exist on NPM and you're just creating the first version. Therefore, if X depends on Y@0.0.1, we cannot generate a proper lockfile yet because Y@0.0.1 still needs to be developed. The problem is, when we try to link these projects together, we get this error that Y isn't in the registry so we're stuck. So I think it's OK to let these lockfiles go out of sync and leave it to the develop to make sure they know what they're doing. So that right before they publish the very first Y@0.0.1 and X@0.0.1, they can run yarn in both to get the latest dependencies.

Let's also example another scenario -- suppose we have these same packages, X and Y, but we never plan on publishing X or Y. Maybe we're building a project for a client and we want to organize the project into separate modules just for the sake of organization and reusability. In this case, we can't use Yarn without actually publishing these packages -- thats annoying.

In the last, what if X and Y were called app and build-system. Those seem like perfectly reasonable package names if I never plan on publishing them. However, both of those package names exist in the registry. And what's going to happen when link build-system to app is Yarn is going to look up build-system in the registry and create a lockfile in app based on the build-system projects that in the registry -- not the one I have in my local filesystem and that certainly isn't the intention.

If you've made it this far, I hope we're on the same page that perhaps Yarn should allow resolving linked local packages. If not, I'm happy to clarify.

So here's what I think ought to be changed:

• if we run yarn link Y; yarn add Y, then Yarn should skip looking up the project in the registry because we intend it to use the package in our local filesystem.
• if we run yarn add Y; yarn link Y, then we need to regenerate the lockfile using the package from our local filesystem, otherwise the package on the registry may be an entirely different package (e.g. if we're using generic local package names like app or build-system)
• if we run yarn inside a project that has linked packages, then we should re-scan the dependencies of the linked project to regenerate the lockfile, ensuring that the linked package is the same version listed in package.json. that way, if we want to publish new versions of both X and Y, we can bump all the versions appropriately (using lerna perhaps) and then re-run yarn in both projects to generate the proper lockfile before committing and publishing.

I'm keep to Lerna working with Yarn ASAP, so let me know what you think and any tips for where to look if you want me to submit a PR.

[ccorcos]

Just forked yarn to try to do this... I don't even know where to start.