difference between npm and yarn behavior with nested dependencies

[sthuck]

Hi all,
I found a slight difference in behavior between npm (v4) and yarn 0.24.5 to yarn 0.27.5.

my package.json dependencies has something like this:

    "@types/angular": "~1.5.0",
    "@types/angular-mocks": "^1.5.0"

now @types/angular-mocks package.json looks like this:

 "dependencies": {
        "@types/angular": "*"
    },

In npm v4 and yarn 0.24.5, the @types/angular: "~1.5.0" I have in my package.json, satisfies this dependency, so I have only 1 @types/angular installed, in node_modules/@types/angular, and it's version is 1.5.23. Which is great.

But in yarn 0.27.5, I think the "@types/angular": "*" is being interpreted as a dependency on "latest".

Whatever the reason is, this causes me to now have 2 @types/angular packages.
Version 1.5.23 in node_modules/@types/angular, and version 1.6.26 (latest) in node_modules/@types/angular-mocks/node_modules/@types/angular. This obviously causes issues for typescript.

Im not sure if this is the intended behavior... If it is the intended behavior, and if someone by chance has an idea which commit is responsible for this change, I would love to know. Like I wrote, in 0.24.5 this issue didn't happen. I tried finding it myself but without success.

node: v6.9.5, npm: 4.1.2, yarn: 0.27.5

[BYK]

Hmm this is interesting. I think we treat * as latest but it actually means "any would do". @arcanis @kaylieEB @voxsim can you back me up on this?

If so I think we have a bug in our hands.

[arcanis]

I wouldn't class this as a bug. We have no strict guarantee on how the hoisting process will put things in the end - for what it's worth, we could implement a --no-hoisting flag and your use case would break similarly.

I feel like the fix should be on @types/angular, not us - they just have to use a peerDependencies entry instead of dependencies.

[BYK]

@arcanis - semver clearly defines * as any version: https://github.com/npm/node-semver#x-ranges-12x-1x-12-

No matter how and what we hoist, we should accept anything for *, right?

[arcanis]

Yes and no. This is a bit tricky.

• Yes, using * means that the hoister is indeed allowed to consider that the dependency can be safely merged with any other version if it wants to

• However, it doesn't mean that the hoister is required to do so. Semantically speaking, a dependencies entry literally means "this package MUST be able to require a package of version $VERSION". There's absolutely no mention about this version being compatible with any other. If you need this behavior, then you have to use peerDependencies, which provide the extra guarantee that the selected version MUST be the same as the one provided by the parent package (which is exactly what is asked for in the first post).

So, to sum up: the hoister is about trimming the fat and avoiding filling your disk with garbage - not ensure your dependencies are correctly setup (that's the responsibility of package authors). Ideally, your code should work with no hoisting, or with super-weird hoisting that would create ten levels of indirection. If you don't, it will eventually break, whether it's in Yarn or in another package manager.

Of course, we could potentially fix this (most likely not without spending some serious time, and breaking other behaviors). But I fear that by doing so, we would actually enable this kind of "We can do whatever we want, the package managers will have to make it work anyway" behavior, which I think is detrimental to everyone in the long term (Yarn, but also other package managers who would have to have the exact same behavior, and package consumers that would have to use sub-par package managers that can't apply aggressive optimizations because some packages do not play by the rules - we've already been hit by that in a few occasions). 😉

---

Now, from an implementation viewpoint, here is what causes the behavior described in the OP. First, you have to understand that what we called the "hoister" is actually splitted in two different parts:

• The optimizer currently runs during resolution, and tries to avoid resolving ranges that have already been satisfied by previous resolution requests, but the requests are mostly independants (so for example, in the OP situation, when we resolve angular, we have no idea that a sub-dependency has also a dependency on angular, and conversely). The result of the optimizer is what is stored inside the lockfile.

• The hoister runs once the resolution (and optimization) has completed, and I think simply tries to merge together packages that share a exact same version, without any consideration for their original range.

So during the first step, both angular versions are resolved to their respective versions, independently from each other, then the hoister tries to merge them, see that their versions don't match, and just doesn't merge them. And as described in the first part of this comment, this is to be expected. If you want them to be merged, you would instead use a peerDependencies entry, which would not be resolved and wouldn't suffer the issue described here.

[BYK]

@sthuck - Okay, we've had an offline chat with @arcanis about this and now I understand why he is right and this is the expected behavior. Here's the thing: hoisting is not a requirement, it is an optional optimization. So if a dependency lists a sub-dependency as *, it should still be okay getting a different version of it being installed, just for itself. If the expectation is to have a single copy of that sub-dependency and not care about the version, then it should be listed as a peerDependency.

Does this make sense?

[sthuck]

@BYK yep. Will open a PR with DefinitelyTyped to fix their dependencies.