Native packages rebuilt every time

[jiripospisil]

Do you want to request a feature or report a bug?

Bug.

What is the current behavior?

It seems that all native packages are rebuilt every time yarn is asked to either add a new package or just install the currently locked.

If the current behavior is a bug, please provide the steps to reproduce.

1. Add some native packages.

yarn add leveldown bcrypt

1. Run yarn again and observe that both of the packages will be rebuilt for no reason.

yarn

The same happens when adding a completely unrelated packages which, as far as I can tell, cannot affect the native packages in any way.

yarn add co

What is the expected behavior?

Native packages should not be rebuilt if there's no reason to do that. Note that #248 seems pretty similar.

Please mention your node.js, yarn and operating system version.

Node.js 6.7.0
Yarn 0.15.1
Ubuntu 12.04

[jkimbo]

Can't reproduce this in Mac OSX (10.11.6) so might be a Ubuntu specific issue?

[Daniel15]

I could repro on Windows 10, but only once. The second time I ran "yarn", it didn't rebuild the native libraries. Strange.

[jiripospisil]

I was playing with it some more and came up with a few more details:

1. If I run yarn add leveldown bcrypt, leveldown will be compiled as the first item in the list and the hash in node_modules/.yarn-integrity will equal to 595306... when it's finished (cut for brevity; I assume that's a checksum which determines whether something needs to be done). Now if I run just yarn again, both of the packages will be rebuilt with bcrypt being compiled as the first in the list (the order is different). The resulting checksum will equal to cbe480.... Subsequent invocation of yarn will not rebuilt the packages and the checksum will stay the same. This contradicts my original report (I probably made a mistake somewhere) but aligns with what @Daniel15 was seeing.
2. When I reverse the order of the packages right from the start (yarn add bcrypt leveldown), bcrypt will be first in the list and the resulting checksum will equal to cbe480... immediately. Subsequent invocation of yarn will not rebuilt the packages.
3. The bcrypt package will always finish first (as expected since there's simply not that much to compile) regardless of the position in the list.

I'm not at all familiar with the internals but it seems to me that the order in which the packages are compiled matter and they are simply not being sorted when first installed (and they are sorted when later invoking yarn) which affects the checksum in some way.

[Daniel15]

Thank you for investigating! That sounds like a good lead. The hash is written here:

yarn/src/cli/commands/install.js

Lines 581 to 583 in f04b157

	const loc = await this.getIntegrityHashLocation();
	invariant(loc, 'expected integrity hash location');
	await fs.writeFile(loc, this.generateIntegrityHash(lockSource, patterns));

. It might be worth debugging that code and seeing what's different in the lockfile, as the hash in .yarn-integrity is based off the lockfile.

[jiripospisil]

It might be worth debugging that code and seeing what's different in the lockfile, as the hash in .yarn-integrity is based off the lockfile.

I suspected that but what threw me off was the fact that the lock file doesn't change, it's always the same. It's just the hash in .yarn-integrity that changes.

$ yarn add leveldown bcrypt
$ cat node_modules/.yarn-integrity
59530680cc0a4ee12feb373980b5abf2edf2fe2aefa16d556bfb531af8299e71
$ cp yarn.lock leveldown_bcrypt_initial.lock
$ yarn
$ cat node_modules/.yarn-integrity
cbe48058288527f95dfe5643976b0735ee784860663e93ed782b98477c824ec1
$ cp yarn.lock leveldown_bcrypt_afterwards.lock
$ diff -s leveldown_bcrypt_initial.lock leveldown_bcrypt_afterwards.lock
Files leveldown_bcrypt_initial.lock and leveldown_bcrypt_afterwards.lock are identical

$ yarn add bcrypt leveldown
$ cat node_modules/.yarn-integrity
cbe48058288527f95dfe5643976b0735ee784860663e93ed782b98477c824ec1
$ cp yarn.lock bcrypt_leveldown_initial.lock
$ yarn
$ cat node_modules/.yarn-integrity
cbe48058288527f95dfe5643976b0735ee784860663e93ed782b98477c824ec1
$ cp yarn.lock bcrypt_leveldown_afterwards.lock
$ diff -s bcrypt_leveldown_initial.lock bcrypt_leveldown_afterwards.lock
Files bcrypt_leveldown_initial.lock and bcrypt_leveldown_afterwards.lock are identical

$ diff -s leveldown_bcrypt_initial.lock bcrypt_leveldown_initial.lock
Files leveldown_bcrypt_initial.lock and bcrypt_leveldown_initial.lock are identical

[hustcer]

I have the same issue: I use bcrypt too. Every time when I install some new modules or upgrade exist ones, I have to run npm rebuild to make my app runnable.

macOS 10.12 && node v7.0.0 && yarn v0.16.1

[jiripospisil]

I can no longer reproduce the original issue. It seems to have been fixed 👍. @Daniel15 Can you confirm?

@hustcer I don't think that's the same issue. You might want to test with the latest version and file a new bug if it's still not working for you.

[hustcer]

@jiripospisil Thanks, It's okay now after upgrade to yarn v0.17.4.

[seansfkelley]

I'm still seeing this, or something very similar, on 0.18.1. Incidentally, it's also leveldown that keeps getting rebuilt repeatedly. Using leftpad as a package with no dependencies (and that is not depended on by leveldown) for demonstrative purposes, repro steps are as follows:

mkdir test && cd test
echo '{ "name": "test", "version": "1.0.0" }' > package.json
yarn add leveldown 
yarn add leftpad
yarn remove leftpad

My output when I run this follows. Note that removing leftpad takes almost 40 seconds, the majority of which is rebuilding leveldown. This happens consistently, with and without leveldown or leftpad in the Yarn cache, though only during remove and never add.

$ mkdir test && cd test
$ echo '{ "name": "test", "version": "1.0.0" }' > package.json
$ yarn add leveldown 
yarn add v0.18.1
info No lockfile found.
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
success Saved 145 new dependencies.
<... package list omitted for brevity ...>
✨  Done in 49.93s.
$ yarn add leftpad
yarn add v0.18.1
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
success Saved 1 new dependency.
└─ leftpad@0.0.0
✨  Done in 2.18s.
$ yarn remove leftpad
yarn remove v0.18.1
[1/2] Removing module leftpad...
[2/2] Regenerating lockfile and installing missing dependencies...
success Uninstalled packages.
✨  Done in 38.76s.
$ yarn why leftpad
<... just to confirm that leftpad and leveldown are entirely unrelated ...>
yarn why v0.18.1
[1/4] 🤔  Why do we have the module "leftpad"...?
[2/4] 🚚  Initialising dependency graph...
warning test@1.0.0: No license field
[3/4] 🔍  Finding dependency...
error We couldn't find a match!
✨  Done in 0.30s.

Versions:

Node v7.3.0
Yarn 0.18.1
OS X El Capitan (10.11.6)

[ndraiman]

Please reopen as this still happens.
Just did yarn add redux and it rebuilt bcrypt, node-sass and several others.

Node v6.9.4
Yarn v0.20.3
Windows 10

[jiripospisil]

@seansfkelley I followed your steps with the latest version and it worked. Could you try again?

/tmp/tp-20170222100611/test ∴ echo '{ "name": "test", "version": "1.0.0" }' > package.json
/tmp/tp-20170222100611/test ∴ yarn add leveldown
yarn add v0.20.3
info No lockfile found.
warning test@1.0.0: No license field
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
success Saved 54 new dependencies.
<cut>
warning test@1.0.0: No license field
✨  Done in 1.64s.
/tmp/tp-20170222100611/test ∴ yarn add leftpad
yarn add v0.20.3
warning test@1.0.0: No license field
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
success Saved 1 new dependency.
└─ leftpad@0.0.0
warning test@1.0.0: No license field
✨  Done in 0.69s.
/tmp/tp-20170222100611/test ∴ yarn remove leftpad
yarn remove v0.20.3
[1/2] Removing module leftpad...
[2/2] Regenerating lockfile and installing missing dependencies...
warning test@1.0.0: No license field
success Uninstalled packages.
✨  Done in 0.66s.
/tmp/tp-20170222100611/test ∴ yarn why leftpad
yarn why v0.20.3
[1/4] 🤔  Why do we have the module "leftpad"...?
[2/4] 🚚  Initialising dependency graph...
warning test@1.0.0: No license field
[3/4] 🔍  Finding dependency...
error We couldn't find a match!
✨  Done in 0.20s.

@nexxado Could you please add a few reproduction steps? I tried a few combinations but it worked.

Node 7.3.0
Yarn 0.20.3
MacOS 10.12.3

[ndraiman]

@jiripospisil I have no reproduction steps to provide, simply installing an additional package causes yarn to link and rebuild everything.

Here's the output of adding one package (lock file already exists):

Linking dependencies:

Rebuilding: