[BUG] npm install doesn't create the .bin directory with a v6 package-lock

[closingin]

Hello there, here's a little (but really annoying) bug I just found out !

Current Behavior:

I just upgraded to node@15 and npm@7.

When deleting my node_modules and reinstalling using npm i, npm doesn't create the .bin directory, which ends up in me having no working scripts at all (eg: npm run build, supposed to run next build, but it can't find it obviously).

If I delete my current package-lock.json and reinstall, the .bin directory is correctly re-created, but I can't do that as it breaks my app.

npm bin correctly returns [redacted]/node_modules/.bin.

From what I can see in my git diff, my old package-lock.json does not have the "bin": {} property on packages that have binaries, while the fresh one has it.

Expected Behavior:

npm install should create and populate the .bin directory with older package-lock.json files, and upgrade the lock file.

Steps To Reproduce:

Having a v6 package-lock I guess ? I don't know what's causing this, but I can give you my package-lock.json and package.json if needed.

Environment:

• OS: Arch Linux
• Node: 15.1.0
• npm: 7.0.8 (the bug also exists in previous versions)

[jaydenseric]

We've noticed a few weird things relating to package scripts, npm v6/7, and lockfile v1/2 interoperability:

https://twitter.com/jaydenseric/status/1318321502098345985?s=20

Each time we tried to remember the exact reproduction steps but ended up confused.

[isaacs]

Can you check with 7.0.10? I believe this was likely fixed by a recent arborist update.

[nornagon]

I also encountered this (on macOS, also 7.0.8). Upgrading to 7.0.10 did not fix the issue.

[dr-js]

Got the bug, too.
Don't know how this happens, but here is a preserved "crash site":

• the branch with bad lockfile:
  • https://github.com/dr-js/imagemin-min/tree/test-npm7-lockfile-bin-missing
• the sample CI error with the bad file: (npm ci with npm@7.0.8)
  • https://github.com/dr-js/imagemin-min/runs/1393820432?check_suite_focus=true#step:7:77
• the passed CI with the recreated good file:
  • https://github.com/dr-js/imagemin-min/runs/1393841665?check_suite_focus=true
• the diff of good/bad lockfiles:
  • dr-js/imagemin-min@9ecd3292

And a typical diff looks like: (good file will add bin|engines|funding fields)

        "table": "^5.2.3",
        "text-table": "^0.2.0",
        "v8-compile-cache": "^2.0.3"
+      },
+      "bin": {
+        "eslint": "bin/eslint.js"
+      },
+      "engines": {
+        "node": "^10.12.0 || >=12.0.0"
+      },
+      "funding": {
+        "url": "https://opencollective.com/eslint"
      }
    },

With the bad lockfile, npm@7.0.10 will not create node_modules/.bin/ with either npm i or npm ci.

[closingin]

Hello @isaacs , I just updated my npm to 7.0.10 to test, but npm still doesn't create the .bin directory.

[christianhaller3000]

@closingin what package are your installing?

[closingin]

@closingin what package are your installing?

Uh, I'm just doing a standard npm install. It's not related to a specific package, the bug occurs when you have a "broken" package-lock.json, alongside any package that has a binary.

@isaacs I think I just found how to work around the bug. It may help you in finding the culprit.

When you upgrade to node 15 / npm 7 and do any operation that would update your package-lock.json to "lockfileVersion": 2, npm actually doesn't migrate the file correctly, as it does not add the "bin", "engines", and other properties. I guess those are now required. To work around it, you just need to manually set lockfileVersion to 1, and do a new npm i.

My bold prediction is that npm works correctly when installing, based on what it has in the lockfile. The problem surely comes from the first operation that updates the lockfile to version 2, I think it should either let the lockfileVersion to 1, or migrate the lockfile completely.

[dr-js]

Might be related, I think the bad lockfile is created after apt updated nodejs@15.2.0 deb from nodesource, and I think the process also reset npm to npm@7.0.8.
Before that I used nodejs@15.1.x with manually downgraded npm@6.14.8, so maybe the bad lockfile is created by npm@7.0.8 in a repo when node_modules and v1 lockfile state match, and later with npm@7.0.10 and do install again will not fix the already bad lockfile.

[dr-js]

A test to reproduce the bad lockfile from the itself: (with nodejs@15.2 and npm@7.0.10)
The related test files: test-package-json.zip

mkdir "test-lock/"
cd "test-lock/"
# copy the `package.json` and `package-lock.json-npm708-bad` file here

cp "package-lock.json-npm708-bad" "package-lock.json"

npx --yes npm@6 install
ls "node_modules/.bin/" # there's file
npx --yes npm@6 ci
ls "node_modules/.bin/" # there's file
cp "package-lock.json" "package-lock.json-npm6-good"

npx --yes npm@7.0.10 install
ls "node_modules/.bin/" # there's file because npm do not reset them yet
npx --yes npm@7.0.10 ci
ls "node_modules/.bin/" || echo "no file" # there's no file
cp "package-lock.json" "package-lock.json-npm710-bad-not-fixed"

[dr-js]

Just tested above process with npm@7.0.11 and this should be fixed
A similar issue: #1957
The fix commit: c3e7aa31

[closingin]

I can confirm that npm@7.0.11 fixed this issue!

[closingin]

Apologies: I wrote too fast, issue not fixed. I did my test with @dr-js 's snippet, but the problem still exists on my personal project, using npm@7.0.12.

[ruyadorno]

Hi @closingin do you happen to have a package-lock.json file that was migrated to v2 before updating to npm@7.0.12 ?

As I mentioned over in #1957 if that's the case you might already have a broken file and you would need to force the migration again in order to fix the missing bins issue. If that's the case you can try this trick:

# Make sure you got npm@7.0.11 or newer:
$ npm install -g npm@7

# npm@6 install:
$ npx npm@6 install

# npm@7 install to migrate package-lock file again:
$ npm install

# You should have a fixed-up package-lock.json file now :)

[closingin]

Alright, looks like it worked @ruyadorno, thanks! Sorry that I didn't find the existing issue and recreated one btw.

[ianef]

I've had very similar problems too. I couldn't get it working properly on 6.14.8 and was considering doing some tests with earlier versions.

However I decided to test out the process in a new Debian docker container, copied the basics of the package.json in my project and it built correctly using npm 7.0.14 with a.bin folder. I was confused! I copied the whole of my project package.json and it failed trying to load fortawesome-pro. I copied in my .npmrc and the build completed but there was no .bin folder.

In my .npmrc there was this, I have no idea why!:

bin-links=false

Removing this line solved the problem and the .bin folder was created correctly.