Wiki instructions for updating `npm`'s bundled copy of `node-gyp` need updating

[DeeDeeG]

See this comment: #2354 (comment)

To restate the main changes needed:

• The Windows instructions are incorrect, as they use the -g flag (meaning, to modify a standalone, global install of node-gyp), whereas the instructions are supposed to be telling you how to cd into the npm folder and locally update the copy/copies of node-gyp installed there. The use of the -g flag is inappropriate and leads to npm retaining its old/outdated copy of node-gyp.
• Also, it seems to be necessary (in my testing) to upgrade node-gyp in two places, as multiple packages directly depend on node-gyp; not only npm itself, but the subdependency npm-lifecycle. One must cd into each package and upgrade its node-gyp dependency manually.

[DeeDeeG]

Can someone update the Wiki page to the following?

https://gist.githubusercontent.com/DeeDeeG/f0493c56f020a08e8e1aaf50f763059f/raw/21cd472ba1c51ef48cbf47e37cc560e888bcadc0/Updating-npm's-bundled-node-gyp.md

• I added the second npm install node-gyp in the npm/node_modules/npm-lifecycle folder.
• And I simplified the Windows instructions a lot, as it's not actually much more complicated on Windows. The only "complicated" thing is having to run cmd.exe as Administrator instead of having the convenience of sudo. npm explore works exactly the same on Windows. (At least for me it does.)

(Here is the diff if anyone is curious...
https://gist.github.com/DeeDeeG/f0493c56f020a08e8e1aaf50f763059f#file-updating-npm-s-bundled-node-gyp-md-diff)

[DeeDeeG]

Arg, never mind the previous comment. The complicated instructions are still needed on Windows if you use the installer. (I have always just downloaded the .zip archive and extracted it on Windows before now...)

[DeeDeeG]

Updating the wiki page to this should work:

https://gist.githubusercontent.com/DeeDeeG/d95a79b590dae05fa7cbbe4fd34c06b2/raw/7659c3a5f2deacb38dfb63b05e9fee87c36c92b4/Updating-npm's-bundled-node-gyp.md

Diff: https://gist.github.com/DeeDeeG/d95a79b590dae05fa7cbbe4fd34c06b2#file-git-diff-diff

[cclauss]

@DeeDeeG Please review the changes made https://github.com/nodejs/node-gyp/wiki/Updating-npm's-bundled-node-gyp

[cclauss]

On supported versions of Windows, do we need to suggest both which and where?
Or has this been resolved to just one of them?

[DeeDeeG]

@cclauss good catch... We can simplify to just where.

Only Windows 7 and newer work with the latest node-gyp. I think it has been that way for a long time. (The most-recent time I researched this: #2333 (comment))

And I have found a couple of sources saying where was added in Windows Vista:

• https://web.archive.org/web/20070423121442/http://computerbits.wordpress.com/2006/07/21/new-command-line-tools-in-vista-beta-2/
• https://www.lifewire.com/list-of-command-prompt-commands-4092302

So, all versions of Windows that work with the latest node-gyp support where.

[DeeDeeG]

It's a bit more complicated considering both npm 6 and 7, the two major versions that are in wide circulation...

npm 7 doesn't have npm-lifecycle anymore, it has @npmcli/run-script. And during npm explore --global, it sets an env var npm_config_global=true that affects the subshell, meaning our subsequent npm install node-gyp command is effectively npm install --global node-gyp... 🤦

Considering all that, this works on npm 7:

npm explore -g npm/node_modules/@npmcli/run-script -- npm_config_global=false npm install node-gyp@latest

Whereas I can confirm the command you put in the wiki works for npm 6:

npm explore -g npm/node_modules/npm-lifecycle -- npm install node-gyp@latest

[cclauss]

OK. I dropped which in favor of where. Can you please tell me exactly what the last part of the page should say? (I do not have a Windows box.)

npm --version

• If the npm version is less than 7 do X.
• If the npm version is greater than or equal to 7 do Y.

[DeeDeeG]

The difference is where (what directory) to cd into.

Default locations based on the official NodeJS installer:

C:\Program Files\nodejs\node_modules\npm\node_modules\npm-lifecycle (for npm 6) or
C:\Program Files\nodejs\node_modules\npm\node_modules\@npmcli\run-script (for npm 7)

These are the steps I would recommend, for ease of reading/following them...

• In an Administrator cmd prompt...
• where node
  • cd [path\to\node.exe, minus the node.exe], i.e. cd into the folder containing node.exe
    • For example, to cd into the default install path: cd "C:\Program Files\nodejs"
• npm --version
  • If the npm version is less than 7, do:
    • cd node_modules\npm\node_modules\npm-lifecycle
  • If the npm version is 7 do:
    • cd node_modules\npm\node_modules\@npmcli\run-script
• npm install node-gyp@latest

[cclauss]

Is https://github.com/nodejs/node-gyp/wiki/Updating-npm's-bundled-node-gyp the right stuff now?

[DeeDeeG]

Thanks for updating it again.

We need the npm 6 vs npm 7 split in the Unix instructions as well:

npm --version, then

if npm <7 :

$ [sudo] npm explore npm/node_modules/npm-lifecycle -g -- npm install node-gyp@latest

if npm 7:

$ [sudo] npm explore npm/node_modules/@npmcli/run-script -g -- npm_config_global=false npm install node-gyp@latest

Nits:

• It says "i.e." in the Windows instructions, but I think it's better to say "e.g." or "for example", since the actual path can be completely arbitrary, and we are only giving the default example.

• I mildly disagree with saying "npm 7 or greater," because we can't really predict how npm 8 will or won't differ from npm 7. The whole reason the logic is split between 6 and 7 is that they rewrote some of their dependencies from scratch, and we have to update node-gyp under the new dependency for npm 7. (That said, there is some chance npm 8 would use the same dependency, just like it was kept in the transition from npm 5 to npm 6... I dunno). There is a tradeoff between trying to be future-proof vs being conservative and only talking about existing versions of npm.

[cclauss]

Will users be squeamish about upgrading npm?
If it is < v7, why not just recommend npm install -g npm which would simplify the remaining instructions?

[DeeDeeG]

I would be squeamish about it personally, but I will admit I am biased.

For Unix/in general: I am not sure how others feel about it... I personally dislike npm 7, as it changes a lot of the finer points of npm's behavior compared with 6. (It has also had many bugs, several of which I've encountered/reported, especially relating to the strange edge cases a node-gyp contributor in particular can experience...) npm 7 has a new lockfile format, so switching between 6 and 7 re-writes the lockfile with a huge diff each time. So it is a bit obnoxious to mix use of npm 6 and 7 on a multi-developer team. I personally have to really think about which I'm using. For some projects I have deliberately stayed on 6 to match other contributors. In general I am much more confident in npm 6 than 7, so I avoid 7 for less hassle and headaches. (My sense of the "hassle" is amplified though, because I feel a duty to understand and report bugs I encounter. This is a big part of my bias against npm 7: I feel like they should have done more Quality Assurance and called for wider testing before hitting publish, and I don't enjoy doing a lot of it myself.) One point in favor of npm 7: It is faster.

That said, it is mostly drop-in compatible. Some contributors/maintainers don't mind a huge (essentially meaningless) diff to package-lock.json, or they don't track package-lock.json in git. And I do a lot more weird/advanced things with npm than most people probably do, so I hit more bugs than the average person.

For Windows: I would advise against npm install -g npm on Windows. npm install -g npm doesn't work well on Windows, and there are entire programs dedicated to doing it right: https://www.npmjs.com/package/npm-windows-upgrade. I suppose folks can run that, but I think it's more trouble than it's worth...

[DeeDeeG]

Given that there are practical reasons for a user to want (perhaps need) to work with npm 6, and given than npm 6 is shipped with Node 12 and 14 LTS, and given that upgrading is not trivial on Windows, I personally think we should provide instructions for both npm 6 and 7.