[BUG] GitHub references to dependencies slow/hang

[dlockhart]

Is there an existing issue for this?

• I have searched the existing issues

This issue exists in the latest npm version

• I am using the latest npm

Current Behavior

As of approximately 10AM EST today, installing using a GitHub reference on NPM 6 became slow to the point where it often simply hangs. This was not the case previously, and works on NPM 7/8.

Unfortunately there are other performance issues with the v2 lock file format also related to using GitHub references that are currently preventing an upgrade to NPM > 6.

Expected Behavior

Installing a dependency using a GitHub reference should be possible.

Steps To Reproduce

1. In NPM 6
2. Run npm install github:Brightspace/d2l-fetch or npm install Brightspace/d2l-fetch
3. Both of those either hang or take 7+ minutes to install

Interestingly, npm i git+https://github.com/Brightspace/d2l-fetch.git#semver:^2 installs in 0.7 seconds.

Environment

• npm: 6.14.17
• Node.js: 14.19.2
• OS Name: macOS 12.3.1
• System Model Name: Apple MacBook Pro (2019)
• npm config:

; cli configs
metrics-registry = "https://registry.npmjs.org/"
scope = ""
user-agent = "npm/6.14.17 node/v14.19.2 darwin x64"

; node bin location = /Users/<username>/.nvm/versions/node/v14.19.2/bin/node
; cwd = /Users/<username>/source
; HOME = /Users/<username>
; "npm config ls -l" to show all defaults.

[ljharb]

Seems similar to #4895, but for swapped npm versions. Maybe something on GitHub’s side?

[dlockhart]

This is happening to everyone at our company -- Mac, Windows, inside GitHub Actions runners -- everywhere. (I don't think it's a Mac-specific issue).

[marcesher]

Definitely seeing this issue, starting Friday May 13th sometime early to mid-afternoon Eastern time

Here's what I see now. Installing with git+https:

time npm install git+https://github.com/segment-integrations/analytics.js-integration-segmentio.git#35b24a7505ac05111a045ee07e75096815faaa08
...

+ @segment/analytics.js-integration-segmentio@4.0.0
added 56 packages from 19 contributors and audited 56 packages in 9.827s
found 1 low severity vulnerability
  run `npm audit fix` to fix them, or `npm audit` for details

real	0m10.404s
user	0m1.944s
sys	0m0.701s

and with github::

Note: this isn't even installing from scratch, just running the same npm install with a different protocol

time npm install github:segment-integrations/analytics.js-integration-segmentio#35b24a7505ac05111a045ee07e75096815faaa08
...

+ @segment/analytics.js-integration-segmentio@4.0.0
updated 1 package and audited 86 packages in 463.228s
found 2 low severity vulnerabilities
  run `npm audit fix` to fix them, or `npm audit` for details

real	7m44.153s
user	0m5.764s
sys	0m1.008s

Importantly, this is with an older version... nodejs 12.22.12 / npm 6.14.16

When I try the most recent node... 16.15.0 with npm 8.5.5, I see no difference in time

[grassick]

If you run npm with -ddd you'll see the failure is attempting to run git ls-remote. e.g.

git ls-remote -h -t git://github.com/mWater/jsonql.git

This also runs slow on the command line. But replacing it with:

git ls-remote -h -t ssh://git@github.com/mWater/jsonql.git

or

git ls-remote -h -t https://github.com/mWater/jsonql.git

Are both fast.

So it's trying to use the git protocol, which is odd. It's not supported by GitHub anymore: https://github.blog/2021-09-01-improving-git-protocol-security-github/

[grassick]

Even more disturbing, if you change the dependency to explicitly use ssh:

e.g. "jsonql": "git+ssh://git@github.com:mWater/jsonql.git",

It still times out attempting to get it over git protocol!

e.g.:
npm sill pacote Retrying git command: ls-remote -h -t git://github.com/mWater/jsonql.git attempt # 2

[grassick]

I have a solution that works for me:

Run git config --global url."ssh://git@".insteadOf git://

This appears to configure it to use ssh instead of the git protocol which no longer works.

Experiencing this issue also with Node 14.19.0 & npm 6.14.17.

Before experimenting with the workarounds, would be awesome to resolve why this started just this suddenly 🤷 ...

[Chuckytuh]

We experienced the same slowness on 6.14.11, we found out that updating npm to version 7.16.0 (at least, that was our go to version for other reasons) solved the issue., although it's not clear the root cause.

[slavoroi]

Similar issues happening to our repos with github packages.

node: 16.13.0
npm: 6.14.17 (was updated from 6.14.15)
OS: Ubuntu 20.04

We started experiencing slowness since yesterday.
A simple npm i takes a long long time.

*** Solved it by removing a package named:
"@types/mysql2": "github:types/mysql2"

And updating mysql2 which had types.