[BUG] npm ci gives timeout error on gitlab runner

[ghostx31]

Is there an existing issue for this?

• I have searched the existing issues

This issue exists in the latest npm version

• I am using the latest npm

Current Behavior

We are running using the node:18 docker image in our CI pipeline running on self hosted Gitlab runner. The runner is hosted on a VM running on GCP and our gitlab is running on GKE v.127.

We started having this error a week ago. This is not a runner issue since we are able to install packages using pip and mix deps.get for elixir and only npm fails. We are not using any proxy and internet is reachable from docker containers.

The runner is configured with the docker executor.
The exact error is:

$ npm ci
npm ERR! code ETIMEDOUT
npm ERR! syscall connect
npm ERR! errno ETIMEDOUT
npm ERR! network request to https://registry.npmjs.org/yargs-parser/-/yargs-parser-20.2.9.tgz failed, reason: connect ETIMEDOUT 104.16.24.34:443
npm ERR! network This is a problem related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network settings.
npm ERR! network 
npm ERR! network If you are behind a proxy, please make sure that the
npm ERR! network 'proxy' config is set properly.  See: 'npm help config'
npm ERR! A complete log of this run can be found in: /root/.npm/_logs/2023-12-12T04_38_02_576Z-debug-0.log
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1

We were able to solve this issue by setting network_mode="host" in the gitlab runner's config but this breaks the container services communication which we need for the builds.

Expected Behavior

Running npm ci should fetch the packages from npm registry correctly.

Steps To Reproduce

• Setup Gitlab self hosted on GKE cluster v.127 and add a runner on a GCP VM.
• Using this Gitlab CI snippet:

.app-build: &app-build
    stage: app-build
    image: node:18
    before_script:
        - npm ci
    script:
        - npm run build

• Produces the error

Environment

• npm: 10.2.3
• Node.js: node:18 docker image - v18.19.0
• OS Name: Debian 11
• npm config:

; node bin location = /usr/local/bin/node
; node version = v18.19.0
; npm local prefix = /builds/our/project
; npm version = 10.2.3
; cwd = /builds/our/project
; HOME = /root
; Run `npm config ls -l` to show all defaults.

[zheng1716148634]

yes，me too；I also experience timeout errors；I tried many times, but it didn't work；

[Larsjep]

Hi,
This might be related to #7072 . We saw a similar problem on our internal proxy cache due to high number of connections opened by npm.

[lxfu1]

The same problem.

[ghostx31]

Hi @Larsjep, thanks for pointing out your issue. Just wanted to confirm how did you check how many connections npm install was opening?

[Larsjep]

Hi @Larsjep, thanks for pointing out your issue. Just wanted to confirm how did you check how many connections npm install was opening?

I'm using wireshark and noticing when and how many SYN packages it sends.

[ghostx31]

I see. Using node:20 image seems to fix this issue for us but I only tested this on a single project. I do not know if this breaks any dependencies for us right now so will need to test more on other projects as well.
I'd be interested in knowing what the root issue in here, from npm maintainers if they get around to this.

[heath-freenome]

We were seeing the same issue on our gitlab runners and we were forced to pinned node to 18.18.x to avoid it. This seems to be an issue with node 10.x. Locally when I'm doing an npm ci I'm seeing some packages taking minutes to be installed (are there timeouts that keep retrying when on the command line?). There seems to be a big issue with the latest version of npm in terms of performance. NOTE: I tried my local npm ci with 10.2.5 and it takes 5+ minutes to install files. We have a local GAR based registry for some of our libraries. Could this be part of the problem?

Now that I'm trying to upgrade our systems to Node 20 I'm afraid we'll start encountering these issues again, soon. Any word from the developers on when this can be fixed?

[ghostx31]

We had this problem on using the node:18 docker image and moving to node:20 on some CIs seems to have fixed this we're also concerned with dependency issues in case we decide to move to node:20 completely.

[rosen-dimitrov]

Hi guys,

posting here as well for visibility. Looks like an issue with the new @npmcli/agent package and the way it handles network connections.

#7072 (comment)

[michalpidanic]

Hello guys, has somebody solved this issue? Im having the same problem, using node:20, my npm version is 10.2.4 and im getting the same error. I tried to do npm install -g npm@9 in before_script however, it fails with same error 🙃 I really dont know how to solve this problem and it is already been few weeks. My runner and job has latest node images which seem to be shipped out with npm 10, so little bit of deadlock here 🙃

$ npm --version
10.2.4
$ npm uninstall -g npm@10
up to date in 320ms
$ npm install -g npm@9
npm ERR! code ETIMEDOUT
npm ERR! errno ETIMEDOUT
npm ERR! network request to https://registry.npmjs.org/npm failed, reason: 
npm ERR! network This is a problem related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network settings.
npm ERR! network 
npm ERR! network If you are behind a proxy, please make sure that the
npm ERR! network 'proxy' config is set properly.  See: 'npm help config'
npm ERR! A complete log of this run can be found in: /root/.npm/_logs/[20](https://gitlab.com/...[24](https://gitlab.com/...-debug-0.log

[dahei]

We face the same issue in our Gitlab pipelines and were forced to pin our node version to 18.18.1 (which includes npm 9.8.1).
The difference also becomes obvious when running npm install locally. I tried 3 different versions on the same project.

rm -rf node_modules
nvm use ...
npm install --prefer-online

Result:
18.18.2 (npm 9.8.1) -> 38s
18.19.1 (npm 10.2.4) -> 48s
20.11.1 (npm 10.2.4) -> 2m13s (!)

In both installs with npm 10.x I could see long running requests for some packages as stated in the other comments. It's not always the same package, so it feels like a npm 10.x issue.

Within the Gitlab CI pipeline it becomes a serious blocker as the jobs take much longer and often fail.

[melroy89]

You should try to use: npm ci --cache .npm --prefer-offline. And cache the .npm directory if possible in gitlab. But I digress.

Despite all this, when using this npm ci command above with cache and prefer offline, my gitlab ci/cd is constantly failing at random on one or more jobs. It's very frustrating!

And I don't understand why npm doesn't give this issue more attention. As you notice a lot of developers are effected by this.