Description
What / Why
When invoking npm ci, the command fails with premature close and little else.
When
- invoking
npm ciand a git dependency with a prepare script is included.
Where
- Our internal Jenkins servers.
- Internal packages are consumed over git. Public packages are fine.
How
Current Behavior
When invoking npm ci, our org sees this error:
[2020-04-15T18:53:51.235Z] + npm ci
[2020-04-15T18:53:56.677Z] npm ERR! premature close
[2020-04-15T18:54:04.979Z]
[2020-04-15T18:54:04.979Z] npm ERR! A complete log of this run can be found in:
[2020-04-15T18:54:04.979Z] npm ERR! /home/jenkins/.npm/_logs/2020-04-15T18_53_56_371Z-debug.log
Steps to Reproduce
This is sticky, but here's my best shot:
- Have a repository that consumes another via
git. - The consumed repository has a
preparescript. Doesn't matter what's in it. - Have a sufficient number of CA certificates? I'm less confident here.
- Run
npm ci. - Observe. Weep.
Expected Behavior
npm ci should succeed, given the prepare script and all of other packages are healthy.
Who
My workplace, but I am happy to be the primary contact.
References
- Perhaps coincidentally related to fd0a802
Other notes
I spent some time on this and it's possible more than one bug will fall out of it. The logging here doesn't seem useful (a bug of its own), but fixing the logging won't get down to the root cause. I intend to track the root cause in this ticket, and may file another for the logging issue.
When we dial up the logging to silly, we more or less see the same thing. We did prune our dependencies until we found the foremost ones involved. They are internal git based dependencies. I can furnish the verbose logs if desired, but I will omit them from the initial report since I believe they will only add noise. The last log seen from the problematic packages is prepareGitDep.
prepareGitDep <repo-name>@git+ssh://git@<repo-url>.git#<commit-hash>: installing devDeps and running prepare script.
There are a lot of other logs between this and other things - I assume this is due to the asynchronous manner of package installation.
The git dependency in question has a prepare script declared in its package.json. If I remove prepare, everything works. If I replace the prepare script with something rudimentary such as "echo 'hi'", "true", or even "" I still get the error above.
This may be noise: We noticed this in 6.14.4 from 6.13.4 - when our Node version transitioned from 12.16.2 to 12.16.3. I was able to walk commits and found the problem reliably appeared at this commit. All it does is points cacache to the _cacache subdirectory, whereas beforehand there was no subdirectory. Later we were able to reproduce the error in 6.13.4, so I think the _cacache change might exacerbate the issue, but is not the direct cause.
I made some direct edits to my local copy of npm and found that if, in pack.js, I change cliArgs to just be [] that everything Just Works(tm). I don't believe this is a sustainable fix though. Further debugged revealed the issue to be E2BIG - an error that a single command line argument exceeded the kernel's restriction. Ours is set to 2097152, which I understand to be the Linux default - and is restricted by the kernel's page size. You can view your own with getconf ARG_MAX at your terminal. I inspected the cliArgs and found that ca was set to the cert data of every certificate on the machine. I perused these and found a standard assortment of CA certs, intermediate certificates, etc. Our organization's internal certificates were included. I assume but have not yet verified that this is simply reading from the local machine's OpenSSL's certificate list - I know that we don't set it explicitly. .npmrc has only cafile=/etc/pki/tls/certs/ca-bundle.crt in relation to certificates. The build machine where this behavior is observed is running CentOS 7.7.1908. Most of us use Macs at the workplace, and we do not see this error on them with the same node/npm versions.
Since this restriction applies to a single argument, I chopped up ca into multiple ca[] arguments. This seems to work for pack.js but it kicks the can down the road. I suspect that other parts of npm need a similar treatment, but the processes have either become too opaque for me to trace much further, or I my simian mind is too simple and unfamiliar with npm innards.