[BUG] Failed to install npm package from git in docker since v6.11.0

[doochik]

What / Why

This bug is related to #476 (comment) and #514

When

npm install packages from git in docker

Where

• packages from public git repositories

How

Current Behavior

# npm i
npm ERR! code 128
npm ERR! Command failed: git clone --depth=1 -q -b v2.0.5 git://github.com/boblauer/MockDate.git /root/.npm/_cacache/tmp/git-clone-c9699b7f
npm ERR! fatal: could not create leading directories of '/root/.npm/_cacache/tmp/git-clone-c9699b7f': Permission denied
npm ERR! 

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2019-12-23T09_49_05_990Z-debug.log

Steps to Reproduce

• Create simple package.json with git dependency

{
  "name": "test-npm",
  "dependencies": {
    "mockdate": "git://github.com/boblauer/MockDate.git#v2.0.5"
  }
}

• run docker run --network=host -it -v $(pwd):/var/work/test -w /var/work/test node:12.14.0 /bin/bash
• ls -l shows that files are owned by unknown user and unknown group (this is typical behaviour on build agents like teamcity, teamcity clones git repo and mounts it inside docker container)

# ls -l
-rw-r--r-- 1 20857 115755  114 Dec 23 09:46 package.json

• run npm install

# Depends on environment you can get

npm ERR! code 128
npm ERR! Command failed: git clone --depth=1 -q -b v2.0.5 git://github.com/boblauer/MockDate.git /root/.npm/_cacache/tmp/git-clone-c9699b7f
npm ERR! fatal: could not create leading directories of '/root/.npm/_cacache/tmp/git-clone-c9699b7f': Permission denied
npm ERR! 

# or

npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t ssh://git@<private server>/admin-template.git
npm ERR! 
npm ERR! No user exists for uid 1000
npm ERR! fatal: Could not read from remote repository.

• run npm i -g npm@6.10 (downgrade to npm 6.10.X)
• run npm install

# all works well!

updated 1 package and audited 1 package in 1.355s
found 0 vulnerabilities

Expected Behavior

• npm install / npm ci works well like npm@6.10

[SabatierBoris]

I have the same issue with npm 6.13.4.
It's working with npm 6.4.1

[doochik]

@isaacs please, take a look on this

[isaacs]

Is the npm command being run as root, or some other user?

I'd like to try to zoom in on the problem and leave the docker aspect out of it.

If I understand this correctly:

• The user running npm install is root
• The $HOME directory is /root or some other root owned directory
• Thus the cache folder in $HOME/.npm is root-owned.
• But pacote is executing a git command based on the user of the ultimate target of the extraction, not the cwd where the git action is taking place, and it fails.

Simpler reproduction, sans docker:

$ sudo su
# echo $HOME
/var/root
# npm i git://github.com/isaacs/abbrev-js
npm ERR! code 128
npm ERR! Command failed: git clone --mirror -q git://github.com/isaacs/abbrev-js.git /var/root/.npm/_cacache/tmp/git-clone-35b93572/.git
npm ERR! fatal: could not create leading directories of '/var/root/.npm/_cacache/tmp/git-clone-35b93572/.git'
npm ERR!

npm ERR! A complete log of this run can be found in:
npm ERR!     /var/root/.npm/_logs/2019-12-30T20_15_05_627Z-debug.log

I will take a look today and see how hard this'll be to fix.

[isaacs]

Note for myself and @npm/cli-team: this problem does not occur with pacote v10, so it's strictly a pacote v9 issue.

To verify this, I applied this diff to pacote:

diff --git a/lib/git.js b/lib/git.js
index ade670b..a79a0e1 100644
--- a/lib/git.js
+++ b/lib/git.js
@@ -36,6 +36,7 @@ class GitFetcher extends Fetcher {
   constructor (spec, opts) {
     super(spec, opts)
     this.resolvedRef = null
+    this.spec.hosted = null
     if (this.spec.hosted)
       this.from = this.spec.hosted.shortcut({ noCommittish: false })
 

(This is required because otherwise pacote v10 will try to fetch github repos from their static tarball endpoint, as that's usually much faster.)

Then ran:

# pacote extract git+ssh://git@github.com/isaacs/abbrev-js abbrev-from-git
{
  resolved: 'git+ssh://git@github.com/isaacs/abbrev-js#b8f3a2fc0c3bb8ffd8b0d0072cc6b5a3667e963c',
  integrity: 'sha512-NFzZKD4tPyRbi5lxIbIhczbuAaIlTCMCwR9GGG9dqptIOM5G7e9w80NOInG5Ny6yUS/QcCp0XHxVPOKrItUUIg==',
  from: 'git+ssh://git@github.com/isaacs/abbrev-js.git'
}
# ls -laF abbrev-from-git/
total 32
drwxr-xr-x    6 isaacs  staff   192 Dec 30 12:22 ./
drwxr-xr-x  133 isaacs  staff  4256 Dec 30 12:21 ../
-rw-r--r--    1 isaacs  staff  2011 Oct 26  1985 LICENSE
-rw-r--r--    1 isaacs  staff   499 Oct 26  1985 README.md
-rw-r--r--    1 isaacs  staff  1763 Oct 26  1985 abbrev.js
-rw-r--r--    1 isaacs  staff   529 Oct 26  1985 package.json

[isaacs]

Looks like npm/pacote#26 will fix this problem, assuming that I am understanding the root cause. (At the very least, it fixes a problem, which is that running npm install of a git dep as a root user with a root-owned npm cache will not fail, and it seems like that's the thing happening here.)

[doochik]

Yes, npm install is being run as root.

Thanks for the fix!