[BUG] tarball data currupted when installing from 'chunked encoding' server.

[everett1992]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

While using npm v7/8 to install packages from our private npm registry we'll regularly see tar extract and integrity errors

npm WARN tar TAR_ENTRY_INVALID checksum failure
npm WARN tar zlib: incorrect data check
npm WARN tarball tarball data for lodash@http://localhost:4000/lodash/-/lodash-4.17.19.tgz (sha512-JNvd8XER9GQX0v2qJgsaN/mzFCNA5BRe/j8JN9d+tWyGLSodKQHKFicdwNYzWwI3wjRnaKPsGj1XkBjx/F96DQ==) seems to be corrupted. Trying again.
npm ERR! code Z_DATA_ERROR
npm ERR! errno -3
npm ERR! zlib: incorrect data check

I can reproduce these errors using a chunked-encoding proxy in front of registry.npmjs.org. I can resolve the issue by changing make-fetch-happen's integrity check from a MinipassPipeline to a basic stream. I don't know if that really fixes the issue or just changes the timing enough to hide it.

everett1992/make-fetch-happen@4c26ad6#diff-e11a8e3d6413e0d326dca64c22417f2670168171c83d2db6da41fb7eae868aa0

Expected Behavior

npm should be able to install with any valid http server.

Steps To Reproduce

1. In this environment...
   I've reproduced this on Ubuntu with node 16, npm v7 and npm v8. I can't reproduce with v6 but one user has reported the issue with npm v6.

2. With this config...

proxy.js

# my example proxy, or any other http registry that uses chunked-encoding and small chunks.
registry=http://localhost:4000/

node proxy.js

3. Run '...'

$ mkdir example
$ cd example
$ # create package.json so node_modules will be installed here
$ echo {} > package.json
$ # remove cache. cannot reproduce with cache disabled or a cache hit.
$ rm ~/.npm/_cacache node_modules -rf
$ install any package
$ npm install react

4. See error...

npm WARN tarball tarball data for react@http://localhost:4000/react/-/react-17.0.2.tgz (sha512-gnhPt75i/dq/z3/6q/0asP78D0u592D5L1pd7M8P+dck6Fu/jJeL6iVVK23fptSUZj8Vjf++7wXA8UNclGQcbA==) seems to be corrupted. Trying again.
npm ERR! code Z_DATA_ERROR
npm ERR! errno -3
npm ERR! zlib: incorrect data check

Environment

• OS: Ubuntu
• Node: 16.11.0, 16.8.0
• npm: v8.0.0, v7.21.0

[everett1992]

I think that my proxy implementation is correct because files downloaded with curl have the correct checksum and extract cleanly.

$ npm info react dist.shasum
d0b5cc516d29eb3eee383f75b62864cfb6800037
$ curl -sS http://localhost:4000/react/-/react-17.0.2.tgz | sha1sum
d0b5cc516d29eb3eee383f75b62864cfb6800037  -

Our actual private registry is written in java, so I don't think it's a bug in common.

[everett1992]

I've traced thru pacote, minipass-fetch, and make-fetch-happen and the stream's integrity is correct thru here

https://github.com/everett1992/make-fetch-happen/blob/chunked-encoding/lib/remote.js#L53

But the stream here is wrong

https://github.com/everett1992/make-fetch-happen/blob/chunked-encoding/lib/cache/entry.js#L288

[booniepepper]

For some context, Caleb (and I) are build engineers at Amazon. This is blocking us from moving our company onto NPM 7 because of the way we handle internal repository mirrors.