[BUG] npx isn't using latest version of package

[Tobbe]

Is there an existing issue for this?

• I have searched the existing issues

This issue exists in the latest npm version

• I am using the latest npm

Current Behavior

npx is running an old version of a script

I've checked to make sure it's not installed (globally or locally)

$ npm ls
rw@ /Users/tobbe/dev/redwood/rw
├── @redwoodjs/core@0.39.2
├─┬ api@0.0.0 -> ./api
│ ├── @paypal/checkout-server-sdk@1.0.3
│ ├── @redwoodjs/api@0.39.2
│ ├── @redwoodjs/graphql-server@0.39.2
│ ├── aws-sdk@2.995.0
│ ├── node-fetch@2.6.6
│ └── pino-logflare@0.3.12
└─┬ web@0.0.0 -> ./web
  ├── @emotion-icons/boxicons-logos@2.10.0
  ├── @emotion-icons/boxicons-regular@2.10.0
  ├── @react-google-maps/api@2.4.1
  ├── @redwoodjs/auth@0.39.2
  ├── @redwoodjs/forms@0.39.2
  ├── @redwoodjs/router@0.39.2
  ├── @redwoodjs/web@0.39.2
  ├── @sentry/react@6.13.3
  ├── @sentry/tracing@6.13.3
  ├── babel-plugin-add-react-displayname@0.0.5
  ├── graphql-hooks@5.4.0
  ├── graphql-request@3.5.0
  ├── logrocket-react@4.0.1
  ├── logrocket@2.1.1
  ├── lunr@2.3.9
  ├── minisearch@3.1.0
  ├── prop-types@15.7.2
  ├── react-dom@17.0.2
  ├── react-query@3.24.4
  ├── react-responsive-carousel@3.2.21
  ├── react-slim-image-viewer@0.2.0
  ├── react@17.0.2
  ├── string-strip-html@9.0.4
  └── theme-ui@0.8.4

$ npm list -g --depth 0
/opt/homebrew/lib
└── npm@8.1.4

$ npx @redwoodjs/codemods --version
Need to install the following packages:
  @redwoodjs/codemods
Ok to proceed? (y) y
0.38.3

$ npx @redwoodjs/codemods@latest --version
0.39.2

This issue is very similar #2329 but it's closed. So either this problem has come back again, or it wasn't truly fixed in the first place.

Expected Behavior

As long as the given package is not installed, I expect npx to always use the latest version

Steps To Reproduce

It's always reproducible with the steps given above in "Current Behavior", but I don't yet know if it's reproducible with any package, or just @redwoodjs/codemods

Environment

• npm: 8.1.4
• Node: 16.12.0
• OS: MacOS 11.6
• platform: M1
• npm config:

; "builtin" config from /opt/homebrew/lib/node_modules/npm/npmrc

prefix = "/opt/homebrew"

; "user" config from /Users/tobbe/.npmrc

//registry.npmjs.org/:_authToken = (protected)

; node bin location = /opt/homebrew/Cellar/node/16.12.0/bin/node
; cwd = /Users/tobbe/dev/redwood/rw
; HOME = /Users/tobbe
; Run `npm config ls -l` to show all defaults.

[Tobbe]

If I temporarily rename ~/.npm/_npx I will get the latest version if I do just npx package-name, but if I rename it back again I get the old version again.

I don't know how the caching works exactly, but maybe the checksum (or whatever it is) is calculated wrongly sometimes or something.

$ npx @redwoodjs/codemods --version
Need to install the following packages:
  @redwoodjs/codemods
Ok to proceed? (y) y
0.38.3

$ cat _npx/f174a06391103c7e/package.json
{
  "dependencies": {
    "@redwoodjs/codemods": "^0.38.3"
  }
}

$ mv _npx __npx

$ npx @redwoodjs/codemods --version
Need to install the following packages:
  @redwoodjs/codemods
Ok to proceed? (y) y
0.39.2

$ cat _npx/f174a06391103c7e/package.json
{
  "dependencies": {
    "@redwoodjs/codemods": "^0.39.2"
  }
}

IDK, but I just found it a little weird that f174a06391103c7e is used for both versions

[cuarti]

I had the same issue with a cli that I'm developing.

When I called npx, it used an old version instead of the new one until I deleted the npx cache (~/.npm/_npx). After cache deletion I tried first to use an old version to download it and after the new one but now it is working as expected. So try to remove the npx cache.

Actually I'm developing an e2e test using a private registry to test that cli and it is not working as expected. I publish the package without updating the version to the private registry (without any package and without proxy this package to the official registry), then when I call npx with the private registry it uses the version of the official registry because it is in the cache.

I fixed this issue removing the npx cache before the e2e test.

The bug that I see (similar to this opened issue) is that npx cache don't have the registry used, so when you want to use a package from one registry that is cached from another it gets the cached version from another registry instead of the version of the desired registry.

[ruyadorno]

Hi @Tobbe, thanks for taking the time to report this!

From my understanding of the internals and the documentation npm exec or npx are not guaranteed to always fetch the latest version of a package. If you want that to be the behavior I believe the best thing is for you to always run the commands with a @latest suffix attached to the package name, that should guarantee you have the latest version available.

I don't know how the caching works exactly, but maybe the checksum (or whatever it is) is calculated wrongly sometimes or something.

Here's the relevant bit in case you're interested into digging more 😊

cli/workspaces/libnpmexec/lib/cache-install-dir.js

Lines 14 to 18 in 76afe15

	const getHash = (packages) =>
	crypto.createHash('sha512')
	.update(packages.sort((a, b) => a.localeCompare(b, 'en')).join('\n'))
	.digest('hex')
	.slice(0, 16)

With that in mind, since this is working as intended I'm going to close this issue. But feel free to open a new ones in case you find any other bug. In case you would like to propose a different behavior to npx, feel free to open an issue or discussion in our RFC repo with some details on how you could/would like to see this changed?

Thanks again!