[BUG] npx isn't using latest version of package

[Tobbe]

Is there an existing issue for this?

• I have searched the existing issues

This issue exists in the latest npm version

• I am using the latest npm

Current Behavior

npx is running an old version of a script

I've checked to make sure it's not installed (globally or locally)

$ npm ls
rw@ /Users/tobbe/dev/redwood/rw
├── @redwoodjs/core@0.39.2
├─┬ api@0.0.0 -> ./api
│ ├── @paypal/checkout-server-sdk@1.0.3
│ ├── @redwoodjs/api@0.39.2
│ ├── @redwoodjs/graphql-server@0.39.2
│ ├── aws-sdk@2.995.0
│ ├── node-fetch@2.6.6
│ └── pino-logflare@0.3.12
└─┬ web@0.0.0 -> ./web
  ├── @emotion-icons/boxicons-logos@2.10.0
  ├── @emotion-icons/boxicons-regular@2.10.0
  ├── @react-google-maps/api@2.4.1
  ├── @redwoodjs/auth@0.39.2
  ├── @redwoodjs/forms@0.39.2
  ├── @redwoodjs/router@0.39.2
  ├── @redwoodjs/web@0.39.2
  ├── @sentry/react@6.13.3
  ├── @sentry/tracing@6.13.3
  ├── babel-plugin-add-react-displayname@0.0.5
  ├── graphql-hooks@5.4.0
  ├── graphql-request@3.5.0
  ├── logrocket-react@4.0.1
  ├── logrocket@2.1.1
  ├── lunr@2.3.9
  ├── minisearch@3.1.0
  ├── prop-types@15.7.2
  ├── react-dom@17.0.2
  ├── react-query@3.24.4
  ├── react-responsive-carousel@3.2.21
  ├── react-slim-image-viewer@0.2.0
  ├── react@17.0.2
  ├── string-strip-html@9.0.4
  └── theme-ui@0.8.4

$ npm list -g --depth 0
/opt/homebrew/lib
└── npm@8.1.4

$ npx @redwoodjs/codemods --version
Need to install the following packages:
  @redwoodjs/codemods
Ok to proceed? (y) y
0.38.3

$ npx @redwoodjs/codemods@latest --version
0.39.2

This issue is very similar #2329 but it's closed. So either this problem has come back again, or it wasn't truly fixed in the first place.

Expected Behavior

As long as the given package is not installed, I expect npx to always use the latest version

Steps To Reproduce

It's always reproducible with the steps given above in "Current Behavior", but I don't yet know if it's reproducible with any package, or just @redwoodjs/codemods

Environment

• npm: 8.1.4
• Node: 16.12.0
• OS: MacOS 11.6
• platform: M1
• npm config:

; "builtin" config from /opt/homebrew/lib/node_modules/npm/npmrc

prefix = "/opt/homebrew"

; "user" config from /Users/tobbe/.npmrc

//registry.npmjs.org/:_authToken = (protected)

; node bin location = /opt/homebrew/Cellar/node/16.12.0/bin/node
; cwd = /Users/tobbe/dev/redwood/rw
; HOME = /Users/tobbe
; Run `npm config ls -l` to show all defaults.

[Tobbe]

If I temporarily rename ~/.npm/_npx I will get the latest version if I do just npx package-name, but if I rename it back again I get the old version again.

I don't know how the caching works exactly, but maybe the checksum (or whatever it is) is calculated wrongly sometimes or something.

$ npx @redwoodjs/codemods --version
Need to install the following packages:
  @redwoodjs/codemods
Ok to proceed? (y) y
0.38.3

$ cat _npx/f174a06391103c7e/package.json
{
  "dependencies": {
    "@redwoodjs/codemods": "^0.38.3"
  }
}

$ mv _npx __npx

$ npx @redwoodjs/codemods --version
Need to install the following packages:
  @redwoodjs/codemods
Ok to proceed? (y) y
0.39.2

$ cat _npx/f174a06391103c7e/package.json
{
  "dependencies": {
    "@redwoodjs/codemods": "^0.39.2"
  }
}

IDK, but I just found it a little weird that f174a06391103c7e is used for both versions

[cuarti]

I had the same issue with a cli that I'm developing.

When I called npx, it used an old version instead of the new one until I deleted the npx cache (~/.npm/_npx). After cache deletion I tried first to use an old version to download it and after the new one but now it is working as expected. So try to remove the npx cache.

Actually I'm developing an e2e test using a private registry to test that cli and it is not working as expected. I publish the package without updating the version to the private registry (without any package and without proxy this package to the official registry), then when I call npx with the private registry it uses the version of the official registry because it is in the cache.

I fixed this issue removing the npx cache before the e2e test.

The bug that I see (similar to this opened issue) is that npx cache don't have the registry used, so when you want to use a package from one registry that is cached from another it gets the cached version from another registry instead of the version of the desired registry.

[ruyadorno]

Hi @Tobbe, thanks for taking the time to report this!

From my understanding of the internals and the documentation npm exec or npx are not guaranteed to always fetch the latest version of a package. If you want that to be the behavior I believe the best thing is for you to always run the commands with a @latest suffix attached to the package name, that should guarantee you have the latest version available.

I don't know how the caching works exactly, but maybe the checksum (or whatever it is) is calculated wrongly sometimes or something.

Here's the relevant bit in case you're interested into digging more 😊

cli/workspaces/libnpmexec/lib/cache-install-dir.js

Lines 14 to 18 in 76afe15

	const getHash = (packages) =>
	crypto.createHash('sha512')
	.update(packages.sort((a, b) => a.localeCompare(b, 'en')).join('\n'))
	.digest('hex')
	.slice(0, 16)

With that in mind, since this is working as intended I'm going to close this issue. But feel free to open a new ones in case you find any other bug. In case you would like to propose a different behavior to npx, feel free to open an issue or discussion in our RFC repo with some details on how you could/would like to see this changed?

Thanks again!

[geekagan]

I had the same problem, when I was at [@v1.0.1]; exec npx <package> , Later, when upgraded to v1.0.2, I exec npx <package>[@latest] , then i exec npx <package> once again; the application is not the latest version of the package, still use [@v1.0.1];
my node version = v18.12.1
my npm version = 9.6.2

[geekagan]

I had the same problem, when I was at [@v1.0.1]; exec npx <package> , Later, when upgraded to v1.0.2, I exec npx <package>[@latest] , then i exec npx <package> once again; the application is not the latest version of the package, still use [@v1.0.1]; my node version = v18.12.1 my npm version = 9.6.2

i think this use way, maybe a suitable way, we can choose version to use , eg. npx <package>[@version];
but, i prefer the way is when npx <package> once again, can use cach latest version;

[SimonEast]

@ruyadorno and other maintainers:

If it is indeed how npx is designed to work — in that npx will always run a cached version of a package if it exists, regardless of age, unless you specify npx some-package@latest — then could this be very clearly noted in the documentation?

I doubt that this behaviour would be obvious to most users, and is likely to create a whole range of bugs from people who run an npx command (as recommended by many popular libraries), and are unwittingly using an outdated version.

There is currently nothing mentioned about this caching behaviour on the npx documentation page. I had to find this out from random StackOverflow users and go searching until I ended up in this thread.

If npx documentation requests belong in a another repo, please point me in the right direction.

[jahooma]

This behavior of not necessarily using the latest version also makes npx not useful for my application.