feat!: use npm for plugin operations #776

mdonnalley · 2024-01-23T17:41:14Z

Breaking Changes

use npm for all plugin operations.
tsc is no longer called on typescript ESM plugins installed from github. If you need that functionality, you should update your plugin's prepare script to build the project on install
Removed export of Plugins
Users relying on a .yarnrc configuration file will have to migrate to .npmrc

Other Changes

Renamed oclif/plugins debug scope to @oclif/plugin-plugins
Rename cli:yarn debug scope to @oclif/plugin-plugins:npm
DEBUG logs always contain verbose npm output, --verbose flag will send output to stdout. Fixes Provide more info about installation failure #763

Issues/Work Items

@W-14581559@
@W-14792145@

BREAKING CHANGES: use npm for all plugin operations

git2gus · 2024-01-23T17:45:10Z

This issue has been linked to a new work item: W-14876941

iowillhoit · 2024-03-18T20:42:35Z

src/commands/plugins/install.ts

-
-      YarnMessagesCache.getInstance().flush(plugin)
-
-      this.log(chalk.green(`\nSuccessfully installed ${plugin.name} v${plugin.version}`))


Did you mean to remove this log message?

Yes. The spinner now shows the success (or failure) message

@salesforce/cli: Installing plugin @oclif/plugin-version@latest... installed v2.0.14

iowillhoit · 2024-03-21T15:03:09Z

QA NOTES

Note: The qa CLI has the beta version of plugin-plugins

MISC

🟢 Installs and updates are significantly faster and bit smaller with npm. This is largely due to not needed a "refresh" with yarn to lock dependencies. The more plugins you have installed, the larger these differences will be.
- Using @salesforce/plugin-source @salesforce/plugin-org @salesforce/plugin-user, I am comparing ~/.local/share/sf/node_modules
  - yarn
    - First run took 41s
    - Second run took 1m7s
    - Size is 554Mb
  - npm
    - First run took 25s
    - Second run took 19s
    - Size is 411Mb
- @jshackell-sfdc this is worth calling out in RNs
🟢 When a npm-shrinkwrap is present, plugin dependencies seem to be much more predictable with npm. For example, running sf plugins install @oclif/plugin-help @oclif/plugin-which @oclif/plugin-version with the qa release of sf; you get nested node_modules directories for each plugin that contain the dependencies defined in their shrinkwrap file. This should prevent some of the odd hoisting issues we have seen in the past with yarn add and the yarn install refreshes.
🟢 Expanding on this ^, going from plugins installed with yarn to npm properly cleans up the top level node_modules directory and you get a nice set of nested deps 🎉
🟢 Was able to create a prerelease of the sf CLI and all NUTs passed.
🔴 The --silent flag is now true by default. There is no way to get non-silent output.
- --silent=false does not work on boolean flags.
- allowNo is not enabled
- --verbose is not an equivalent of "non-silent"
🟢 The --verbose flag works.

RELATED ISSUES

🔴 (yarn and npm) Running sf plugins reset --reinstall does not work with github url installs.
- Run sf plugins install https://github.com/oclif/plugin-version.git
  - Same behavior when using a branch or version (#ew/testing-plugin-plugins-npm-branch-installs)
- Confirm version: cat ~/.local/share/sf/package.json | jq .dependencies
- Run sf plugins reset --reinstall
- Confirm version: cat ~/.local/share/sf/package.json | jq .dependencies
- Notice that it is no longer a github url, but a npm package name
  - The reinstall message does show the correct url, despite installing from npm package: Reinstalled @oclif/plugin-version (https://github.com/oclif/plugin-version.git)
🟡 (yarn and npm) You can install from github url using a lot of different short-names, but you cannot use the same github shortnames to uninstall. For example: sf plugins uninstall oclif/plugin-version does not work. You have to uninstall with the npm package name (@oclif/plugin-version)
🟡 (yarn and npm) When a core plugin is manually installed (like plugins-version), running sf plugins --core does not show (user), it just removes (core). Would it be more clear if we added (user) when core plugins were "overwritten"
🟡 (yarn and npm) Unable to allow-list github url installs with unsignedPluginAllowList.json
🟡 (yarn and npm) When you install a specific version of a non-JIT plugin (like @salesforce/plugin-org@3.4.5, sfdx-hardis@4.28.0, or https://github.com/oclif/plugin-version.git#2.0.11), running sf plugins update it does not update the plugin to a newer version. This does work for JIT plugins. Maybe this is WAD because the user intentionally installed a different version of a non-JIT plugin, but at the very least a warning message would be nice. "Did not update plugin @salesforce/plugin-org because it was manually installed.
- For core plugins you could suggest: "Run sf plugins uninstall @salesforce/plugin-org to revert to bundled version."
- For 3PP plugins you'd need to suggest installing it again without a branch or version: "Run sfdx plugins install sfdx-hardis to install the latest version."
- Should we make core plugins work the same way as JIT? If the version of the core plugin is "behind", update it to match what is in the CLI's package.json. If it is ahead, show a warning.
🔴 (yarn and npm) Odd uninstall behavior removes the wrong plugin, assuming this is a shortname bug
- Run sf plugins install @salesforce/plugin-org
- Run sf plugins --core
  - Note: org does not have (core) next to it
- Run sf plugins uninstall @oclif/plugin-org
  - Notice the @oclif/, not @salesforce ^
- Run sf plugins --core
  - Note: @salesforce/plugin-org was uninstalled. org has (core) next to it

JIT PLUGINS

🟢 JIT plugins can be installed manually with npm
🟢 JIT plugins install when running the command for the first time with npm
🟢 JIT plugin installed with yarn can be updated with npm
🟢 Can reinstall a JIT plugin with plugins reset --reinstall with npm

CORE NPM PACKAGE

🟢 Able to install @salesforce/plugin-org with npm
- 🟢 Able to update plugin with npm
🟢 Able to install @salesforce/plugin-org with yarn
- 🟢 Able to update plugin with npm
🟢 Able to install @salesforce/plugin-org with yarn
- 🟢 Able to uninstall @salesforce/plugin-org with npm
- 🟢 sf plgugins --core shows org as (core)
🟢 Able to install @salesforce/plugin-org with yarn
- 🟢 Running sf plugins reset --reinstall reinstalls the correct version

CORE NPM PACKAGE WITH VERSION

🟢 Able to install @salesforce/plugin-org@3.4.5 with npm
- 🟢 Able to update plugin with npm
🟢 Able to install @salesforce/plugin-org@3.4.5 with yarn
- 🟢 Able to update plugin with npm
🟢 Able to install @salesforce/plugin-org@3.4.5 with yarn
- 🟢 Able to uninstall @salesforce/plugin-org@3.4.5 with npm
- 🟢 sf plgugins --core shows org as (core)
🟢 Able to install @salesforce/plugin-org@3.4.5 with yarn
- 🟢 Running sf plugins reset --reinstall reinstalls the correct version

3PP PLUGIN NPM PACKAGE

🟢 Able to install sfdx-hardis with npm
- 🟢 Able to update plugin with npm
🟢 Able to install sfdx-hardis with yarn
- 🟢 Able to update plugin with npm
🟢 Able to install sfdx-hardis with yarn
- 🟢 Able to uninstall sfdx-hardis with npm
🟢 Able to install sfdx-hardis with yarn
- 🟢 Running sf plugins reset --reinstall works with npm

3PP PLUGIN NPM PACKAGE WITH VERSION

🟢 Able to install sfdx-hardis@4.28.0 with npm
- - 🟡 sf plugins update with npm installs same version of plugin (see Related Issue above)
🟢 Able to install sfdx-hardis@4.28.0 with yarn
- - 🟡 sf plugins update with npm installs same version of plugin (see Related Issue above)
🟢 Able to install sfdx-hardis@4.28.0 with yarn
- 🟢 Able to uninstall sfdx-hardis with npm
🟢 Able to install sfdx-hardis@4.28.0 with yarn
- 🟢 Running sf plugins reset --reinstall installs the correct version with npm

SHORTNAME PLUGINS

🔴 Shortname npm-package plugin installs not working on the first install.
- Delete all plugins
- With the npm version run sf plugins install org
- You'll get This plugin isn't signed by Salesforce. Only install the plugin if you trust its creator. Do you want to continue the installation?. By adding a log to plugin-trust you can see that it is not correctly resolving @salesforce/plugin-org. It is just looking up org on npm.
  - 🟠 Related: We need to update the plugin-trust message to include the name of the plugin (instead of This plugin). This would also be nice when installing from github using a short name. For example if someone ran sf plugins oclif/plugin-version (forgot the @), it would be nice to know that this was installing from github and not npm. "Trying to install from https://github.com/oclif/plugin-version. [...]"
    - 🟡 You could also mention that signed plugin validation does not work on github urls: " Plugins installed from Github cannot be verified. Proceed? (y/N)"
  - ❓❓ What if we did away with shortnames in this major version? I have always found them convoluted. I think users should be explicit when installing code on their machines.
- It is worth noting that if you already have org installed via yarn, the package.json in the ~/.local/share/sf dir will have @salesforce/plugin-org as a dependency. If you then try to install org using the npm version, it will correctly install and look up the signature for @salesforce/plugin-org
- 🟡 Need to test shortname with a version (org@3.4.0)
🟢 Can uninstall using a shortname (org)
🟢 Able to install a shortname plugin with yarn
- 🟢 Able to update a shortname plugin with npm
🟢 sf plugins reset --reinstall works with a shortname plugin

GITHUB ORG/REPO Installs

🟢 Able to install oclif/plugin-version with npm
- 🟢 Able to update plugins (oclif/plugin-version) with npm
🟢 Able to install oclif/plugin-version with yarn
- 🟢 Able to update plugins (oclif/plugin-version) with npm
🟢 Able to install oclif/plugin-version with yarn
- 🟢 Able to uninstall @oclif/plugin-version with npm (note: this requires npm package name)
🟠 We should add a warning about the breaking change "tsc is no longer called on typescript ESM plugins installed from github." With something like salesforcecli/plugin-org (that does not have a compile as part of its prepare script) it appears to install correctly, but then none of the commands are recognized because the lib dir is not there.
🟢 Bonus fix! Github url installs with yarn could get confused and have duplicate oclif entries in the ~/.local/share/sf/package.json. This does not appear to be an issue with npm.
- Reproduce with yarn:
  - sf plugins install oclif/plugin-version
  - sf plugins install https://githbu.com/oclif/plugin-version
  - cat ~/.local/share/sf/package.json
- 🟢 NOTE: If users are in this bad state with yarn installs (multiple entries), running a sf plugins update with npm fixes the package.json

GITHUB FULL URL

🟢 Able to install https://github.com/oclif/plugin-version with npm
- 🟢 Able to update plugin with npm
🟢 Able to install https://github.com/oclif/plugin-version with yarn
- 🟢 Able to update plugin with npm
🟢 Able to install https://github.com/oclif/plugin-version with yarn
- 🟢 Able to uninstall @oclif/plugin-version with npm (note: this requires npm package name)
🟡 sf plugins reset --reinstall does not work with github urls (see Related Issue above)

GITHUB FULL URL (with .git)

🟢 Able to install https://github.com/oclif/plugin-version.git with npm
- 🟢 Able to update plugin with npm
🟢 Able to install https://github.com/oclif/plugin-version.git with yarn
- 🟢 Able to update plugin with npm
🟢 Able to install https://github.com/oclif/plugin-version.git with yarn
- 🟢 Able to uninstall @oclif/plugin-version with npm (note: this requires npm package name)
🟡 sf plugins reset --reinstall does not work with github urls (see Related Issue above)

GITHUB URL WITH BRANCH

🟢 Able to install https://github.com/oclif/plugin-version#ew/testing-plugin-plugins-npm-branch-installs with npm
- 🟢 Able to update plugin with npm
🟢 Able to install https://github.com/oclif/plugin-version#ew/testing-plugin-plugins-npm-branch-installs with yarn
- 🟢 Able to update plugin with npm
🟢 Able to install https://github.com/oclif/plugin-version#ew/testing-plugin-plugins-npm-branch-installs with yarn
- 🟢 Able to uninstall @oclif/plugin-version with npm (note: this requires npm package name)
🟡 Once the plugin reset --reinstall issue is fixed, confirm that it reinstalls the same branch

GITHUB URL WITH BRANCH (with .git)

🟢 Able to install https://github.com/oclif/plugin-version.git#ew/testing-plugin-plugins-npm-branch-installs with npm
- 🟢 Able to update plugin with npm
🟢 Able to install https://github.com/oclif/plugin-version.git#ew/testing-plugin-plugins-npm-branch-installs with yarn
- 🟢 Able to update plugin with npm
🟢 Able to install https://github.com/oclif/plugin-version.git#ew/testing-plugin-plugins-npm-branch-installs with yarn
- 🟢 Able to uninstall @oclif/plugin-version with npm (note: this requires npm package name)
🟡 Once the plugin reset --reinstall issue is fixed, confirm that it reinstalls the same branch

GITHUB URL WITH VERSION

🟢 Able to install https://github.com/oclif/plugin-version.git#2.0.11 with npm
- 🟡 sf plugins update with npm installs same version of plugin (see Related Issue above)
🟢 Able to install https://github.com/oclif/plugin-version.git#2.0.11 with yarn
- 🟡 sf plugins update with npm installs same version of plugin (see Related Issue above)
🟢 Able to install https://github.com/oclif/plugin-version.git#2.0.11 with yarn
- 🟢 Able to uninstall @oclif/plugin-version with npm (note: this requires npm package name)
🟡 Once the plugin reset --reinstall issue is fixed, confirm that it reinstalls the same version

LINKING PLUGINS

🟢 Able to link plugin with . with npm
🟢 Able to unlink with sf plugins unlink (in plugin dir)
- 🟡 The output could maybe be improved here: @salesforce/cli: Uninstalling .... done. The first . is the current dir that you are unlinking
🟢 Able to link plugin with full path ~/dev/plugin-org with npm
🟢 Able to uninstall plugin with shortname (org) with npm
🟢 Both uninstall and unlink remove a linked plugin with npm
🟢 Able to link a plugin with npm
- 🟢 Able to update a linked plugin with npm
🟢 Able to link with yarn
- 🟢 Able to update a linked plugin with npm
🟢 sf plugins reset --reinstall works with a linked plugin

mdonnalley · 2024-03-21T21:05:36Z

🔴 The --silent flag is now true by default. There is no way to get non-silent output.

Fixed

🔴 (yarn and npm) Running sf plugins reset --reinstall does not work with github url installs.

Fixed

🔴 (yarn and npm) Odd uninstall behavior removes the wrong plugin, assuming this is a shortname bug

Fixed

🔴 Shortname npm-package plugin installs not working on the first install

Can't repro

🟠 We should add a warning about the breaking change "tsc is no longer called on typescript ESM...

Done.

❯ echo y | sf plugins install salesforcecli/plugin-org
 ›   Warning: @oclif/plugin-plugins is a linked ESM module and cannot be auto-transpiled. Existing compiled source will be used instead.
? This plugin isn't signed by Salesforce. Only install the plugin if you trust its creator. Do you want to continue the installation?, yes
Because you approved this plugin, you can avoid future installation confirmations by adding the plugin to the unsignedPluginAllowList.json file. For details, see https://developer.salesforce.com/docs/atlas.en-us.sfdx_setup.meta/sfdx_setup/sfdx_setup_allowlist.htm.

up to date in 7s
 ›   Warning: This plugin from github may not work as expected because the prepare script did not produce all the expected files.
@salesforce/cli: Installing plugin https://github.com/salesforcecli/plugin-org... installed v3.5.0

iowillhoit · 2024-03-22T19:43:31Z

🔴 Shortname npm-package plugin installs not working on the first install

Can't repro

Huh, odd. This only seems to happens if the ~/.local/share/sf dir does not exist... If the sf dir exists, even if it is empty, this works as expected.

iowillhoit · 2024-03-22T20:18:40Z

QA UPDATE

🔴 The --silent flag is now true by default. There is no way to get non-silent output.
Fixed

🟢 Looks good! Non-silent output shows dep warnings. Using --silent works as expected

🔴 (yarn and npm) Running sf plugins reset --reinstall does not work with github url installs.
Fixed

🟢 Confirmed.
🟢 Works with and without the --hard flag
🟢 Can install via url with yarn, then plugin reset --reinstall with npm

🔴 (yarn and npm) Odd uninstall behavior removes the wrong plugin, assuming this is a shortname bug
Fixed

🟢 No longer uninstalling the wrong plugin

🔴 Shortname npm-package plugin installs not working on the first install
Can't repro

🤷 See comment above

🟠 We should add a warning about the breaking change "tsc is no longer called on typescript ESM...
Done.

🟢 Confirmed, looks good.

iowillhoit · 2024-03-22T20:54:48Z

QA UPDATE

🟡 Need to test shortname with a version (org@3.4.0)

🟢 Works as expected with npm

🟡 sf plugins reset --reinstall does not work with github urls (see Related Issue above)

GITHUB FULL URL
- 🟢 sf plugins install https://github.com/oclif/plugin-version with yarn
- 🟢 sf plugins reset --reinstall with npm
GITHUB FULL URL (with .git)
- 🟢 sf plugins install https://github.com/oclif/plugin-version.git with yarn
- 🟢 sf plugins reset --reinstall with npm
GITHUB URL WITH BRANCH
- 🟢 sf plugins install https://github.com/oclif/plugin-version#ew/testing-plugin-plugins-npm-branch-installs with yarn
- 🟢 sf plugins reset --reinstall with npm
GITHUB URL WITH BRANCH (with .git)
- 🟢 sf plugins install https://github.com/oclif/plugin-version.git#ew/testing-plugin-plugins-npm-branch-installs with yarn
- 🟢 sf plugins reset --reinstall with npm
GITHUB URL WITH VERSION
- 🟢 sf plugins install https://github.com/oclif/plugin-version.git#2.0.11 with yarn
- 🟢 sf plugins reset --reinstall with npm

iowillhoit · 2024-03-25T16:03:39Z

🟢 Able to install with shortname when the ~/.local/share/sf dir does not exist

iowillhoit · 2024-03-25T19:43:25Z

QA UPDATE

🟢 Able to install with a custom registry
- Using verdaccio and SF_NPM_REGISTRY=http://localhost:4873/
🟢 Published a new version, it was installed with sf plugins update
🟢 Able to install with a custom registry with yarn
- 🟢 Published a new version, it was installed with sf plugins update with npm
🟢 Disconnected from verdaccio and it fell back to npm's registry version of plugin-org

mdonnalley added 5 commits January 18, 2024 09:05

feat: add npm and pnpm options

ba6a18c

Merge branch 'main' into mdonnalley/revamp

ea2163f

feat!: use npm for installs

94f0668

BREAKING CHANGES: use npm for all plugin operations

chore: bump to v5 prerelease

1e80a62

Merge branch 'main' into mdonnalley/revamp

3953b9c

mdonnalley added the enhancement label Jan 23, 2024

mdonnalley added 2 commits January 24, 2024 13:49

fix: remove yarn.lock

737b967

test: skip sf integration tests

d87fcbf

This was referenced Jan 24, 2024

feat: reset --reinstall --hard #777

Merged

feat: deprecate and hide lock command oclif/oclif#1264

Merged

mdonnalley added 19 commits January 25, 2024 09:07

fix: force rm yarn.lock

6c3cb6f

fix: use fork for npm show

6ebb1fc

feat: remove all yarn functionality

b155136

chore: clean up

d7c09cb

test: debug failing windows tests

b1ea1f1

feat: add --npm-log-level flag

3cc66c8

chore: clean up

fd4f30f

test: debug failing windows tests

85b2da0

test: compilation errors

002675b

fix: use CLIError

b2a4d14

test: debug failing windows tests

9b736f6

test: debug failing windows tests

503a9d3

test: debug failing windows tests

6d23bb9

chore: remove unused dep

0e6c372

chore: clean up

daac083

fix: add suggestion for failed install

7e3f0a8

fix: npm-run-path is actually needed

dd8f69c

test: debug failing windows tests

b4da5fd

test: debug failing windows tests

409db00

iowillhoit reviewed Mar 18, 2024

View reviewed changes

mdonnalley removed the enhancement label Mar 20, 2024

mdonnalley added 5 commits March 21, 2024 10:21

fix: reinstall plugin from url if applicable

a127059

feat: better logging

d9a67ae

fix: uninstall mis-scoped plugin

7167090

fix: warn about missing expected files after github install

6fdb181

fix: add type

c07eccf

mdonnalley and others added 4 commits March 21, 2024 15:35

feat: improve ux when no output from npm

b799b38

chore: merge main conflicts

2c722b4

chore(release): 5.0.0-beta.5 [skip ci]

fa30d5e

fix: display name when uninstalling wiht no args

a53853b

test: set timeout on install test

ac2c394

mdonnalley force-pushed the mdonnalley/revamp branch from d0e35c4 to ac2c394 Compare March 22, 2024 20:23

mdonnalley and others added 3 commits March 22, 2024 15:06

fix: ensure dir exists before checking for name

566af61

Merge branch 'main' into mdonnalley/revamp

034a067

chore(release): 5.0.0-beta.6 [skip ci]

183be3f

iowillhoit approved these changes Mar 25, 2024

View reviewed changes

Merge branch 'main' into mdonnalley/revamp

12808a5

mdonnalley merged commit 8e632b7 into main Mar 25, 2024
13 checks passed

mdonnalley deleted the mdonnalley/revamp branch March 25, 2024 21:04

justinwilaby mentioned this pull request Apr 2, 2024

plugin-mtls is not installing using heroku cli heroku/cli#2761

Closed

justinwilaby mentioned this pull request May 8, 2024

node 16 EOL ended 7 months and 3 weeks ago heroku/cli#2870

Open

mshanemc mentioned this pull request May 13, 2024

Signature for Community Plugin missing forcedotcom/cli#2848

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat!: use npm for plugin operations #776

feat!: use npm for plugin operations #776

mdonnalley commented Jan 23, 2024 •

edited

Loading

git2gus bot commented Jan 23, 2024

iowillhoit Mar 18, 2024

mdonnalley Mar 18, 2024

iowillhoit commented Mar 21, 2024

mdonnalley commented Mar 21, 2024

iowillhoit commented Mar 22, 2024

iowillhoit commented Mar 22, 2024

iowillhoit commented Mar 22, 2024

iowillhoit commented Mar 25, 2024

iowillhoit commented Mar 25, 2024


		YarnMessagesCache.getInstance().flush(plugin)

		this.log(chalk.green(`\nSuccessfully installed ${plugin.name} v${plugin.version}`))

feat!: use npm for plugin operations #776

feat!: use npm for plugin operations #776

Conversation

mdonnalley commented Jan 23, 2024 • edited Loading

Breaking Changes

Other Changes

Issues/Work Items

git2gus bot commented Jan 23, 2024

iowillhoit Mar 18, 2024

Choose a reason for hiding this comment

mdonnalley Mar 18, 2024

Choose a reason for hiding this comment

iowillhoit commented Mar 21, 2024

QA NOTES

MISC

RELATED ISSUES

JIT PLUGINS

CORE NPM PACKAGE

CORE NPM PACKAGE WITH VERSION

3PP PLUGIN NPM PACKAGE

3PP PLUGIN NPM PACKAGE WITH VERSION

SHORTNAME PLUGINS

GITHUB ORG/REPO Installs

GITHUB FULL URL

GITHUB FULL URL (with .git)

GITHUB URL WITH BRANCH

GITHUB URL WITH BRANCH (with .git)

GITHUB URL WITH VERSION

LINKING PLUGINS

mdonnalley commented Mar 21, 2024

iowillhoit commented Mar 22, 2024

iowillhoit commented Mar 22, 2024

QA UPDATE

iowillhoit commented Mar 22, 2024

QA UPDATE

iowillhoit commented Mar 25, 2024

iowillhoit commented Mar 25, 2024

QA UPDATE

mdonnalley commented Jan 23, 2024 •

edited

Loading