[epic] npm logging, output, and cleaning

[wraithgar]

Epic

We are going to consolidate and clean up the code npm uses for output, logging, and for sanitizing strings before doing those things.

Currently these are the modules used:

• proc-log (extend)
• proggy (leave alone for now)
• npmlog (deprecate)
• gauge (deprecate)
• are-we-there-yet (deprecate)
• lib/npm.js
• lib/utils/log-shim.js (will go away)
• lib/utils/log-file.js
• lib/utils/replace-info.js (move to new module)
• npm-registry-fetch#cleanUrl (move to new module)
• lib/utils/display.js
• lib/utils/error-message.js
• lib/utils/pulse-till-done.js (will go away)
• lib/utils/reify-output.js

This is likely not a comprehensive list, and not everything here belongs in our new consolidated package for npm output concerns.

Tasks

Preview Give feedback

1. proc-log should have current capabilities moved to sub method .log()
2. proc-log should get a method for "standard output"
3. proc-log should get a method for "error output" (distinct from log.error)
4. replace-info and cleanUrl will be moved to a new package @npmcli/redact
5. All of the other functionality will live in npm until proven to need to live elsewhere
6. The single console.log in @npmcli/run-script will need to move to the new proc-log standard output method
7. feat: refactor terminal display cli#7339
   +0
8. feat: add time and input proc-log#80
   +0
9. Rewrite proc-log#77
   +0
10. Use proc-log for stdout and stderr cli#7373
    +0
11. Include metadata that can be emitted along with events proc-log#81
    +0
12. fix: better output colors cli#7377
    +0
13. feat: add flush method to output proc-log#85
    +0
14. feat: add META symbol to exports proc-log#84
    +0
15. https://github.com/npm/cli/blob/latest/lib/utils/format-search-stream.js#L104
16. Remove tables (where possible)
17. Move https://github.com/npm/disparity-colors to libnpmdiff
18. Remove log "labels"
19. Stop prefixing every log to stdout with npm ${level}
20. [BUG] npm-pack --json emits prepare script commands, making json invalid cli#7354
    Bug Priority 1 Release 10.x +3
    
21. Remove more table output cli#7407
    +0
22. Remove table output cli#7401
    +0

Notes

npm progress bars will go away for now

This replaces the following issues

• feat(display): update progress display (aka. remove bar) #545
• feat(display): standardize stderr & stdout #546
• Epic: Display Progress #422

[lukekarrys]

Putting some notes here for future improvements that this will now unlock:

npm.chalk/logChalk/color/logColor

Remove npm getters for chalk, logChalk, color and logColor. These are used throughout npm/cli so we have can call chalk.red(someText) and have it respect the --color config including the inferred value based on the stdout stream. This was introduced to take the place of color/logColor booleans but those are still in use some places.

• They are also used in other dependencies (non-exhaustive list)
  • libnpmexec
  • npm-audit-report
• In order for dependencies to emit colorized output we would need a solution like chalk-template which would allow for the emitted data to contain the colorized that should be used if colors are allowed, and to have them stripped out if not.
  • chalk-template is one solution to this. Another would be something like (first draft untested code follows). The only important bit is that producers can send signal data hinting at what colors to use without needing a parent to pass any configuration in.

// producer
require('proc-log').output.standard({
  // Note this is not a good API because we pass everything through util.format
  // and this solution relies on a string property like `withChalk` being well-known
  // and not used anywhere else. We could symbols instead but they'd need to be exported
  withChalk: (c) => `${c.red('this is red') ${c.green('this is green'}`
})

// consumer
process.on('output', (level, value) => {
  if (level === 'standard' && typeof value.withChalk === 'function') {
    process.stdout.write(value.withChalk(myStdoutConfigChalkInstance))
  } 
})

read()

Previously anywhere we called read() (or anything that requested user input like readline()) we needed to manually call methods like npmlog.(endableProgress|disableProgress)() or more recently proc-log.(pause|resume)().

Now that we have all display functionality in a single place, we can make requesting user input also be events based. The only difference is that those events need to be await-ed.

Here's an example of what could happen now:

// add this to proc-log
module.exports = {
  input: {
    read: function (...args) {
      let resolve, reject
      const promise = new Promise((_res, _rej) => {
        resolve = _res
        reject = _rej
      })
      process.emit('input, 'read', resolve, reject, ...args)
      return promise
    }
  }
}

// producer
const { log, input, output } = require('proc-log')

log.warn('Some warning')
output.standard('About to ask for some input')
const res = await input.read({ prompt: 'favorite color?' })
output.standard(`You said ${res}`)

// consumer
const { read } = require('read')

process.on('input', (level, ...args) => {
  if (level === 'read') {
    // Should hide progress, logs, etc
    const [resolve, reject, opts] = args
    read(opts).then(resolve).catch(reject).finally(() => {
      // Should restore progress, logs, etc
    })
  }
})

[lukekarrys]

Customizable colors

npmlog used to be in charge of colors which we set once at runtime. The Display class now generate a color palette on load which takes configured options.

Our goal is to have one (1) good and accessible color palette for all output shown to the terminal. BUT if we find cases where that is impossible (eg dark vs light terminal backgrounds) it is now at least possible to have a config option to change colors.

[lukekarrys]

Command templates

What if a command could do this?

// producer
const data = { name, version }
output.standard(data, {
  templates: {
    // pick any templating DSL
    human: `{{ name }}@{{ version }}`,
    parseable: `{{ name }}\t{{ version }}`
  }
})

// consumer
process.on('output', (level, data, { templates }) => {
  if (level === 'standard') {
    const templateName = templates[this.human ? 'human' : this.parseable : 'parseable' : '']
    process.stdout.write(template(data, templateName || JSON.stringify))
  }
})

[wraithgar]

Closing this out. This epic is considered done. There are follow up tasks that we will put back into the planning queue. This will all go out w/ the next npm release.