Incoming: Arborist

[kemitchell]

Isaac was kind enough to confirm that read-package-tree will probably be deprecated in favor of Arborist. Arborist will expose a promise API.

I have watched release on the Arborist repo. When the time comes, licensee will want to switch. But I'd strongly recommend that we hold off doing any substantial development until that decision is taken.

[kemitchell]

https://gist.github.com/isaacs/84a2d788258c6bd120664e582bff658f

[isaacs]

Would you be open to a PR that changes this to use Arborist instead of using read-package-tree or shelling out to npm ls? I think it's about ready to start poking at, and having our licensee test meaningfully passing in CI would be nice.

[kemitchell]

@isaacs I've been strongly considering rewriting Licensee as a new package, dropping old Node support, using Arborist, offering substantial additional functionality, and charging for closed and commercial use. The code would remain public. I'm sure I'd run CI.

I've probably spent more time on all sides of licensing in npm than most anyone I know, but it's not exactly fun (or risk free) coding package metadata corrections, staying on top of SPDX upstream, writing generators to spit out due diligence tables, and so on. Especially after doing it for years.

If Arborist fells ready, I can put this on my weekend list.

[isaacs]

Yeah, it's definitely close. We'll probably have a v7 beta in a few weeks. I'd set the dependency to ~0.0.9 || 1 for now, and then bump it to just 1.x once it's out. It'll have a few more bugfixes I'm sure (just found a new one today, actually), but the API surface isn't expected to change between now and 1.x. The biggest shortcoming right now is that the docs are woefully incomplete.

I'm happy to help with all the tree handling stuff, but I'm not eager to wade into the actual license logic :)

[jsha]

FYI it looks like Arborist is stable now, and read-package-tree says:

The functionality that this package provided is now in @npmcli/arborist

One small reason it would be nice to update: read-package-tree depends on path-is-absolute, which is also deprecated:

This package is no longer relevant as Node.js 0.12 is unmaintained.

Since npm depends on licensee, that means npm indirectly depends on a pair of deprecated packages.

[kemitchell]

@jsha thanks for the heads-up. I still think that Licensee should get in line with npm CLI on Arborist.

Any personal interest here? I'd certainly welcome a PR. I'll get to it eventually, but can't say when.

PS: Three cheers for Let's Encrypt! I've more than a few LE certs myself.

[jsha]

I'm pretty rusty on the Node ecosystem lately - I noticed the issue because I just installed npm for the first time in a while and wanted to look at its dependencies. So the migration is probably more than I can handle right now, but who knows!

PS: Three cheers for Let's Encrypt! I've more than a few LE certs myself.

Aw, thanks! That's always lovely to hear.

[kemitchell]

Poking around with Arborist for the license checker a little.

Good News: Arborist can spit out a nice, flat structure representing the current node_modules tree.

Bad News:

• AFAICT, the nodes of the trees Arborist spits out do not have extended metadata from package.json, such as the sanitized license value or homepage. We are going to have to load package.jsons on our own, though of course there's a package for that.

• Arborist does a lot more than old read-package-tree used to do, including all the npm install-related diffing. It's a bigger dep with a bunch of direct dependencies for functionality behind that Licensee simply isn't gonna need.

[ljharb]

The former seems likely for arborist to add, assuming that data is in the packument.

[kemitchell]

Arborist is no longer a separate repo. They've folded it into the larger CLI repo.

It's clearly processing license metadata. There's a subcommand of the arborist CLI that prints package counts by license: https://github.com/npm/cli/blob/2db3eff44d7bb86b956207cc63d279806fd14ed0/workspaces/arborist/bin/license.js

[isaacs]

Last I was involved in this, I believe that the blocker to having license info in Arborist by default is for it to be included in the minified packuments served by the registry. This is still the only blocker:

$ curl -s https://registry.npmjs.org/abbrev -H accept:application/vnd.npm.install-v1+json | npx json versions["'1.1.1'"]
{
  "name": "abbrev",
  "version": "1.1.1",
  "devDependencies": {
    "tap": "^10.1"
  },
  "dist": {
    "integrity": "sha512-nne9/IiQ/hzIhY6pdDnbBtz7DjPTKrY00P/zvPSm5pOFkl6xuGrGnXn/VtTNNfNtAfZ9/1RtehkszU9qcTii0Q==",
    "shasum": "f8f2c887ad10bf67f634f005b6987fed3179aac8",
    "tarball": "https://registry.npmjs.org/abbrev/-/abbrev-1.1.1.tgz",
    "signatures": [
      {
        "keyid": "SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA",
        "sig": "MEYCIQDvQCH2XtwIWIVnBSH4P51+UstW+ybuYvlEWwSQoGW7fgIhAJleZ3eJj+NTABBRNuW2xhR8pQwFRPSd9cFjP/aS3RrE"
      }
    ]
  }
}

Once the registry starts including license info in the minified packument, Arborist will track it, as it is on the list of fields that Arborist would record in the lockfile and add to Node objects.

[kemitchell]

It looks like sometimes I'm getting data on node.package.license, and sometimes I'm not.

@isaacs, sorry to bug you here---I'm a bit afraid of wading into npm/cli issues. Is Arborist still going to rely on minified pakuments from the registry if we always just ask for the local tree with loadActual?