JavaScript IPLD: ESM migration tracking issue

[rvagg]

Ref: ipld/js-dag-pb#52

Background

The "new" IPLD stack was designed as a dual-publish - it's written in ESM, with TypeScript annotations and then compiled to CJS & ESM in a dist/ directory and the types are checked and compiled into dist/types/ directory (there may be a case where it's just an index.d.ts, not a types/ directory) and then npm publish is performed from the dist/ directory only.

So, publish: https://unpkg.com/browse/@ipld/dag-cbor@7.0.2/, doesn't match source: https://github.com/ipld/js-dag-cbor, but it can then be consumed with either require() or import and those will then use the correct forms--import giving you proper ESM so you get the proper static tree that you can shake etc.

Compiling uses ipjs which uses rollup under the hood to compile the code and a bunch of fairly static rules to decide how to create the new dist/ output - including parent package.json, its export map and cjs/ & esm/ directories and their package.json files (for "type": ".." switching).

But, ipjs locks us into a very singular pattern that has been difficult to evolve, and the latest problem @ ipld/js-dag-pb#52 is that we can't add a new entry to the package.json's "exports" map because ipjs has opinions about what should be in there so it knows what to compile and what to output. We could fix ipjs (again), or we could drop the compile step entirely and go with ESM-only.

The JS ecosystem has been rapidly embracing ESM, with many popular packages going ESM-only (most notably all of sindresorhus's many and very popular packages are now ESM only, node-fetch is ESM-only, it's getting hard to do pure CJS and rely on current ecosystem packages without awkward breakage). @achingbrain has been converting the libp2p and ipfs JS packages to publish ESM only (as well as fully embracing TypeScript authoring in the process). So .. it's time.

TODO

1. We need a repeatable pattern to implement in these repos, there already is a lot of consistency but there's also little bit of variation and switching to ESM could introduce more if we're not careful.
   a. A stretch-goal here might be to extract parts of the pattern to unified-ci - mainly the Actions bits.
2. Build step: we still need to compile the types before publishing, so there should probably still be an "build" script that does this. In https://github.com/rvagg/js-bitcoin-block/blob/e91b70c7a38183e9314cb2cb55ee973396724f35/package.json#L24-L31, which doesn't dual-publish, I had a simple "build" that was included with "lint" on every "test", maybe that's appropriate? An addition to GitHub Actions that checks git state to make sure that post-"build" doesn't produce different files might be good too.
3. We need to figure out type definition export settings - currently we have "types" in some packages (but not all) and "typesVersions" in most (all?). fix: add support for node16 module resolution js-dag-pb#52 introduces a "types" export field as well. Some packages have many exports and some have a single export, hence the difference. "typesVersions" with "*" mapping was originally introduced to deal with a TS bug (history of this will probably be in js-multiformats somewhere), we may be able to remove that? Moving forward with 3 separate ways to indicate type definition files is not going to be happy.

The following repos are the most important to make the conversion in:

• js-multiformats: v10.0.0 tracking multiformats/js-multiformats#199
• js-dag-cbor
• js-dag-json
• js-dag-pb: feat!: publish as esm-only js-dag-pb#56
• js-car

Additional repos that are actively maintained that can be converted with lower priority:

• js-murmur3
• js-sha3
• js-blake2
• js-printify
• js-ipld-schema (NOTE: this repo needs more work than just ESM, it shouldn't publish until it's back in sync with the fixtures .. so this conversion could be deferred)

Other repos that are currently in my GitHub could probably be included too (I could move these to the ipld org at the same time): iamap, js-ipld-hashmap, js-ipld-garbage, js-ipld-schema-validator

[achingbrain]

Could I suggest here, if it's not too controversial, maybe using aegir? It solves pretty much all of the problems above and works with Unified CI already.

More than happy to remove functionality (and deps) from it, if the install time is still an issue, but it seems like there's no reason to reinvent the wheel and all the projects can benefit from some collaboration on the tooling.

[rvagg]

I'm still allergic to kitchen-sink packages like aegir, but with all the tooling we're accumulating in these repos it's becoming a hard case to maintain since the devDependency is getting a bit out of hand as it is. My main objection is the contribution hostility for outsiders faced with weird custom tooling ... but we kind of already have that and we don't exactly have people lining up to contribute! So, no, I won't stand in the way of trying to use aegir for this the person(s) tackling this decides it's a more straightforward way to go. The case for using it to get these into unified-ci is pretty strong too.

[RangerMauve]

Some thoughts/questions:

I think it'd be good to ditch commonjs and any build steps if we can.

Do we need to publish the type definitions in separate files when we already have the annotations? Is tsc unable to parse the types out of dependencies with it?

It feels like it should be possible to have all the code be ESM-only, with typescript annotations in the source files as is. Then from JS (in a recent node version or with rollup) you could use import on the js files directly, and use the same import semantics from TypeScript and have the compiler pull the annotations out of the JS file itself. 🤷

Since it seems like we're comfy with having potentially breaking changes, it might be a good time to rip out any non-essential tooling.

[rvagg]

Good question @RangerMauve, I'm not sure TypeScript is quite that intelligent but maybe @Gozala can answer this question.

[Gozala]

Do we need to publish the type definitions in separate files when we already have the annotations? Is tsc unable to parse the types out of dependencies with it?

I’m afraid TS just won’t do it 😭 I really wish there was a way to opt in though.

That said I find compiling just types not to be nearly as painful as transpiling JS, as You can copy & paste code into node repl, import etc without running compilation. Most of type generation usually happens in CI

[RangerMauve]

The whole "can't import jsDoc types" thing might be related to this: microsoft/TypeScript#33136

Doing some digging and testing today.

[RangerMauve]

So, we can get this working if people add to their includes section in their tsconfig.json.

E.g. if we have dag-json only using JSDoc for types, users would need to do the following:

tsconfig.json

{
  "includes": [
    "node_modules/dag-json/**/*.js"
  ]
}

Is this something we would want to subject users to for the sake of getting rid of tooling? (I'm guessing no, but wanted to put it out there anyway)

Related to: microsoft/TypeScript#29824

[rvagg]

I think we're aiming for making users' lives simple here, so making them add that kind of config isn't going to be awesome UX!

We're still going to have tsc in the toolchain, it's been very useful as a lint-like tool for our codebases even when we don't use TypeScript; so given that it's not going away, what we're dealing with is an output-typedefs step, which I don't think is a big deal. It's a bit like a lint but it sometimes involves in some changed files to commit. We just need to make sure that what we publish has the right types—I'm +1 on checking in the typedefs in a types/ directory if we do this, but since we auto-publish anyway we have the option of .gitignoring that but including it on the publish, where the publish involves a build step. Either way is fine as long as we have consistency and don't run the risk of having typedefs out of sync I think.

[Gozala]

Is this something we would want to subject users to for the sake of getting rid of tooling? (I'm guessing no, but wanted to put it out there anyway)

This additionally implies that version of typescript on user device may produce different results and sometimes even error if library uses newer version with more features than client.

Overall I think generating those types on publish is a better compromise than subject users to configuring ts differently which I'm almost certain would be a big can of worms.

[RangerMauve]

I've been doing some more playing around with how we could go about ESM/TS stuff. You can see a mockup of what it would look like to import the dependency in this folder: https://github.com/RangerMauve/ts-dependency-test

If we're okay with going full-ESM and having some breaking changes (major version release for affected packages), I'd like to propose the following:

• Remove the ipjs dependency
• Stick with TS annotations
• Generate d.ts files for all the js files using regular tsc with emitDeclarationOnly: true in the tsconfig.json
• Add those d.ts files to .gitignore so they aren't commited (or we can commit them so that the type changes will be more visible. 🤷)
• Make sure the d.ts files do get published on NPM
• Remove the types, typesVersions, and types fields from package.json
• Remove mentions of cjs from docs
• Have the exports field point to the js files (I think that's the current behavior anyway)

With that in place:

• ESM based JavaScript will be able to import whatever files are in the exports of the package.json
• TypeScript will also follow the exports field to resolve the JS files, and will then find the d.ts files for the JS file automatically
• The overall folder structure will be more simple to inspect since the source will be the actual source
• We can get rid of extra tooling and stick to using raw tsc for generating the types for when we publish

How does this sound?
Should I also do some testing with Deno before working on the conversion?

[RangerMauve]

Just did a test with deno, and all I needed to add was to make sure all the exports keys and values in package.json started with ./