Dual package hazard #409
Description
I’ve created a new demo repo at https://github.com/GeoffreyBooth/dual-package-hazard that shows how the hazard discussed in #371 is still present and capable of causing issues in dual packages that use the 'pkg' / 'pkg/module' approach.
When I created issue #371, I assumed that the hazard was specific to what I called divergent specifiers; specifiers like 'pkg' that resolved to different targets in CommonJS and ES module environments within the same application. But as this new example shows, that understanding was incomplete; the hazard is caused when any package publishes both CommonJS and ES module sources, even if those sources are made available via different specifiers. Hence the new name “dual package hazard.”
Before I realized this, I thought that the 'pkg/module' approach would prevent the hazard and also that asking package authors to define separate entry points for each module system (e.g. 'pkg' and 'pkg/module') was a minor inconvenience that was justified in order to prevent this class of bugs. Unfortunately, since 'pkg/module' doesn’t actually accomplish the goal of preventing such bugs, I don’t see how we can justify 'pkg/module'. It’s clear that the community’s preference is to create packages where the same bare specifier ('pkg') points to multiple entry points. Without the hazard as a justification, I don’t see a coherent argument for permitting dual packages while not supporting divergent specifiers such as via conditional exports. Conditional exports don’t “make things worse”—the hazard is the same in both scenarios. Conditional exports have an advantage over 'pkg/module', however, in that tools can definitively determine all the entry points of a package, rather than needing to make an educated guess after reviewing each export. And of course, conditional exports provide the UX that users want.
So I think the choice isn’t between conditional exports and 'pkg/module'; it’s between permitting the hazard and preventing the hazard. Permitting the hazard, at the moment, means conditional exports; but other proposals (like a separate field similar to "module" just for the package entry point) could also work. Preventing the hazard, however, isn’t something we’ve really discussed since until recently we thought that 'pkg/module' did that job.
I think the only way to truly prevent the hazard is to limit each package scope to a single type. So if a package.json has "type": "module", all files imported from that package must be ES modules; trying to use a CommonJS file contained within would throw an error. And the inverse for "type": "commonjs" or packages that lack "type": the only supported files from within such a package must be usable as CommonJS. We would also need to somehow not let packages work around this limitation via multiple package.jsons, e.g. something like ./esm/package.json with "type": "module" and ./cjs/package.json with "type": "commonjs". Furthermore we would have to find some solution for the global scope, e.g. node file.mjs, which would under these rules throw an error since the default global scope is CommonJS and therefore ES module files would be disallowed. There’s also the trick that packages could load a file as a string via fs.readFile and then evaluate that string; needless to say, trying to truly prevent the hazard would mean finding and plugging lots of loopholes. Whereas all the community needs to do is to find one that we’ve missed, and then they can flood the npm registry with packages using that method.
Such an approach would also be a pretty dramatic departure from how Node traditionally works, where generally code can open any file that the node process has access to. It’s also quite paternalistic toward users; we would be preventing dual packages purely because we don’t trust package authors to be able to author bug-free dual packages. I’ve already written detailed docs explaining how they can do so; I think most package authors will be able to handle it.