disambiguate "web compatibility"

[bmeck]

similar to "transparent" interop, the use of the term "web compatibility" is a bit muddy in usage.

I think we need to discern 2 main things:

1. Code compatibility concerns: if users are being encouraged to/must produce code that cannot run on web browsers. Examples include:
   • using things that are not present on import.meta in browsers
   • using magic variables like __filename which are not going to be present in the browser
   • using modules written in CJS
2. Platform compatibility concerns: if the Node implementation of ESM varies from the WHATWG implementation in a way that prevent people from shipping an ESM module graph that works in Node to the web. Examples include:
   • having a different cache mechanism for indirection
   • having a different value for import.meta.url
   • having a different resolution algorithm

I want to be very clear that platform concerns are about how ESM graphs are loaded and what contextual data is provided to those modules. Underlying formats used to ship modules is unaffected and may be overloaded through different container mechanism such as using webpackage or BinaryAST.

Notably, I would like to make a discerning line about if importing non-ESM is a compatibility concern on the Code level or the Platform level.

I believe very firmly that importing non-ESM is neither.

• For the Code compatibility concern, the simplest example of this is to show an application that contains only ESM. The Code compatibility concern is about requiring code be written using syntax or APIs that cannot work on the web. If you don't import CJS, you don't have this compatibility concern, even if the feature of importing CJS exists.

• For the Platform compatibility concern, people can claim that import 'foo'; resolving to CJS in Node is a platform compatibility concern. I want to claim this is a false concern.

  1. Having foo resolve to ESM is still possible either by porting, bundling, or shipping multiple builds. At no point is foo required to be CJS by writing import 'foo';.
  2. If we look at the idea of the web shipping a format that Node does not support (such as HTML modules). There is nothing preventing that author of HTML modules to take the inverse approach so that it could be loaded in Node. If we are to state that importing CJS is a platform compatibility concern due to requiring specific dependencies be written in CJS, we would also need to state the inverse for HTML modules ; which is false, as import does not require the format of the dependency being loaded to be anything , be it CJS or HTML.

I will leave migration concerns up to @demurgos 's incoming review of the issue, but felt that we should address the topic of what "web compatibility" is in a way that we can be concrete in what breaks when we talk about features.

I am open to changing definitions above, but want to make us be more mindful of using the term "web compatibility" without explaining what breaks by including a feature. Is it the ability to use ESM loading mechanisms in both places, or is it the ability to run the same source text in both places? We should also seek to prioritize if the ability to run the same source text in both places is mandatory given that any usage of CJS will not be possible to run without some assistance in the browser.

[jkrems]

I would call import 'cjs' a code compatibility concern, just like __filename or accessing node-specific globals like Buffer. In all cases it's valid syntax but will do different things in the browser (in most cases: it will break in the browser). You can change another part of your project to "fix" it but at that point you changed the code and it is no longer import 'cjs'.

Magically switching import 'cjs' once a library exposes a module also has the disadvantage that we get one of two undesirable outcomes:

1. Libraries need to keep exporting default as an object containing a copy of their named exports.
2. Libraries need to treat adding module support as a breaking change.

There's an implicit third outcome, equally undesirable, where adding support for <insert mechanism that allows named exports for CJS> is a breaking change or forces a weird reserved default export.

[bmeck]

@jkrems can I ask for examples that mandates people to import CJS, or presents a path that prevents them from shipping ESM?

What you describe is that there a lack of feature parity between the platforms, I claim that is not a compatibility concern since nothing enforces people to do something that must be incompatible with the web.

You are describing a migration problem as well, but you are not stating anything that prohibits people from shipping a full ESM graph to the web. Do you have examples of how allowing import of formats not supported by the web prevents importing web supported formats?

[bmeck]

@jkrems maybe to clarify a bit, what about import 'cjs'; has different aspects than import.meta.require('cjs') that makes import.meta.require able to avoid breaking module graphs shipped to both platforms.

[jkrems]

what about import 'cjs'; has different aspects than import.meta.require('cjs') that makes import.meta.require able to avoid breaking module graphs shipped to both platforms.

Nothing. It just makes it immediately apparent that a file containing import.meta.require('some-lib') won't work in the browser. For import 'some-lib' I have to start researching and/or reading some-lib's code.

[bmeck]

@jkrems would the notification of linkage failing and certain parse failures not serve the same purpose? In addition as I mention above some-lib could take multiple routes to support shipping both formats. Is your concern just about wanting to define the format of the module being imported at the location it is imported?

[GeoffreyBooth]

I’ve thought it was odd that Node allows import './file' but browsers require the full filename including extension, e.g. import './file.js'/.mjs. Ditto for folders resolving to /index.mjs. Regardless of whether you think this is behavior that Node should allow in ESM, I think this automatic resolving of file extensions and of folder entry points is something that should be on the list of code compatibility concerns.

I think there’s a compelling argument to be made for dropping automatic extension resolution and automatic folder index.mjs discovery, in the interest of browser compatibility. The use case for those features, it seems to me, is import interoperability where Node can automatically resolve to .mjs or .js, or index.mjs vs index.js. That’s a valuable use case, to be sure, but I wonder if it shouldn’t be something that users opt into by adding a loader.

Such an approach would pair with the package name maps proposal to provide ways to do things like import 'lodash' in both browsers and Node. Assuming that gets adopted, lots of NPM modules might be otherwise perfectly web-compatible aside from their dropping of file extensions in import statements, for example. It might be a good idea for Node to nudge them in the direction of browser compatibility, by somehow making them opt in to the browser-incompatible syntax.

[bmeck]

@GeoffreyBooth I've put the resolution algorithm into Platform compatibility concerns.

[bmeck]

I'd also note that even with the example of import 'lodash' which doesn't have an extension it would still be able to resolve to any format, so it is separate from the ability to import non-ESM.

[ljharb]

@jkrems you have to do that research regardless, because any module anywhere in the graph might use fs. Are you suggesting that ESM shouldn't be able to import node's core modules?

[ljharb]

@GeoffreyBooth a) browsers don't mandate the extension, they use URLs. if the URL lacks an extension, so too can the import path. b) with package name maps, you'll be able to omit extensions in browsers (just like is the best practice in node/npm), and it will work the same without a build step. This is a good thing.

[GeoffreyBooth]

a) browsers don’t mandate the extension, they use URLs. if the URL lacks an extension, so too can the import path.

I don’t quite follow. I didn’t mean to imply that browsers care about extensions, I only meant that they require fully resolvable paths/filenames (yes, as part of a URL). My example import './file.js' is using a relative URL, like <script src="./file.js">. If the URL lacks an extension, doesn’t that just mean that the webserver needs to decide what to serve for that? So for import './file', the webserver would need to either serve an extensionless file named file or know to serve file.js, the same way that many webservers automatically serve index.html for folders?

Since it’s not standard for webservers to resolve an URL ending in file to file.js or file.mjs, it feels like something that Node shouldn’t do either, at least not by default. If it becomes standard, such as via package name maps, sure, that would be great. In that case, though, I would think that Node should do it the same way, through package name maps, rather than its own custom implementation.

[ljharb]

It's something node already does by default, and it's something users expect. Whether it's implemented in terms of package name maps or not, I think that it would be extremely hostile to users to omit.

[MylesBorins]

I'm going to modify the pr I opened for file extensions. Seems based on this conversation that the appropriate behavior would be to load the path without any resolution, instead of enforcing file extensions

…

On Thu, Jun 28, 2018, 6:30 PM Jordan Harband ***@***.***> wrote: It's something node already does by default, and it's something users expect. Whether it's implemented in terms of package name maps or not, I think that it would be extremely hostile to users to omit. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#142 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAecV0KevcwNdZ3LQHGyEYapGdOGNSuAks5uBVjugaJpZM4U7_X5> .

[GeoffreyBooth]

@ljharb I hear what you’re saying, and the user hostility is why there should still be some way for them to do it: via a loader, via package name maps, or something else. But put simply, either we’re trying to be equivalent with browsers by default or we’re not.

The current behavior certainly wouldn’t change in CommonJS, it’s only ESM we’re discussing here.

I’ve come to accept that Node isn’t going to support everything that current transpilers do, at least not without loaders or other hacks/patches. I think the explanation that “Node has added support for import and export in the same way that browsers support that syntax and ES modules” is a compelling explanation that most users will grasp, and helps defuse complaints about things that don’t behave as users expect or might want.

[ljharb]

I don't think we should be trying to be equivalent by default with browsers - browsers don't have filesystem access, nor a massive CJS ecosystem it needs to retain compatibility with _{(they have a much more massive legacy ecosystem to retain compatibility with)}.