Support for Import Maps

[wesleytodd]

I was hoping to start the conversation around getting Import Map support into core. I figured starting here with a proposal was better than starting right off with a PR.

What Are Import Maps

This proposal allows control over what URLs get fetched by JavaScript import statements and import() expressions. This allows "bare import specifiers", such as import moment from "moment", to work.

The proposal was mainly targeted at browsers where URL imports have many downsides without bare specifier support. It enables a standardized map file format to be used when resolving import specifiers, giving developers more control of the behavior of import statements.

Here is an example of the format:

{
  "imports": {
    "todayis": "node_modules/todayis/index.js"
  },
  "scopes": {
    "node_modules/todayis": {
      "english-days": "node_modules/english-days/index.json"
    }
  }
}

Proposal link: https://github.com/WICG/import-maps

Why should they be supported in Node.js?

Currently we use the structure of the filesystem and a resolution algorithm to map import specifiers to files. These files are typically installed by a package manager in a structure which the Node.js resolution algorithm knows how to resolve. This has some drawbacks:

1. It is slow
2. It can lead to unexpectedly resolving modules
   • if you have a node_modules at your filesystem root for example
   • if your package manager has hoisted a transitive dep and you access it from the top level
3. Tools are required to implement copies of the Node.js resolution logic
4. Users and package managers jump through hoops to try and make the filesystem performant

If we had import maps we could circumvent most of this. Package mangers can generate an import map, enabling perf improvements and simplicity in their implementations. It increases startup performance of apps because they don't have to do as much filesystem access.

One interesting example of what import maps enables is filesystem structure independent workspaces (think yarn workspaces but without the requirement of the modules being under the same root directory).

Implementation

As an example implementation I created this gist:

https://gist.github.com/wesleytodd/4399b2351c59438db19a8ffb1f3fcdca

To run it:

$ git clone git@gist.github.com:4399b2351c59438db19a8ffb1f3fcdca.git hello-today
$ cd hello-today
$ npm it

This uses an experimental loader which loads an importmap.json and falls back to the original filesystem lookup if it fails. I am also pretty sure this very naive implementation is missing edge cases, but in these simple cases it appears to work.

Obviously for support in core we would not use a loader, but I think the rest would be similar. Open questions I have are:

Where would node find the importmap.json?

My first idea is at node_modules/importmap.json. I think this would be expected behavior from users if their package managers started writing this file.

How would users override the importmap.json?

For this I think a reasonable approach would be a cli flag (--importmap) and a package.json ("importmap": "./importmap.json") field.

Do we want to wait for browsers to standardize import maps?

#49443

I think that this is a very valid concern. Is there a way to help push that forward? The benefits are pretty large in node IMO, and it would superseded all the work currently being done on yarn pnp and npm 8. If we let those go forward but then this spec lands it might be even worse for shifting the community.

Could we release it as flagged & experimental until browsers standardize? This would mean users could opt in, but as it changed we could follow along.

---

Thoughts? Next steps?

[ljharb]

Since import maps seem far from being finalized or standardized in browsers, it feels like it’d be best to confine it to userland loaders for the time being.

[wesleytodd]

@ljharb Added an open question at the end with my concern:

it would superseded all the work currently being done on yarn pnp and npm 8. If we let those go forward but then this spec lands it might be even worse for shifting the community.

[ljharb]

I don’t think it would supersede it - they’d just get to provide an import map instead of their current approach, whenever it became available. Perhaps @arcanis can weigh in here.

[jkrems]

The benefits are pretty large in node IMO, and it would superseded all the work currently being done on yarn pnp and npm 8. If we let those go forward but then this spec lands it might be even worse for shifting the community.

I assume that import-map.json would be written by package managers (if they choose to support it). Both yarn and npm have already somewhat committed to a public interface for their resolution structure. Without strong signals from their side that they'd realistically move to a shared metadata format, I don't see them making the switch. pnp.js would stick around anyhow. And require-style resolution would still be needed for backwards compatibility. So they would need two files instead of the one they already have.

That being said: Having a well-supported loader that uses an import map would be neat to have. But I agree with @ljharb that at this point it doesn't really belong into node core. There are some pretty major open questions with import maps in node as well. E.g. your example lays out relative file URLs. One major development in more recent package layouts was that the files do not actually live in node_modules/ (if possible). And in that world, a stable (version controlled) import-map.json can't (?) really use actual file URLs afaict.

[wesleytodd]

I don’t think it would supersede it - they’d just get to provide an import map instead of their current approach, whenever it became available. Perhaps @arcanis can weigh in here.

"Instead" is superseding it. Users will couple to what the package managers release, and it will not be a simple matter to just replace it. This could be a large hurdle to overcome in the long run.

Both yarn and npm have already somewhat committed to a public interface for their resolution structure. Without strong signals from their side that they'd realistically move to a shared metadata format, I don't see them making the switch.

I agree we would want strong buy in from the package managers. I presented a PoC of workspaces built on top of this to @ruyadorno today and he is going to show it internally at npm. I would love to also hear what @arcanis has to say on this.

pnp.js would stick around anyhow.

This is part of the problem I hope we could solve here. If node core supported a shared format we wouldn't fragment the ecosystem with package managers providing their own loaders. It is much better IMO to have them build to a shared spec the runtime understands.

One major development in more recent package layouts was that the files do not actually live in node_modules/ (if possible).

Import maps specifically solve this problem. They can point anywhere on the filesystem. In the workspace example I have (not oss yet) I am working on moving them into a shared workspace cache.

And in that world, a stable (version controlled) import-map.json can't (?) really use actual file URLs afaict.

In the global shared cache scenario it would not be portable. The example I showed using relative urls to make it portable, but my workspace PoC uses full paths. I think it is reasonable to support both.

[guybedford]

Amazing to see this work, thanks @wesleytodd for investigating.

@ljharb would you disagreement over stability be mitigated by flagging such an implementation. Personally I see no harm in experimentation in core, rather than pushing out experimentation here entirely to userland. It shows a willingness for core to cooperate with projects exploring this space over forcing completely separate approached entirely leading to the fragmentation @wesleytodd mentions.

It may well work out that we find that projects do need their own full import maps implementations, in which case Node.js can always deprecate such an experimental flag based for that outcome.

As @jkrems has touched on, one of the issues here is that import maps as a replacement for the entire resolution system is tricky due to these details. For example, with conditional exports, the same package can resolve different depending on environment information. Import maps don't have this same functionality and likely won't... rather they can be considered an output of an environment function at some level - an import map specific to the browser is different to an import map specific to Node.js.

In this way I prefer to think think of import maps somewhat like "ephemeral views" into the project, representing a caching / precomputation of the resolution operation. They have a locking property, but that should not be confused with lock files, which are about install reproducibility, something that is clearly a superset of the features import maps provide.

Thinking about the user features, I see these as goals of import maps in Node.js, when they can be allowed to be complementary to the Node.js resolver:

• Mocking (map just the mocked URLs)
• Vendoring (map just the vendored packages)
• Resolution inlining for performance / encapsulation from FS / security (mapping everything)

I do see having import maps as being the default and only form of resolution in Node.js as a non-goal here though. Rather an approach where the import map can allow intercepting the resolution and then having that work with userland approaches to explore the above space seems most beneficial to me.

[wesleytodd]

Amazing to see this work, thanks @wesleytodd for investigating.

Thanks for responding on twitter and sending me down this path :)

For example, with conditional exports, the same package can resolve different depending on environment information.

I think this is also solvable inside import maps. The spec doesn't really specify what the scope can be used for, so I see no reason that information cannot be encoded into the scope mapped from the conditional exports. Either that or the tooling would generate a separate environment import map (as you mention).

They have a locking property, but that should not be confused with lock files, which are about install reproducibility, something that is clearly a superset of the features import maps provide.

In the PoC I have not shared yet I actually use npm (@npmcli/arborist technically) to generate the tree and then the import map. This means that it respects the lock file, and will reproduce the same import map from the same lockfile.

I do see having import maps as being the default and only form of resolution in Node.js as a non-goal here though.

Agreed, and thus my example falling back to the existing file system based resolution. I think this would be a very important key feature to keep.

[coreyfarrell]

Could we release it as flagged & experimental until browsers standardize? This would mean users could opt in, but as it changed we could follow along.

Does --experimental-loader make this possible already? For testers npm i wesleytodd/import-map-loader would be easier than rebuilding node.js.

In the PoC I have not shared yet I actually use npm to generate the tree and then the import map. This means that it respects the lock file, and will reproduce the same import map from the lockfile.

I'm interested in how large the generated import maps would be. The node_modules of one of my current projects contains 576 modules and 6355 .js files. Does every .js file get listed in the import-map? At what point does the load/parse of a large import-map exceed the current resolver overhead for loading only a couple files from node_modules? I'm also interested in the potential memory overhead of keeping a large import map loaded.

[wesleytodd]

Does --experimental-loader make this possible already? For testers npm i wesleytodd/import-map-loader would be easier than rebuilding node.js.

That is really how I am testing it today. If you check the test script in the package json of the example you see it passes --experimental-modules --loader ./loader.mjs

Does every .js file get listed in the import-map?

No. It only lists packages then falls back to file system imports for the rest. Although that was a decision I made, which you could change for many interesting reasons (as Guy mentioned, mocking).

At what point does the load/parse of a large import-map exceed the current resolver overhead for loading only a couple files from node_modules?

My guess? Never. That said we should test it. Also, it would be really simple to prune the import maps based on static analysis if the app entry point does not load anything from that module. It would also make sense that the package managers would generate a --production which would only be the production dependencies.

I'm also interested in the potential memory overhead of keeping a large import map loaded.

I also am guessing here, but not much. I will try this out on the largest projects I can find at work and report back on the size. Also, we can easily optimize the the memory footprint from the import map format to something better at runtime then just throw away the json format.

[coreyfarrell]

Does --experimental-loader make this possible already? For testers npm i wesleytodd/import-map-loader would be easier than rebuilding node.js.

That is really how I am testing it today. If you check the test script in the package json of the example you see it passes --experimental-modules --loader ./loader.mjs

Ah not sure how I missed the example link. I'll check that out as soon as I have a chance.

It only lists packages then falls back to file system imports for the rest. Although that was a decision I made, which you could change for many interesting reasons (as Guy mentioned, mocking).

If the import-map is has module granularity (not listing every file within) then this greatly reduces my performance/memory concerns.

At what point does the load/parse of a large import-map exceed the current resolver overhead for loading only a couple files from node_modules?

My guess? Never. That said we should test it. Also, it would be really simple to prune the import maps based on static analysis if the app entry point does not load anything from that module. It would also make sense that the package managers would generate a --production which would only be the production dependencies.

Static analysis sounds difficult if possible, especially on the development side. I'm thinking of babel plugins as a perfect example, without specific knowledge of babel you cannot know that {presets: ['@babel/env']} in babel.config.cjs means that @babel/preset-env is loaded.

[ljharb]

@guybedford flagging it is better, for sure, but i take it as a given that it'd start out flagged. My main concern is that unless import maps stabilizes, is standardized in browsers, and also is usable for node without doing backflips, then i'd hope node core would never ship any form of import maps - put more positively, when those things happen, node core would obviously ship import maps! Given the large uncertainty, it seems premature to me to even signal a possible future core implementation when there may be none at all.

[arcanis]

One interesting example of what import maps enables is filesystem structure independent workspaces (think yarn workspaces but without the requirement of the modules being under the same root directory).

The problem behind disjoint workspaces isn't in term of resolution - that's fairly simple to implement - but rather developer experience. For example, if you run yarn add foo into the disjoint workspace, how does Yarn knows that it needs to update the lockfile from a completely different folder¹?

¹ It's a rhetorical question; there are ways, but they are all less good that just telling our users to use link: or portal: to add their disjoint workspaces as external dependencies. Not everything has to be in the core 🙂

it would superseded all the work currently being done on yarn pnp and npm 8. If we let those go forward but then this spec lands it might be even worse for shifting the community.

I don't think it would. It's not even clear whether import maps would have all the features we would need. For example, Yarn is able to throw semantic exception when a package isn't reachable, explaining why the package cannot be found. It's quite harder for import maps, which don't have package-level semantics - just folder-level semantics:

Something that got detected as your top-level application (because it doesn't seem to belong to any package) tried to access a peer dependency; this isn't allowed as the peer dependency cannot be provided by any parent package

Required package: react (via "react")
Required by: frontend/app.js

I agree we would want strong buy in from the package managers. I presented a PoC of workspaces built on top of this to @ruyadorno today and he is going to show it internally at npm. I would love to also hear what @arcanis has to say on this.

I think my main question is: what do you think this would improve? I know you mentioned you would find it cleaner, but that's subjective. I personally find a JS API clearer than a file format, for example. So in concrete terms, what would it bring to the table?

• Compatibility would be the most obvious place where it could bring some wins. At the same time, even with the node_modules system being more or less universal, everyone still ends up reimplementing the Node resolution somehow. This to me signals more a lack of good API that the need for another universal format.

• Feature set is definitely not one - for obvious reasons loaders are able to do everything the import maps can do, plus more.

• Cleanliness is maybe one, but again that's very much open to debate and I don't think there's a good or bad opinion on this one. There are some parts in PnP that I find a bit messy myself (mostly the filesystem overlay), but those aren't the ones that the import map would remove.

• Performance could be one, but I don't know how true that it, and it could easily go in both ways. For example Yarn currently has this "fallback" feature, where an undeclared dependency can still be accessed if listed by the top-level package (because it's the only safe and unambiguous case). Implementing that in import maps would require to duplicate the top-level dependencies into each and every package record. I don't know how efficient this would be.

  • Additionally, since PnP is already doing static resolution, there is very little performance gain to have by using import maps instead. We already don't need to query the filesystem, except for the extension part. If something needs to be improved in this regard, I'd suggest to rather focus on making file extensions required ... but that's another battle 😛

Overall I'm just not convinced this is the most impactful place to put energy on. I'd rather talk about making the fs module extendable in a proper way, for example.

[bmeck]

So, we already have this feature of static resolution mappings to some extent using https://nodejs.org/api/policy.html#policy_dependency_redirection ; in all honesty, even though I wrote the feature and have spent some time tuning performance, the benchmarks there are not pleasant when you run them on medium sized apps particularly regarding startup time. I think we should be more critical about the benefits here as it is not a sure win in terms of performance to merely have static resolution.

If we could also address the other concerns about the limitations of only having data for resolution and adding additional meta-data that seems fine, but starts to seem to just be working towards a different policy file.

[wesleytodd]

Given the large uncertainty, it seems premature to me to even signal a possible future core implementation when there may be none at all.

Where can we look to help provide more certainty? Is it just a matter of "we cannot do it unless the browsers ship it first"? I obviously do not want to repeat the mistakes of the past, but this seems like a hard line to take long term.

but rather developer experience

I think tooling DX is a separate concern. I think the goal of this as a core api would be to enable experimentation in user land on top for what DX should look like. The goal of this proposal is to focus on if this is a good idea for inclusion into core.

they are all less good that just telling our users to use link: or portal:

Two concepts which are exclusive to yarn, made up by yarn, and so not a solution in this context unless you seek to standardize them. This only continues to widen the chasm between package managers, and increase the barriers to entry for another package manager.

For example, Yarn is able to throw semantic exception when a package isn't reachable

IMO, this could be a good addition to core, It bridges the gap and brings the whole community up together. If there are hints or metadata which pacakge managers can give to the runtime to improve in this way it would be great! We should look for a loader api which enables this, and I believe that import maps could be that api.

I personally find a JS API clearer than a file format, for example.

I am fine with it being a js api. My internal workspace PoC is that, it follows more closely the approach in PnP, but uses an import map as the structured input format.

This to me signals more a lack of good API that the need for another universal format.

Completely agree! This paragraph this sentence is in is exactly my goal here.

loaders are able to do everything the import maps can do, plus more.

Do you mean this spec? https://whatwg.github.io/loader/

we already have this feature of static resolution mappings

If we could also address the other concerns about the limitations of only having data for resolution and adding additional meta-data that seems fine, but starts to seem to just be working towards a different policy file.

This is close but not equivalent right? Also I think the use case of security concerns remapping modules is a bit different scope than import maps. I actually think that import maps with the features in the policy api would be a better way to implement what is being done because it would be more generic and more approachable by users.

the benchmarks there are not pleasant when you run them on medium sized apps particularly regarding startup time

Do you have links for this? I would love to look at them. I admit that I have done no benchmarking in this regard, and it for sure would have to be done.

[bmeck]

@wesleytodd see stuff like #29527 which would add a benchmark to node specifically about that feature.