Import meta require

[MylesBorins]

From nodejs/node#21317

Hey All,

This builds on top of @targos's earlier implementation of import.meta.require. It has been refactored to work with our current layout for the module system. I've also added some basic tests, let me know if you think we need something more robust.

@nodejs/modules I am not planning to land this without consensus from the group, but I think that this is a necessary base feature no matter what implementation we move forward with. I'll open a corresponding issue to discuss.

[zenparsing]

Thanks for the example @michael-ciniawsky , I was wondering what it would look like.

What's the value in making users perform this createRequireFunction incantation in order to do something that they will need to do for interoperability purposes? import.meta.require is searchable and obviously not a web thing.

Generally speaking, if we provide a feature because users need it, then we shouldn't also punish them for using it by intentionally making it difficult to use.

[michael-ciniawsky]

It's definitely debatable and if users absolute dislike (e.g shown by a survey) it, it's definitely better to favor DX and UX over long-term concerns in regard to potential technical debt etc. Still createRequire should be put on the agenda before this lands as it is a considerable alternative imho. Ideally with a vote by nodejs/modules to choose between one of the two for the minimal kernel implementation afterwards (to avoid getting lost in endless bikeshedding 😛)

[devsnek]

i'd like to make sure we have this repo set up to automatically pull updates from nodejs/node before we're merging changes

[bmeck]

@zenparsing This is not just about UX/DX. createRequireFunction has utility that import.meta.require is unable to replicate:

1. Web Compatibility Path:

createRequireFunction is able to allow developers to write ESM code that can function in both client and server environments without modification to the ESM source text:

import {createRequireFunction} from 'module';
createRequireFunction(import.meta.url);

// web 'module'
let base = import.meta.url;
let bundle = {
  '/foo.js': ...
}
export function createRequireFunction(url) {
  return function require(path) {
    return bundle[resolve(base, path)];
  };
}

import.meta.require is unable to do likewise because you cannot create a shim of any kind for it by its placement on import.meta. The only way to achieve this is to require source code transformations of ESM on the web.

2. Static analysis

import.meta.require has the same level of static analyzability as import.meta.require at a naive level. However, module namespaces are not mutable and thus do not have to deal with an entire category of static analysis bail outs that import.meta.require must deal with:

// don't bail out here
import.meta.require('fs');
import.meta.require = function () {};
// bail out here
import.meta.require('fs');


// using window.require
require('fs');

// import.meta.require
var require = import.meta.require;

This is only a realistic compatibility concern if people choose to have incompatible implementations of import.meta.require due to using things like AMD, etc. However, the alternative doesn't suffer this complexity.

3. Simpler code mods / upgrade path

Tools and/or humans can add a simple prefix to files if they wish to use require in the expected ways without facing bailout concerns:

import {createRequireFunction} from 'module';
var require = createRequireFunction(import.meta.require);

Doing so with import.meta.require faces the bailout complexity mentioned previously.

4. Better Errors

import.meta.require throws errors at runtime when it is not defined. This allows for code to be written in such a way that problematic usage is not found until after an application has fully started up, such as in the case of lazy requires:

db.onconnect = (conn) => {
  // errors here
  const models = import.meta.require('models')(conn);
};

The more conventional way of using createRequireFunction would be to use static import:

// errors before any evaluation
import {createRequireFunction} from 'module';
var require = createRequireFunction(import.meta.require);
db.onconnect = (conn) => {
  const models = require('models')(conn);
}

It should be noted that createRequireFunction faces similar problems at runtime if you do dynamic import, but I don't know of a good reason to use dynamic import for this:

// I don't know of any reason to do this?
// maybe just to see if `module` exists???
// but if you need to use `require` what help is that
var {createRequireFunction} = await import('module');

5. More complete feature detection

Even though it is statically analyzable, sometime feature detection is desirable for modules. This feature is still preserved using module namespaces.

import * as module from 'module';
// note that this is able to be analyzed even if `module` is a shim
// this is because we can statically analyze what `module` exports
if (module.createRequireFunction) {
  // ...
}

import.meta.require is unable to be shimmed and thus faces problems with feature detection if you are unable to determine the deployment environment.

---

In addition, even with the added utility, createRequireFunction has a better upgrade story towards isomorphic code. We should aim to encourage users to write compatible code. This is not by any means an attempt at "punishment".

To disallow code that uses createRequireFunction you can use a loader that changes the module record resolved for "module" to be one without createRequireFunction, one that has deprecation warning, one that uses a shim, etc.. In order to disable code that uses import.meta.require you do not have similar approaches, and instead must rely on the rudimentary undefined is not a function error that users know well. The only alternative is to add a prefix to the source text being loaded during loading:

{
  const _require = import.meta.require;
  // wrap it or replace it:
  import.meta.require = wrap(_require);
}

However, these once again reach bailout considerations mentioned above.

---

Overall, we should encourage simple usage of code that migrates towards compatibility stories in multiple environments and tooling. However, I do not believe that import.meta.require works towards that goal. createRequireFunction has simplistic usage if you do the prefix behavior mentioned above and does not require rewriting all usages of require to be import.meta.require to prevent bailout possibilities.

[GeoffreyBooth]

Remind me never to get into a debate with @bmeck.

[mcollina]

@bmeck can we call it module.createRequire(url) instead of createRequireFunction?

(+1 on the approach anyway)

[WebReflection]

The usage of synchronous XHR is discourage and the creation of that module file would need bundlers regardless, meaning that Web Compatibility is a myth here.

No Web developer ever used require right away if not for development purposes and these days the majority of developers use bundlers.

The static analysis compares apples to oranges and gives for granted every developer will implement properly that require function.

The moment you have to tell developers how to properly create such function is the moment @zenparsing is right.

Generally speaking, if we provide a feature because users need it, then we shouldn't also punish them for using it by intentionally making it difficult to use.

The import.meta.require is the simpler code mods / upgrade path, and there are no magic better errors handling in user land + feature detection is another myth because the word import cannot be used at all in non ES6+ code but import.meta.require can be easily found, replaced, patched both via bundlers or synchronous loaders.

TL;DR all @bmeck points are very subjective and some point does not solve any real-world use case while import.meta.require would.

[mcollina]

I've quite a few usage of require('module').createRequire(url) or even better require('module').createRequire(path) that are not ESM specific. This can help with unit testing and enable frameworks to create better isolation right now. Such a function enables a Node.js script to "see" what a script located in another folder could require. This enables a level of flexibility that is currently not possible or very hard to achieve.

Consider this code (full example at https://github.com/fastify/fastify/blob/master/test/internals/handleRequest.test.js):

const t = require('tap')
const test = t.test
const internals = require('../../lib/handleRequest')[Symbol.for('internals')]
const Request = require('../../lib/request')
const Reply = require('../../lib/reply')
const buildSchema = require('../../lib/validation').build
const Schemas = require('../../lib/schemas')

Would it be better if rewritten as:

const t = require('tap')
const test = t.test
const project = require('module').createRequire(new URL(`file://${__dirname}/../../`)

// this block can now be copy-pasted in various part of the tests, and it is independent of
// the location of the file itself
const internals = project('./lib/handleRequest')[Symbol.for('internals')]
const Request = project('./lib/request')
const Reply = project('./lib/reply')
const buildSchema = project('./lib/validation').build
const Schemas = project('./lib/schemas')

import.meta.require might be the way to go, but it's a more specific mechanism and one does not prohibit the other. I would love to have require('module').createRequire() in core right now.

[MylesBorins]

I agree with Matteo, the two fearures are not mutually exclusive.

I'd personally like to see makeRequire land as part of minimal kernel (potentially land on core soon). We can then dig into specifics of import meta require as part of a larger vision... We could also implement import.meta.require with makeRequire 😂

[WebReflection]

@mcollina this

const project = require('module').createRequire(new URL(`file://${__dirname}/../../`)

is basically

const project = source => require(`${__dirname}/../../${source}`);

which is even shorter to write.

That doesn't look like a compelling reason to block shipment of import.meta.require.

I am also not saying there are no use cases whatsoever for a createRequire, I am saying import.meta.require is what developers need/like and a better/easier way to move forward, IMO.

[mcollina]

@WebReflection require('module').createRequire() is aware of node_modules, while your example is not.

const project = require('module').createRequire(new URL(`file://${__dirname}/../../`)
const myLibrary = project('awesome-library') // is resolved from `file://${__dirname}/../../`

This could be very helpful in a lot of cases (CLI tools etc).

A similar capability is available in the ecosystem through https://github.com/sindresorhus/resolve-from which is downloaded 6 millions time per month, and it's currently based on undocumented functions: https://github.com/sindresorhus/resolve-from/blob/60cd04e69135b96b98b848fff719b1276a5610c0/index.js#L29.

IMHO we should just add this feature to core, as it will have the side benefit of reducing the usage of private functionality. I'd kindly ask @bmeck to send a PR to core for that if he is willing.

I am also not saying there are no use cases whatsoever for a createRequire, I am saying import.meta.require is what developers need/like and a better/easier way to move forward, IMO.

Have you got any evidence of that? How can we collect evidence?

[WebReflection]

Have you got any evidence of that? How can we collect evidence?

Currently every project behind bundlers use either import and let the bundler solve the kind or just require which works out of the box.

Everything a bundler or synchronous loader, needs to do with import.meta.require is to drop import.meta. and keep require, if it's targeting CJS and transforms all pseudo ESM into CJS, or do nothing if the env is ESM and the native import.meta.require is used.

This is a migration path that requires zero effort for users and tiny effort for bundlers.

Anything else that needs user-land solution won't be as simple, straight forward, and effective as import.meta.require and bundlers won't be able anymore to understand where and how files should be loaded because every logic to require CJS will be potentially different and harder to analyze/understand.

Goodbye tree shaking, code splitting, and ease of parsing if we push out a user-land based, potentially always different per project/company, module loader.

This is my logical evidence.

[zenparsing]

@bmeck Thanks for the counter-points, we'll definitely need to weigh them.

I'm not sure why we're discussing extending Node core here. Shouldn't that be out of scope?

The only other comment I'd like to make is that import.meta is designed specifically for this kind of module-context-specific stuff. Having users import a function from core and supply the context via a "url" is a bit like circling the house to go in the front door.

[bmeck]

@zenparsing all action here require extending core to support ESM. I consider import.meta.require an actively harmful direction if our goal is to allow code to be written for both browser and node consumption. I see no reason why new APIs would be problematic in order to support ESM. import.meta.require is a new API as well. Introducing things like a MIME parser would also be a new API.

If the counterpoint to all the problematic behavior listed above is solely that it requires adding an API, I see no validity to the counterpoint given that we are extending/implementing all sorts of new things already by exposing ESM.

[bmeck]

@WebReflection

Currently every project behind bundlers use either import and let the bundler solve the kind or just require which works out of the box.

I'm not sure this really relates as bundlers implement their own form of require instead of using the Node one, and they also convert import to their own bundling format. The discussion here is mostly tailored around how to let users write as much code in ESM that is unchanged when passed to both node and potentially other environments.

Everything a bundler or synchronous loader, needs to do with import.meta.require is to drop import.meta. and keep require, if it's targeting CJS and transforms all pseudo ESM into CJS, or do nothing if the env is ESM and the native import.meta.require is used.

This tailors the experience to the bundler, I am more concerned with allowing the user to be encouraged down a path of maximal compatibility than pleasing tool makers.

This is a migration path that requires zero effort for users and tiny effort for bundlers.

Bundlers could easy boilerplate out the bundle example I have above. This doesn't seem to discount that nor does it address that users must also migrate all code to import.meta.require instead of require, so there is migration effort on both. I have little concern for overall implementation effort on bundlers for createRequireFunction as there are plenty of ways to do bailouts and tree shaking without relying on import/export and using variable tracking of different kinds.

Anything else that needs user-land solution won't be as simple, straight forward, and effective as import.meta.require and bundlers won't be able anymore to understand where and how files should be loaded because every logic to require CJS will be potentially different and harder to analyze/understand.

I don't understand how this relates when bundlers could just alter output formats. This also seems to be a claim about CJS being hard to analyze that is applied to both import.meta.require and createRequireFunction. If this could be expanded upon and explained in a more concrete way, that would potentially show why this point is relevant.

Goodbye tree shaking, code splitting, and ease of parsing if we push out a user-land based, potentially always different per project/company, module loader.

This doesn't make sense.

This is my logical evidence.

I'd once again encourage you to not claim your own point as a conclusion at the end of your statements. It often creates both friction and confusion. In this case, mostly confusion on my part.

[WebReflection]

I'd once again encourage you to not claim your own point as a conclusion at the end of your statements.

At least I always state it makes sense and it's logical for me, I don't claim my personal point of view as the only truth out there, which is how I often interpret your statements.

Your post is subjective as much as mine, if you want, and specially the static analysis one is a deviation of how import.meta.require would shine instead of a in-house, per user, loader.

So, I guess we are both confused on each other statements, but if you want a concrete example on how I can statically find import.meta.require even via grep and never be able to understand anything meaningful via a createRequireLoader I'd be happy to write down that code.

[WebReflection]

P.S. it would be very valuable if bundlers related developers would say something about this too, i.e. @TheLarkInn

[michael-ciniawsky]

Bundlers mainly care about the CallExpression or keyword when parsing the code and apply their own handling and logic afterwards anyways. import.meta.require is likely simpler to parse here (on the edges, whatever the are) since createRequire needs to be tracked from the call to the import to know where it originally came from

[zenparsing]

@bmeck I just mean that we should try to resist the temptation to justify things one way or the other by supposing that we're solving problems that are out of scope.

As far as the general issue goes, my opinion is that the benefits of "createRequire" are pretty well non-existent considering that the "require" problem (CJS-in-the-browser) was solved a long time ago by bundlers.

Let's say that I want to create an ESM entry point for a CJS package with a default and named exports.

With createRequire it might look like this (assuming that createRequire can handle a URL):

// An ESM entry point for my CJS package
import { createRequire } from 'module';
const exports = createRequire(import.meta.url)('./index.js');
export default exports;
export const { a, b, c } = exports;

With import.meta.require it looks like this:

// An ESM entry point for my CJS package
const exports = import.meta.require('./index.js');
export default exports;
export const { a, b, c } = exports;

Either way, it's not that big of a deal. As a user, I would think the first is OK, but I would like the second better. Bundlers are going to have an easier time with import.meta.require.

For me, the scale seems to tip slightly toward import.meta.require. It's just really intuitive and "feels" right.

[WebReflection]

Thanks @michael-ciniawsky , that underlines my statement: import.meta.require is zero effort for users (no in-house loaders, no best practice or error handling done right to teach) and little effort for bundlers (at least littler).

[guybedford]

Imagine teaching someone new to Node.js how to use modules in Node. They're writing an ES module, and want to run it in both the browser and Node.js, and they are loading a third-party package, which they've installed from npm, but that also available on a CDN for the browser.

How would you explain import.meta.require to them? How would you explain to them how they can run this ES module in the browser? Does that feel intuitive to this user? Or is it just because we've been writing require for so many years?

[guybedford]

@domenic createRequire as discussed here is not a context-dependent import, it is a factory function that would be passed import.meta.url, within the bounds of the module system semantics.

[WebReflection]

just pointing out that others favorited import.meta.require before and reasons look always the same:
https://gist.github.com/ceejbot/b49f8789b2ab6b09548ccb72813a1054#gistcomment-2330869

[devsnek]

how about some loader hooks https://gist.github.com/devsnek/cd88c66d7beb36ea6f74694a609fb40a

[guybedford]

@WebReflection we are all well aware that import.meta.require is a nice pattern for Node integration in isolation, the concerns are browser and tooling compatibility (as well as the inability for node bins to work cross platform).

[zenparsing]

@domenic Thanks (and apologies if you now get pinged on this thread!)

@devsnek

We'll have the same problem with any of these variations, I think. If getFilePathFromURL is given a non-file URL it would have to throw or something, and I imagine that as @bmeck suggested, import.meta.dirname would have to throw if it's not a file-system module.

I suppose the principle at play here is that properties on import.meta are less obviously non-portable. The counter-argument is that import.meta is by definition non-portable. From that point of view, the fact that we have overlap in terms of "url" is somewhat misleading.

[devsnek]

@zenparsing loader assumes you're just on the filesystem because this behaviour anywhere else is objectively terrible, which is why i don't want it by default in node. A future where everyone wraps accessing properties from import.meta in try/catch because we couldn't design a proper system of integration sounds terrible to me.

My process for designing our implementation is figuring out 1) defaults that work in all cases and then 2) how to allow people to override those defaults for their own specific needs as painlessly as possible.

[zenparsing]

A future where everyone wraps accessing properties from import.meta in try/catch because we couldn't design a proper system of integration sounds terrible to me.

Don't users have to do the same thing with getPathFromURL? I suppose that by requiring the indirection from "url" we would discourage (via friction and annoyance) non-portable code. I guess I don't follow how that helps anything though. If I need to get a dirname, I'm going to get it, whether Node makes me write one line or two or five. Either way, I'm writing non-portable code.

There's another tool for discouraging certain patterns: linters. Linters can warn if you're using import.meta.dirname in a portable codebase. Developers have to use that strategy for globals and things already.

[michael-ciniawsky]

import.meta.node ?

const { require, __dirname, __filename } = import.meta.node

// or (depending on individual feature usage and use cases)
const require = import.meta.require
const { __filename, __dirname } = import.meta.node

[ljharb]

Linters can also warn against usage of any object properties, any global variables, and any imports or requires - import.meta.url just makes the job of detection slightly easier for the < 100 (< 1000?) people that will ever be writing bundler and linter code directly.

[devsnek]

@michael-ciniawsky we already own the import.meta object, we can put whatever we want on it. again this isn't a problem of putting environment specific properties on import.meta.

[guybedford]

It is possible to write an implementation of url that works in the browser, so that code that does import { getPathFromFileURL } from 'url'; import fs from 'fs'; fs.readFile(getPathFromFileURL(import.meta.url)) could be portable using a form of virtual fs.

[devsnek]

@guybedford browserify (and by extension webpack) both use npmjs.org/url to polyfill in browser. as long as that module is updated, i don't see why that wouldn't work.

[michael-ciniawsky]

What is the concrete problem then ?

Why is something like getPathFromFileURL needed ?

import { getPathFromFileURL } from 'url'; 
import fs from 'fs'; 

fs.readFile(getPathFromFileURL(import.meta.url))

Why not

import fs from 'fs'

const { pathname } = new URL(import.meta.url)

fs.readFile(pathname + 'file.ext', cb)

What is the difference ?

[devsnek]

@michael-ciniawsky https://github.com/nodejs/node/pull/22506/files#diff-dd182d238def46a3af0d112f617e8838R883

[michael-ciniawsky]

🙄

const url = import.meta.url

const file = path.from(url)

@devsnek thx

[jasnell]

The difference is that fs.readFile(pathname + 'file.ext', cb) would not produce a valid path across platforms. File URL pathname values are very different from filesystem paths and must be translated in a platform specific way in order to be used.

[michael-ciniawsky]

Can't the various node API's which currently accept a path either be restricted from URL usage (where appropiated) or nomalize (File) URL's themselves instead of having users to deal with that in (clunky ways) within userland ?

🤢

import url from 'url'
import path from 'path'

const p = path.from(import.meta.url)
const p = url.getPathFromFileURL(import.meta.url)
// etc etc...

const file = p + 'file.ext'

fs.readFile(file, cb)

😃

const file = url || path + 'file.ext'

fs.readFile(file, cb)

But I think this should ideally be a separate topic and issue, I'm for e.g currently confused why && where File URL usage is even needed, since e.g the fs example in particular would be node only anyways and something in the vein of const { __dirname, __filename } = import.meta.x should be sufficient (?) 🤷‍♂️. What are other (popular) use cases for node API's where URL usage would be needed/beneficial (for portable code)

[jasnell]

They already do, to an extent. You can pass a File URL to fs.readFile() and it will convert it for you. What you have to be careful about, however, is passing just the pathname segment or using concatenation to modify the input. You have to pass in File URL object itself in order for it to work.

[jasnell]

So if import.meta.url returns a URL object, you could just do fs.readFile(import.meta.url) .... If it returns a string you would do fs.readFile(new URL(import.meta.url))

[devsnek]

and for the record, import.meta.url must be a string. URL objects are mutable.

https://url.spec.whatwg.org/#url-apis-elsewhere

[SMotaal]

@bmeck I think from an interop standpoint, filename and dirname are not file-system specific, they could be URL derived outside of node with roughly fileURL.replace(/^.*[/]([^/?#]*?)(?:[?#].*|)$/, '$1') and new URL('.', fileURL).pathname not withstanding which of RegExp or URL would be the better here

Ultimately, the fallback is undefined, but then the question is do we expect people to want to do something like 'filename' in meta (so undefined still counts in this logic) or just const {filename = 'fallback'} = meta (I use meta to avoid suggesting endorsement at this point).

[MylesBorins]

Explicitly adding blocked to make it clear there is not an expectation for this to land right now, but rather this is a archive and documentation of the work on import.meta.require so it doesn't need to be done again in the future.

Open to closing the PR if people think that is a better appraoch

[SMotaal]

@MylesBorins Can you clarify why "blocked" makes its was to this PR? Is that the direct result of module: add createRequireFunction method?

Can you help me connect the dots please.

[MylesBorins]

@SMotaal as we have agreed to focus on the minimal kernel there is no reason to land this right now. I don't think it makes sense to close it and lose the work. As such the blocked label clearly communicates there is not a current intent to merge this

edit: worth being explicit that I don't think blocked is a bad thing. Better to be explicit

[SMotaal]

Awesome… makes sense.

…

On Sep 18, 2018, at 5:01 PM, Myles Borins ***@***.***> wrote: @SMotaal <https://github.com/SMotaal> as we have agreed to focus on the minimal kernel there is no reason to land this right now. I don't think it makes sense to close it and lose the work. As such the blocked label clearly communicates there is not a current intent to merge this — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAz2za6NwV3V9r6vBywFQ5H6sbL6hWdPks5ucV9EgaJpZM4WIhtS>.

[MylesBorins]

This is not likely landing anytime soon, we can revisit another time