module: ESM loader approach

[JakobJingleheimer]

There are 2 leading approach proposals for ESM loaders and chaining them.

Similarities

Both approaches:

• consolidate hooks into
  • resolve(): finding the source (equivalent to the current experimental resolve()); returns {Object}
    • format?: see esm → getFormat
    • url: same as current (see esm → resolve)
  • load(): supplying the source (a combination of the current experimental getFormat(), getSource(), and transformSource()); returns {Object}
    • format: same as current (see esm → getFormat)
    • source: same as current (see esm → getSource)
• Hooks of the same kind (resolve and load) are chained:
  1. all resolve()s are executed (resolve1, resolve2, …resolveN)
  2. all load()s are executed (load1, load2, …loadN)

Differences

Next()

This approach is originally detailed in #36396.

Hooks are called in reverse order (last first): a hook's 3rd argument would be a next() function, which is a reference to the previous loader hook. Ex there are 3 loaders: unpkg, http-to-https, and cache-buster (cache-buster is the final loader in the chain):

cache-buster invokes http-to-https, which in turn invokes unpkg (which itself invokes Node's default):
cache-buster ← http-to-https ← unpkg ← Node's default

The user must actively connect the chain or it (likely) fails: If a hook does not call next, "the loader short-circuits the chain and no further loaders are called".

Done()

This approach was also proposed in #36396 (in this comment).

The guiding principle of this approach is principal of least knowledge.

Hooks are called in the order they're declared/listed, and the return of the previous is fed as the input of the subsequent/next hook, and each hook is called automatically (unless short-circuited):

unpkg → http-to-https → cache-buster (if none of the supplied loaders output valid values, node's default loader/hook is invoked, enabling a hook to potentially handle only part and avoid re-implementing the native functionality node already provides via the default hook).

Hooks have a done argument, used in rare circumstances to short-circuit the chain.

Additionally, this proposal includes a polymorphic return:

hook returns	continue?	result	scenario
done(validValue)	no	use validValue as final value (skipping any remaining loaders)	module mocking (automated tests)
false	no	skip file (use empty string for value?)	file is not needed in current circumstances
invalid value	no	throw and abort instead (current behaviour)	user error
nullish	yes	loader did nothing: continue to next	loader isn't for the file type
valid value	yes	pass value to next loader	expected use

Examples

--loader https-loader \
--loader mock-loader \
--loader coffee-loader \
--loader other-loader

Resulting in https-loader being invoked first, mock-loader second, etc, and node's internal defaultLoader last.

For illustrative purposes, I've separated resolve and load hooks into different code blocks, but they would actually appear in the same module IRL.

Resolve hook chain

HTTPS Loader

const httpProtocols = new Set([
  'http:',
  'https:',
]);

/**
 * @param {Object} interimResult The result from the previous loader
 * (if any previous loader returned anything).
 * @param {string} [interimResult.format='']
 * @param {string} [interimResult.url='']
 * @param {string} context.originalSpecifier The original value of the import
 * specifier
 * @param {string?} context.parentUrl
 * @param {function} defaultResolver The built-in Node.js resolver (handles
 * built-in modules like `fs`, npm packages, etc)
 * @param {function(finalResult?)} done A short-circuit function to break the
 * resolve hook chain
 * @returns {false|{format?: string, url: string}?} If participating, the hook
 * resolves with a `url` and optionally a `format`
 */
export async function resolve(
  interimResult,
  // context,
  // defaultResolver,
  // done,
) {
  let url;

  try {
    url = new URL(interimResult.url);

    // there is a protocol and it's not one this loader supports: step aside.
    if (!httpProtocols.has(url.protocol)) return;
  }
  catch (err) {
    // specifier does not meet conditions for this loader; step aside.
    if (!determineWhetherShouldHandle(interimResult)) return;
  }

  return {
    url: '…',
  };
}

Mock Loader

export async function resolve(
  interimResult,
  context,
  defaultResolver,
  // done,
) {
  let url;

  try { url = new URL(interimResult.url) }
  catch (err) { url = new URL(defaultResolver(interimResult /* , … */).url) }

  url.searchParams.set('__quibble', generation);

  return {
    url: url.toString(),
  };
}

Load hook chain

HTTPS Loader

const contentTypeToFormat = new Map([
  ['text/coffeescript',		'coffeescript'],
  ['application/node',		'commonjs'],
  ['application/vnd.node.node',	'commonjs'],
  ['application/javascript',	'javascript'],
  ['text/javascript',		'javascript'],
  ['application/json',		'json'],
  // …
]);

/**
 * @param {Object} interimResult The result from the previous loader (if any
 * previous loader returned anything)
 * @param {string} [interimResult.format=''] Potentially a transient value. If
 * the resolve chain settled with a `format`, that is the initial value here.
 * @param {string|ArrayBufferView|TypedArray} [interimResult.source='']
 * @param {Object} context
 * @param {Array} context.conditions
 * @param {string?} context.parentUrl
 * @param {string} context.resolvedUrl The module's resolved url (as
 * determined by the resolve hook chain).
 * @param {Function} defaultLoader The built-in Node.js loader (handles file
 * and data URLs).
 * @param {Function} done A terminating function to break the load hook chain;
 * done accepts a single argument, which is used for the final result of the
 * load hook chain.
 */
export async function load(
  interimResult,
  { resolvedUrl },
  // defaultLoader,
  // done,
) {
  if (interimResult.source) return; // step aside (content already retrieved)

  const url = new URL(resolvedUrl);

  if (!httpProtocols.has(url.protocol)) return; // step aside

  const result = await new Promise((res, rej) => {
    get(resolvedUrl, (rsp) => {
      const format = contentTypeToFormat.get(rsp.headers['content-type']);
      let source = '';

      rsp.on('data', (chunk) => source += chunk);
      rsp.on('end', () => res({
        format,
        source,
      }));
      rsp.on('error', (err) => rej(err));
    });
  });

  return result;
}

Mock Loader

export async function load(
  interimResult,
  { resolvedUrl },
  defaultLoader,
  // done,
) {
  const isQuibbly = (new URL(resolvedUrl)).searchParams.get('__quibble');

  if (!isQuibbly) return;

  const mock = defaultLoader(urlToMock); // or some runtime-supplied mock

  return { source: mock };
}

CoffeeScript Loader

const exts = new Set([
  '.coffee',
  '.coffee.md',
  '.litcoffee',
]);

export async function load(
  interimResult,
  context,
  defaultLoader,
  // done,
) {
  if (
    !!interimResult.format
    && interimResult.format !== 'coffeescript'
  ) return; // step aside

  const ext = extname(context.resolvedUrl);

  if (!exts.has(ext)) return; // step aside

  const rawSource = interimResult.source || defaultLoader(
    {
      format: 'coffeescript', // defaultLoader currently doesn't actually care
    },
    context
  ).source;
  const transformedSource = coffee.compile(rawSource.toString(), {
    whateverOptionSpecifies: 'module'
  });

  return {
    format: 'module',
    source: transformedSource,
  };
}

Updates to ESMLoader.load()

class ESMLoader {
  async load(resolvedUrl, moduleContext, resolvedFormat = '') {
    const context = {
      ...moduleContext,
      resolvedUrl,
    }
    let shortCircuited = false; // should we support calling done with no arg?
    let finalResult;
    let format = resolvedFormat;
    let source = '';

    function done(result) {
      finalResult = result;
      shortCircuited = true;
    }

    for (let i = 0, count = this.loaders.length; i < count; i++) {
      const tmpResult = await loader(
        { format, source },
        context,
        defaultLoader,
        done,
      );

      if (shortCircuited) break;

      if (tmpResult == null) continue; // loader opted out

      if (tmpResult === false) {
        finalResult = { source: '' };
        break;
      }

      if (tmpResult?.format != null) format = tmpResult.format;
      if (tmpResult?.source != null) source = tmpResult.source;
    }

    finalResult ??= interimResult;

    // various existing result checks and error throwing
  }
}

Concerns Raised

Next()

1. This creates an Inception-like pattern, which could confuse users: loaders would be specified in a different sequence than called, as loaders are called in a nested manner: the final loader calls the previous, and the previous calls its previous, etc all the way back to the beginning.
2. The next function does not behave as many current, well-known implementations behave (ex javascript's native generator's next is the inverse order to this's, and not calling ExpressJS's route-handler's next does not break the chain).
3. Requires the user to have specific knowledge: next is effectively required (not calling next will likely lead to adverse/undesirable behaviour, and in many cases, break in very confusing ways).
4. Unit testing is more difficult (requiring spying in almost all cases, whereas done's needs spying very rarely, by, likely, more advanced users)

Done()

1. This could potentially cause issue for APMs (does the next approach also?) After chatting with @bengl, it seems like this is not an issue as V8 exposes what they need.
2. A hook that unintentionally does not return / returns nullish might be difficult to track down I believe this was resolved in the previous issue discussion?

[JakobJingleheimer]

@GeoffreyBooth I feel rather vindicated RE Concerns with Next() #3: When I was setting up the resolve hook test fixture for #37468, I indeed did forget the default-case return next(…) at the end. Took me a few minutes to figure out why the heck suddenly all my cases were failing.

I think if we opt for the next() approach (as I was authoring #37468, I'm increasingly leading toward "shouldn't", not least of which because the implementation looks to be much more difficult—read: buggy, real or perceived), I think some extra fault detection will be needed. Without, I think this could be a table-flip moment for users (I was getting annoyed, and I wrote the darn thing—so if anybody should know/remember, it would be me (and you)).

[GeoffreyBooth]

This is worth discussing, sure, but I didn’t think #37468 was supposed to implement chaining; I thought it was just meant to combine the hooks and stop there, and a later PR would add chaining. Isn’t this something to figure out after #37468 has merged in?

[JakobJingleheimer]

Yes #37468 isn't/doesn't, and yes that's all #37468 does. #37468 is ready to go, so I think this is next?

[giltayar]

One point of concern about the Done method: when implementing a loader for mocking (as I did for testdouble's ESM support), you need to add cache busting to the final url, which means that you want to be the last in the chain. And yet, when loading the source, you want to be the first in the chain, because you know you're not really transforming the code, but rather replacing it.

This is why I prefer the Next() approach. While it means that loader developers need to be more aware of how the chaining works, it gives them more flexibility in how they implement the resolve and load functions of the loader, and does not "chain" them to the opinionated thinking of how things should happen, which is what the Done approach is trying to do.

Exhibit A 😊, the ESM mock loader for testdouble: https://github.com/testdouble/quibble/blob/main/lib/quibble.mjs

[JakobJingleheimer]

@giltayar sorry, I'm not following (but would like to understand your concern). done() would ensure it's the last hook executed. If you mean out of all possible hooks, all other hooks need to run and a particular one dynamically needs be last in queue, yes, next() would facilitate that and done() would not. Needing to do that dynamically is a critical differentiator but it sounds like very much an edge-case (please do correct me if I'm misunderstanding you).

Regarding needing the particular resolve hook's sibling load hook to be called first, both done() and next() could do that because ESMLoader will be aware of what file they both came from. But why do you need that?

I looked through your code example (I've looked thru Quibble previously—slick stuff!), but the "why" didn't jump out at me.

At first read, it sounds like what you're describing meticulously strong-arms a process that would probably naturally result in what you want anyway.

If this is an edge-case, unless it's a very compelling edge-case, I would likely lean towards the option that does not ensure all developers have a much more complicated and gotcha-prone experience so that a small subset might use a small bit of extra functionality.