Feature request: Utility functions for the steps of resolution and loading

[jonaskello]

Today it is possible to alter the resolve algorithm to some extent using --experimental-specifier-resolution. However that switch only supports a couple of fixed use-cases and is not available programmatically within a loader implementation. When implementing a custom loader it would be nice to have access to a fine grained API so we can re-use parts of the default resolve algorithm but still be able to interject whatever logic you want in between steps, or skip/replace steps, without having to re-implement the full algorithm. That way a custom loader can be composed from the API rather than have to re-implement the whole resolve algorithm.

The old node resolve algorithm was quite easy to re-implement so for that we did not really need a fine grained API. The new algorithm supports more features and is therefore more work to re-implement. So IMHO it would make sense to expose an API for the steps in this more complex algorithm rather than having custom loaders re-implement or copy it in full.

The use-cases for this would be for example implementing a loader for .ts files in ts-node where the algorithm is the same but only differ in file extension. Also, Yarn PnP could use this for its resolve. See below for those requests.

---

ORIGINAL POST (before the issue was re-formed as a feature request):

I'm sorry if this is not the right forum to ask this, if not pls let me know where it is appropriate to ask.

I'm curious about how the loader specification and --experimental-specifier-resolution interact. Specifically I'm looking into loading ES modules directly from typescript .ts files using any specifier (bare, relative). With the PR "esm: consolidate ESM Loader methods #37468" merged, this seems to have gotten easier. From what I understand it would now be possible to have something like this:

import { foo } from "./foo.ts"

And then just implement the loader hook as the resolve hook will not throw on the unknown .ts extension. This seems like a nice improvement as we don't have to re-implement the full resolve anymore. However most typescript projects are setup to import without extensions, like this:

import { foo } from "./foo"

I believe having this resolved is related to --experimental-specifier-resolution but AFAIK this only support .js. So in order to resolve the above to foo.ts the only way I see today is to re-implement the whole module resolution algorithm, which with the new spec is a rather big task. I think that is what ts-node is doing here, basically copying the impl from the internal nodejs loader.

I'm guessing the --experimental-specifier-resolution=node flag somehow augments the internal loader to check for .js and index.js (I have not looked deep into the source code though). The point being I think it would be nice if a loader could be allowed to somehow do the same augmentation. Specifically allowing imports without extension to be resolved to .ts files without having to re-implement the full resolution algorithm. Or is there some other way to achieve this?

[GeoffreyBooth]

I’m guessing the --experimental-specifier-resolution=node flag somehow augments the internal loader to check for .js and index.js (I have not looked deep into the source code though). The point being I think it would be nice if a loader could be allowed to somehow do the same augmentation. Specifically allowing imports without extension to be resolved to .ts files without having to re-implement the full resolution algorithm. Or is there some other way to achieve this?

So to paraphrase, basically what you’re asking for is a hook within Node’s resolution algorithm, to inject custom logic to add .ts to the list of inferred suffixes, which under --experimental-specifier-resolution=node is .js, .json, .node, /index.js, /index.json, /index.node. (In the explicit/ESM-default version of the resolution algorithm, this inference doesn’t happen.) Is this more or less correct?

I understand why you’d want this, because the full resolution algorithm is daunting to reimplement. But I think that’s exactly what popular tools like ts-node are already doing, often depending on a library resolve that provides exactly that.

In the near term, I think that’s the path you’ll need to take; we have enough on our plate at the moment trying to complete the loaders roadmap, and providing utility functions for things that can be (and already are) implemented in the ecosystem should naturally be the lowest priority. That’s not to say that such utilities aren’t worth providing in Node core; we already provide require.resolve, and an ESM equivalent (or multiple equivalents, to perhaps break up the parts of the resolution algorithm) might be a request you can get the core collaborators to agree to support. But it wouldn’t happen soon, unless you want to take the initiative and make the PRs yourself. If that’s something that interests you, we can use this repo to discuss what the various functions should be and their APIs, so that you have an approved design before you start coding anything.

Here are some references to catch you up on related discussions that have happened so far (deep linking to my own comments to spare you from reading the entirety of these long threads, and because I think I summarized some consensuses in those comments):

• esm: unflag import.meta.resolve node#33464 (comment)
• Special treatment for package.json resolution and exports? node#33460 (comment)
• add API for "package dir" node#49445

Also, have you tried simply checking for existence of `${specifier}.ts`? What edge cases would be missed by a loader like the following?

// This code block is untested

import { access, constants } from 'node:fs/promises';

export async function resolve(specifier, context, defaultResolve) {
  // For specifier './file', see if './file.ts' exists
  try {
	const inferredSpecifier = new URL(`${specifier}.ts`, context.parentURL);
    await access(inferredSpecifier.pathname, constants.R_OK);
    return { url: inferredSpecifier.href };
  } catch {
    return defaultResolve(specifier, context, defaultResolve);
  }
}

[cspotcode]

But I think that’s exactly what popular tools like ts-node are already doing, often depending on a library resolve that provides exactly that.

I want to clarify for anyone else reading this: ts-node is not technically reimplementing node's resolver.

We ship a copy of node's source code, with a minimal patch applied to essentially expose the API being described here. Ideally node would provide this API so we don't need to ship a copy of logic that is already within node's binary.

This is not to say I disagree about priorities or time constraints, but there's also the risk that this use-case becomes harder than it needs to be because it's not considered in earlier loader design work.

What edge cases would be missed by a loader like the following?

I think it'll miss stuff like when package.json "main" points to ./index which should get ./index.ts

[arcanis]

Ideally node would provide this API so we don't need to ship a copy of logic that is already within node's binary.

To support this with another example, Yarn PnP as well doesn't want to replace the full resolver. We only want to inject into a very limited part of the resolution (package name to directory), and it's the way Node currently works that requires us to copy a (much) larger part of the Node logic.

This isn't helped with Node's internals sometimes relying on hidden bindings that not even loaders have officially access to (cf nodejs/node#39513).

[jonaskello]

@GeoffreyBooth Thanks for the reply! :-). Yes you are correct that is what I'm looking for. I currently can see two main ways it could be implemented. Either as options to the default loader or as a separate API. Your example would work for relative specifiers but not for bare specifiers? For bare specifiers currently you need to re-implement the full ESM algorithm?

To give an example of what I mean as options to default loader it could be something like this (which probably is not a good idea):

export async function resolve(specifier, context, createDefaultResolve) {
  const defaultResolve = createDefaultResolve({ extensions: [".ts", ".js"]});
  return defaultResolve(specifier, context, createDefaultResolve);
}

EDIT: To clarify, the above is intended as a programmatic and more flexible version of --experimental-specifier-resolution.

However, I think an API would be more useful so you can compose your own resolve from that without having to fully re-implement all of it. I did not fully follow if the discussions you linked are about such an API or are they about something else? I guess I need to read the full threads after all :-).

providing utility functions for things that can be (and already are) implemented in the ecosystem should naturally be the lowest priority.

I totally understand about the priorities. It is sure possible to re-implement the algorithm so perhaps I will do that but just wanted to check if there was an easier way. I'm not aware of the ESM algorithm being implemented in the ecosystem yet? There is an issue on the resolve repo but that seems to not have gotten far yet:

browserify/resolve#222

As mentioned above ts-node is maintaining a full copy of the whole ESM resolver from node's code with some tweaks (to solve what I mentioned above).

[cspotcode]

For what it's worth, the relevant files are in raw and dist-raw. The former are copy-pasted from node, kept there merely for convenience when using the diff command. The latter are patched and distributed.

https://github.com/TypeStrong/ts-node/tree/main/raw
https://github.com/TypeStrong/ts-node/tree/main/dist-raw

For example, you can diff ./raw/node-esm-resolve-implementation-v15.3.0.js ./dist-raw/node-esm-resolve-implementation.js to see what we changed.

[JakobJingleheimer]

At some point recently, we did discuss exposing defaultGetFormat and defaultGetSource in the load hook's arguments (instead of defaultLoad).

Or do you mean you'd want it exposed from some builtin package, like Node's module builtin?

[GeoffreyBooth]

My first instinct is to create publicly accessible versions of many of the functions in https://github.com/nodejs/node/blob/master/lib/internal/modules/esm/resolve.js, which I guess is more or less what https://github.com/TypeStrong/ts-node/tree/main/raw?rgh-link-date=2021-09-12T22%3A14%3A08Z is doing. This would correspond with the steps in the algorithm, listed at https://nodejs.org/api/esm.html#esm_resolution_algorithm. So then within your resolve hook, you could call the same functions that Node’s internal defaultResolve calls, but choose to replace one of them (or inject extra logic between some of them).

[jonaskello]

Yes this is exactly what I meant. Having a fine grained API with functions that does a little part each so you can compose your own loader from that and interject whatever logic you want in between, or skip/replace steps, without having to re-implement the full algorithm.

[GeoffreyBooth]

Yes this is exactly what I meant. Having a fine grained API with functions that does a little part each so you can compose your own loader from that and interject whatever logic you want in between, or skip/replace steps, without having to re-implement the full algorithm.

Okay, perhaps we should rename the issue (and reword the initial post)? And this can be a feature request. Once you can compose your own resolve and override only the pieces you want, there won’t be a need for --experimental-specifier-resolution anymore since it would be trivial to make a custom loader that does what --experimental-specifier-resolution does. (Or does what it does, plus an extra inferred extension, in your case.)

[JakobJingleheimer]

I think there might be 2 separate issues here, the second being:

import { foo } from "./foo.ts"

And then just implement the loader hook as the resolve hook will not throw on the unknown .ts extension. However most typescript projects are setup to import without extensions, like this:

import { foo } from "./foo"

You'll need a custom resolve hook to return a url because Node's defaultResolve will throw. But that's trivial to write (there's an example for CoffeeScript in the landed and soon-to-be-released update to the ESM docs).

(I realise, Andrew, you are quite familiar with TypeScript; including extra detail for others)

In both examples you will suffer the same problem forced by the TypeScript team. The former is the correct usage; however, the latter was the lesser-of-two-evils (as well as a popular corner-cut) at the time as a workaround to avoid the TS team's decision to disallow TypeScript file extensions in import specifiers.

For the first example, that will work just fine because it's factually accurate (the file being read literally does contain the extension .ts in its name); however, if that itself appears in a typescript file, you'd need to replace the .ts file extension after the file/source is read but before it is passed to tsc to be transformed (otherwise, tsc will self-destruct). This is likely the easiest avenue because it leverages the most built-in behaviour of the various tools involved. I think this might be enough to get both sides to do what they're supposed to do.

I think relying on --experimental-specifier-resolution is not the way to go (for a number of reasons, most obvious that it is "experimental").

---

My first instinct is to create publicly accessible versions of many of the [resolve utility] functions

I think this is a promising avenue. We've talked about doing that for at least one of them already; @cspotcode @jonaskello do you need all of them or just a subset?

[cspotcode]

Another couple cases that I need to support:

• Determining format of foo.ts when the only file on disk is foo.ts
• Determining format of foo.ts when both foo.js and foo.ts exist

(In this example, the user has followed my recommendation and their code is import { foo } from './foo.js';, and our loader has resolved it to /foo.ts)

When we need to getFormat any .ts file, we can replace the extension with .js and ask defaultGetFormat how it will classify a .js file in the same location. Effectively, .ts follows the same rules as .js. In the example above, we call defaultGetFormat('/foo.js') without risking a filesystem error or needlessly loading foo.js from disk.

Ideally, I imagine we could do something like this pseudocode:

import {createGetFormat} from 'node:esm-resolver-helpers';
const defaultGetFormat = createGetFormat({extensionsWhichObeyPackageJsonType: ['.js', '.ts', '.tsx']});
defaultGetFormat('/foo.ts'); // reads /package.json and classifies based on "type"

[cspotcode]

We've talked about doing that for at least one of them already; @cspotcode @jonaskello do you need all of them or just a subset?

I will have to think about this.

We need defaultGetFormat, either with pluggable extensions as described above, or we can get away with using today's extension-replacement hack.

We also need resolve, but it will not be useful to us if we cannot customize its behavior. We can make a diff of our changes to highlight which behaviors need to be customized. One change: we added .js -> .ts resolution, which is a bit different than --experimental-specifier-resolution.

[GeoffreyBooth]

We need defaultGetFormat, either with pluggable extensions as described above, or we can get away with using today’s extension-replacement hack.

See the updated CoffeeScript loader example: https://github.com/nodejs/node/blob/master/doc/api/esm.md#transpiler-loader. We added a new helper function that’s probably a better solution in general than the extension-replacement hack. I think there’s a good argument for this helper being part of core, since if it’s in core it could reference the cached package.json files rather than needing to crawl the filesystem for files that the Node process has already loaded. (This is also much more generic and flexible than something particular to file extensions.)