Support registering worker specific custom ESM loader hooks (Regression from #52706)

[alan-agius4]

What is the problem this feature will solve?

Following the implementation of #52706 in Node.js version 22.2, workers can no longer specify custom ESM loader hooks. In the Angular CLI, we utilized this feature as part of our prerendering (SSG) pipeline.

Here’s a summary of our build pipeline and process: In the main process, the application is transpiled and bundled, generating all files in memory. The main process then spawns multiple thread workers (using Piscina) with a customized loader hook to intercept imports and redirect them to the in-memory files when present. As of version 22.2 this is no longer possible.

It's worth noting that in our scenario, multiple threads (e.g., Thread A and Thread B) may be initialized, resulting in varying module resolutions and ESM module hooks across these threads.

In the following example, the workerData contains the files and modules utilized by the ESM loader hooks:

const { Worker } = require("worker_threads");

const filesInMemoryInItalian = {
  'main.mjs': 'export const hello = "ciao"';
};

const filesInMemoryInEnglish = {
  'main.mjs': 'export const hello = "hello"';
};

const workerA = new Worker("./lib/child.js", {
  workerData: { files: filesInMemoryInItalian },
  execArgv: ["--import", "./lib/register.js"],
});

const workerB = new Worker("./lib/child.js", {
  workerData: { files: filesInMemoryInEnglish },
  execArgv: ["--import", "./lib/register.js"],
});

What is the feature you are proposing to solve the problem?

Thread workers should be permitted to specify custom ESM loader hooks.

What alternatives have you considered?

An alternative approach is to spawn a forked process from the main process, specifying the custom loader hook in the forked process. This forked process then spawns the worker threads. As demonstrated in angular/angular-cli#27721.

However, It is important to note that this approach is not backward compatible. In previous versions, the ESM loader specified on the main thread did not propagate to child workers. Consequently, from a maintenance perspective, tools now need two methods to use ESM custom loaders with thread workers and need to support Node.js LTS and Active versions.

[ShogunPanda]

Hello!
Me and @dygabo are working on two possible solutions to this problem, which basically fix the regression introduced in #52706.
Will keep you posted.

[GeoffreyBooth]

@nodejs/loaders

[GeoffreyBooth]

@alan-agius4 Thanks for this. I’m a little confused by your example, because it has execArgv: ["--import", "./lib/register.js"] sent into both workers. Are you intending to have the same hooks apply to all workers?

Aside from the bug introduced by #52706, the intended design is that hooks registered on the main thread would apply to all worker threads automatically. From your example, I gather that this is the API you want, but somehow you get the differing context of each thread available to the hook? So like I see the differing workerData passed into each thread; would you want that data available within the hook, like on the context property for example?

export async function load(url, context, next) {
  const { files } = context.workerData // Either Italian or English, depending on the thread of the module
  // ...

[alan-agius4]

I’m a little confused by your example, because it has execArgv: ["--import", "./lib/register.js"] sent into both workers. Are you intending to have the same hooks apply to all workers?

Yes, we intend to use the same hook logic but with different contexts/data. Before version 22.2, this was feasible as each thread independently registered its own hook with different workerData. The API you described would suit our needs. Currently, we have something similar for older Node.js versions:

register.js

register('./loader-hooks.js', {
  parentURL: pathToFileURL(__filename),
  data: workerData,
});

loader-hooks.js

let outputFiles;

export function initialize(data) {
  outputFiles = data.files;
}

export function resolve(specifier, context, nextResolve) {
  // resolve logic here
}

app.js

const workerA = new Worker("./lib/child.js", {
  workerData: { files: filesInMemoryInItalian },
  execArgv: ["--import", "./register.js"],
});

const workerB = new Worker("./lib/child.js", {
  workerData: { files: filesInMemoryInEnglish },
  execArgv: ["--import", "./register.js"],
});

the intended design is that hooks registered on the main thread would apply to all worker threads automatically

I understand this and it's acceptable. However, I'm unsure if it's a bug or not, but with version 22.2, you can opt-out of this behavior by setting execArgv: [] in the Worker constructor. For example, the following would force the worker not to use the main process loader hooks:

const workerB = new Worker("./lib/child.js", {
  execArgv: [],
});

Questions

• Will loaders registered in thread workers eventually apply to the main process? I believe they shouldn't.
• Would a hook registered in a thread worker also apply to sibling workers? For instance, in the provided scenario above, if workerB doesn't specify execArgv: ["--import", "./register.js"], would it inherit hooks from the other worker that were created previously even though no hooks are registered in the main process?

[GeoffreyBooth]

• Will loaders registered in thread workers eventually apply to the main process? I believe they shouldn’t.

I don't know; we haven't decided yet.

• Would a hook registered in a thread worker also apply to sibling workers?

If we decide that calling register from a worker thread adds hooks to the one hooks thread that all threads share, then yes.

I’m unsure if it’s a bug or not, but with version 22.2, you can opt-out of this behavior by setting execArgv: [] in the Worker constructor.

This is essentially the problem introduced by #52706. It wasn’t on our radar that hooks could be applied to (or removed from) worker threads via execArgv; that was unintentional. Currently (and pre-22.2) calling register from a worker thread is a no-op. It shouldn’t be; it should either error, register hooks to apply to the entire process, or somehow register hooks that apply just to that thread. But we were planning on tackling that as a follow-up after landing #52706.

Right now we just want to unbreak people so that we can take some time to think about how to handle hooks for worker threads. The off-thread hooks code is already monstrously complex, so I worry about the maintainability of something so specific as “if register is called from a worker thread, the registered hooks apply only to that thread.” I understand the usefulness of such an ability, but we’re already up against the limits (if not beyond them) of what the Node team can achieve and support in this area. If there’s a simpler solution like the context.workerData idea that can also satisfy the use case, I think we need to pursue the simpler options.

Something else to consider is, what are you using worker threads for here? It seems like their purpose is a way to group sets of modules to be customized in a particular way. You probably don’t have easy access to edit the source code of the files you’re loading, but if you did, you could use import attributes for that purpose: import './file.js' with { translations: 'italian' }. The import attributes from the import statement/expression are already provided in the context argument sent into each hook. Another option would be to put the group marker into the URL somehow, like as a prefix or a query string param: import './file.js?translations=italian'. And so on.

Another option is that we have an old PR for a deregister method. You could call register with a particular set of hooks and data to pass into initialize, then import some modules you want customized by these hooks and settings, and then call deregister to tear it down.

Maybe none of these options are preferable to the isolation that worker threads provide, but my point is just to try to define the problem in a way that doesn’t prescribe a particular solution. As in, I assume that the threads are an implementation detail to whatever you’re ultimately trying to achieve. Maybe they are the best solution, better than child processes or import attributes or query strings or other ideas, but for the purposes of evaluating approaches it would help to define the goal as naively as possible when it comes to how it should be achieved.

[alan-agius4]

I can see that the deregister function can be useful and we could work as in our case we could create the workers for each locale sequentially (Which we already do). However, we don't have easy access to modify the source file because, in our use case, the same source file is transpiled and localized into numerous locales as part of the build pipeline. I've created a diagram to help you better understand the workflow.

[mcollina]

I think there are four cases we need to discuss:

1. one thread has loaders registered, all worker threads created by it should reuse the loader thread
2. one thread does not have loaders registered, all worker threads created by it will have their own loader thread
3. independently of all the status of the running thread, a pool of the worker threads created should use the same loader.
4. one thread has loaders registered, a worker thread created by it wants to have the same loaders of the main thread.

I think that in order to support those cases, we need to have an option in the Worker class.

[mcollina]

@alan-agius4 based on your diagram, it would be advantageous for you to have one loader thread for all WorkerPoolEN and one for WorkerPoolIT. Is it correct?

[alan-agius4]

@mcollina, correct.