Add API for dumping or inspecting the consolidated CJS and EJS module graph

[ggoodman]

What is the problem this feature will solve?

With the #51977 having landed, we're moving to a world where the logical graph will cross freely to and from CJS and ESM. I anticipate that this will break some stuff in unpredictable and hard-to-debug ways given the long tail of wacky stuff that has made its way into NPM.

These hard-to-debug and hard-to-anticipate situations will be difficult to detect and inspect from user-land code. The module graph is private state held by the runtime and V8. Having a way to extract a read-only view of this will allow tools and users to inspect, share and fix these situations.

What is the feature you are proposing to solve the problem?

I propose an API that would allow the caller to obtain a read-only representation of the module graph across both CJS and ESM nodes.

As a straw man, let's imagine require('node:module').getModuleGraph(). It could provide data with providing the following:

1. The fully-qualified module ID (maybe it's URL).
2. The type of that module (ESM vs CommonJS).
3. That module's loaded dependencies, where we see the following for each:
   • The reference type require, import_dynamic, import, .. anything else?
   • The module specifier for that dependency.
   • The fully-qualified module ID of the resolved dependency.

It might look like this:

const graph = [
  {
    id: 'file:///path/to/a.cjs',
    module_type: `CommonJS',
    dependencies: [
      {
        specifier: './b',
        reference_type: 'require',
        id: 'file:///path/to/b.cjs',
      },
    ],
  },
  {
    id: 'file:///path/to/b.cjs',
    module_type: `CommonJS',
    dependencies: [
      {
        specifier: './c',
        reference_type: 'require',
        id: 'file:///path/to/c.esm',
      },
    ],
  },
  {
    id: 'file:///path/to/c.esm',
    module_type: `ESM',
    dependencies: [
      {
        specifier: './c',
        reference_type: 'import',
        id: 'file:///path/to/d.esm',
      },
    ],
  },
  {
    id: 'file:///path/to/d.esm',
    module_type: `ESM',
    dependencies: [],
  },
];

Side-note questions:

• Is it possible through some convoluted means for the same on-disk module to have both a CJS and ESM in-memory representation? If so, that probably changes the way modules need to be identified in the proposed schema above.

What alternatives have you considered?

N/A. This is uncharted territory! 🚀

[WebReflection]

I welcome and approve this proposal, for what it matters, I don’t welcome the type suggestion … it’s not super hard to have a consistent naming convention, and we have a super simple Boolean like value of the type field of a module, mandatory to define one a module or a commonjs. Hence I think if this would ever land as helper for anything introspecting, the type name should be either lowercase module or commonjs, so that nobody ever and no tool needs to mistake the module’s author intent.

thank you.

[WebReflection]

amend: by mandatory I mean in the module case, commonjs can be the OR default if module is not matched.

[joyeecheung]

Is it possible through some convoluted means for the same on-disk module to have both a CJS and ESM in-memory representation? If so, that probably changes the way modules need to be identified in the proposed schema above.

It kind of depends on what you mean. For example, imported CJS have both a Module entry in Module._cache and a ModuleWrap facade, but the two are essentially connected together (via closures, for now). We are able to tell the facade is a facade because it's a SyntheticModule, not a SourceTextModule. I suppose in the serialization, we can ignore the facade for imported CJS, similar to how we can ignore the Module entry of required ESM, these wrappers shouldn't be a problem.

[joyeecheung]

The reference type require, import_dynamic, import, .. anything else?

I think with the current data structure it is probably only possible to provide require and import. However it should also be possible to provide more detailed information if this is kept behind a flag - it could be quite expensive to track all these by default. But it would be less of a big deal if users opt into the memory & performance overhead of module tracking with a flag.

Also I guess this isn't necessarily tied to require(esm) and can be implemented on its own, because import cjs or createRequire(import.meta.url)(cjs) already happen, and can already give you unpredictable bugs...

[ggoodman]

I think the important bit of the feature request is setting the stage for introspection or debugging. So from that angle, I think we'd already be adding a lot of value even if the we limit things to require and import.

Also the reason I was asking about whether a source file could have separate ESM and CommonJS representations is that it would mean that the module identifier alone would be insufficient to uniquely identify that node. If that's the case, the entries in the dependencies[] array would also need a field identifying the module format of the target.

@WebReflection as for naming things, I have no strong opinions but wanted to get an idea that could be easily understood out there.

[joyeecheung]

Also the reason I was asking about whether a source file could have separate ESM and CommonJS representations is that it would mean that the module identifier alone would be insufficient to uniquely identify that node.

That is not supposed to happen. Though I think there's always the possibilities of bugs (both the module loaders are heavily undertested but it seems the world just have been living with them okay...).

[WebReflection]

"when in Rome" I think the type should reflect the parsing goal and its source:

• module for ESM only affairs
• commonjs for CJS only affairs
• require-esm (or commonjs-module) for CJS trying hard to have ESM
• commonjs-import for CJS explicitly opting into the ESM world via dynamic import
• module-import for ESM requiring dynamic import for reasons
• module-require (or module-commonjs) for ESM trying hard to have CJS

I don’t think exist other use cases, but I agree just module or commonjs would already be a great starting point.

edit in case it wasn’t clear, these types are not for the importer, rather for the imported module as graph … to showcase how modules get imported or bootstrapped.
In this case maybe it makes sense to disambiguate between the module type itself that’s being imported, and the importing type, although the module type can be easily inferred: all types except commonjs and module-require mean the module is ESM, not CJS

[joyeecheung]

I think it would make more sense to put the the type of reference in the edges, not in the nodes. So a.cjs (commonjs) ->(require) -> b.mjs (module) or a.cjs (commonjs) -> (import) -> b.mjs (module) can both be represented without duplication. The type name can just follow the "format" enum we have internally for the ESM "translators" (or an extended version of it, because the addons are currently represented by the commonjs format internally).

[ggoodman]

I think it would make more sense to put the the type of reference in the edges, not in the nodes.

Yesterday I updated the example data structure to differentiate top-level and dependency-level type variables. I think that the suggested approach encodes what you're suggesting, @WebReflection. Instead of encoding every combination of dependent type, edge type, dependency type as a different enum string, a consumer of this API would be able to do so from the normalized form.

I've taken a stab at encoding it as a type definition:

type ModuleReferenceImport = "import";
type ModuleReferenceRequire = "require";
type ModuleReferenceKind = ModuleReferenceImport | ModuleReferenceRequire;

interface ModuleReference {
  readonly reference_type: ModuleReferenceKind;
  readonly specifier: string;
  readonly resolved_url: string;
  // I think we probably also need this since two otherwise similar imports
  // could have different behavior based on the import atributes.
  // This could also be a `ReadonlyMap`.
  readonly import_attributes?: Readonly<ImportAttributes>;
}

type ModuleTypeESM = "module";
type ModuleTypeCommonJS = "commonjs";
type ModuleTypeJSON = "json";
type ModuleTypeAddon = "addon";
type ModuleTypeBuiltin = "builtin";
type ModuleType =
  | ModuleTypeESM
  | ModuleTypeCommonJS
  | ModuleTypeJSON
  | ModuleTypeAddon
  | ModuleTypeBuiltin;

interface Module {
  readonly module_type: ModuleType;
  readonly url: string;
  readonly module_requests: ReadonlyArray<ModuleReference>;
}

[joyeecheung]

The type definition looks great.

Some modification I'd suggest:

1. id on LoadedModule could just be url. That's also internally how we key this.
2. module_type can also include "json" "addon" and "builtin"
   • While it may be possible to include "wasm" too, we don't always get to control WASM compilation. It is possible to hook into V8 to record WASM compilation done via e.g. WebAssembly.compile(), but I suspect it would take quite a bit of overhead to find out about the referrer when it's compiled using the global API.
3. dependencies can be module_requests, per spec terminology.
4. Somewhat pedantic: Loaded in the name LoadedModule may not be quite necessary - this may be used to dump the graph while a module itself is still being loaded.

[ggoodman]

Thanks @joyeecheung I'll edit to get the terminology aligned.

[joyeecheung]

Somewhat related: with #52219 it should be possible to just implement this in the user land. There might still be value to have some sort of "standardized" format in core. But the universal hooks allows creating a prototype in the user land.