JS source should not be wrapped.

[hashseed]

When Node.js executes JavaScript, it internally wraps them into an IIFE:

Module.wrap = function(script) {
  return Module.wrapper[0] + script + Module.wrapper[1];
};

Module.wrapper = [
  '(function (exports, require, module, __filename, __dirname) { ',
  '\n});'
];

There are some problems with this:

1. Invalid JavaScript is parsed as correct. This following example executes just fine:

});

(function() {
  console.log(1);
})();

(function() {

2. This wrapper shows up as part of source in the DevTools. Since the start line and column are passed as 0, the source positions of errors and break points at least for the first line of the script are off:

$ cat test.js
throw new Error();

$ node test.js
/usr/local/google/home/yangguo/node/test.js:1
(function (exports, require, module, __filename, __dirname) { throw new Error();
                                                              ^
Error
    at Object.<anonymous> (/usr/local/google/home/yangguo/node/test.js:1:69)
    at Module._compile (module.js:573:30)
    at Object.Module._extensions..js (module.js:584:10)
    at Module.load (module.js:507:32)
    at tryModuleLoad (module.js:470:12)
    at Function.Module._load (module.js:462:3)
    at Function.Module.runMain (module.js:609:10)
    at startup (bootstrap_node.js:158:16)
    at bootstrap_node.js:578:3

3. This disparity between actual source and source being passed to V8 also affects code coverage. See https://bugs.chromium.org/p/v8/issues/detail?id=7119
   Since Node.js reports 0 for start line and column, V8 has no way to adjust the source offsets reported for coverage.

The better way to do it might be to use ``v8::ScriptCompiler::CompileFunctionInContext`, where we ask V8 to wrap the source in a function context. See examples here.

cc @schuay @bcoe

[fhinkel]

/cc @nodejs/v8

[MylesBorins]

@hashseed is it safe to assume that V8 function would allow us to continue to inject all the same lexically scoped variables?

Would there be a way to call this function via a macro in js land to avoid having to juggle things between the layers? Can you imagine any reason this would cause execution to be different? My biggest concern right now would be how this would affect current ecosystem tools that might be doing magic to account for the wrapper

My gut is that this may be semver major, and potentially break some stuff in the ecosystem... If my gut is wrong this could potentially be semver patch, which would be quite exciting

[bnoordhuis]

The better way to do it might be to use ``v8::ScriptCompiler::CompileFunctionInContext`, where we ask V8 to wrap the source in a function context.

I don't remember the details but it didn't quite work last time I tried it: #9273 (comment)

[hashseed]

Hehe... yeah now that I looked closer v8::ScriptCompiler::CompileFunctionInContext suffers from the same issue. It also just wraps the source at some point.

Looks like I'll have to fix this in V8 first.

@MylesBorins my hope is that it would semantically not break anything.

[benjamingr]

We need to be careful to not break existing code - returning in a module for instance. My gut feeling is that this breaks a lot of code in the wild doing something like:

if(!process.env.SOME_CONFIG) {
    module.exports = "bar";
    return;
}

causeEffects();
module.exports = "foo";

[devsnek]

i feel like we should treat that like we treat

});
some code
(function() {

and just consider it unsupported behavior, as users shouldn't depend on implementation details

[isaacs]

Breaking return in a commonjs module context will cause significant havoc.

A V8 api that preserves current behavior without an iife wrap would be nice, though. What about using new Function()? Does that have the same problems?

[devsnek]

wouldn't new Function() just become function anonymous(some args) { some body } and have the same problem?

[benjamingr]

From what I understand the suggested CompileNewFunctionInContext would still be a function and would work - I was pointing out return in case different solutions are suggested.

(idea: would dropping the \n in Module.wrapper fix the line numbering?)

[isaacs]

@devsnek I just tested, and yes, it does. What's worse, the offset may change between V8 versions, unlike the known iife wrapper Node uses.

@benjamingr I think the same problem applies to CompileNewFunctionInContext. It still adds wrapper code, but then it's wrapper code we can't safely account for.

My suggestion is to leave the CommonJS system frozen, and instead focus on making things right with the es-module system. Whatever shortcomings Node.js's CommonJS implementation has (and it has several), they haven't prevented a vibrant ecosystem from using them to build things, and fixing those shortcomings is almost certainly going to break that ecosystem in profound and hard-to-predict ways.

[isaacs]

(idea: would dropping the \n in Module.wrapper fix the line numbering?)

Only in cases where the error happens on the last line of the script because something wasn't terminated. For example:

$ cat end.js
if ('foo') {

$ node end.js
/Users/isaacs/dev/js/ayo/end.js:3
});
 ^

SyntaxError: Unexpected token )

But otherwise, the only issue is the column offset on line 1.

[jdalton]

⚠️ From a user-land view I want to be very careful about changes to Node CJS wrapping. If there is an API change (should try to avoid) it should still be user-land accessible since it does aid in dev related packages. ⚠️

Update:

Just noticed @isaacs' comment:

My suggestion is to leave the CommonJS system frozen, and instead focus on making things right with the es-module system. Whatever shortcomings Node.js's CommonJS implementation has (and it has several), they haven't prevented a vibrant ecosystem from using them to build things, and fixing those shortcomings is almost certainly going to break that ecosystem in profound and hard-to-predict ways.

💯 ++

[devsnek]

+1 to freezing commonjs system, anything to keep people moving to esm is good

[bcoe]

@jdalton @isaacs @hashseed locking the existing approach for CJS seems workable, as long as the new ESM system provides appropriate context in the scriptParsed event of the debugger. Here's what the meta information looks like currently when you parse a module in v8:

{
  "method": "Debugger.scriptParsed",
  "params": {
    "scriptId": "72",
    "url": "/Users/benjamincoe/bcoe/c8/test/fixtures/b.js",
    "startLine": 0,
    "startColumn": 0,
    "endLine": 12,
    "endColumn": 3,
    "executionContextId": 1,
    "hash": "1A47AD409B904C96B5BC49D6A58187D4B2D1C2C4",
    "isLiveEdit": false,
    "sourceMapURL": "",
    "hasSourceURL": false,
    "isModule": false,
    "length": 171,
    "stackTrace": {
      "callFrames": [
        {
          "functionName": "createScript",
          "scriptId": "41",
          "url": "vm.js",
          "lineNumber": 79,
          "columnNumber": 9
        }
      ]
    }
  }
}

My expectation would be that anything that comes through the new module system would have isModule = true and would not have wonky offsets?

Another thing that I think would be nice would be to expose the fact that we have a 62 byte prefix in a variable, e.g., require.cjsPrefix? This would allow tooling to avoid the 62 byte magic #.

[jdalton]

@bcoe

Another thing that I think would be nice would be to expose the fact that we have a 62 byte prefix in a variable, e.g., require.cjsPrefix?

Can be gotten from require("module").wrapper[0] or module.constructor.wrapper[0].

[ofrobots]

Or

export const MODULE_WRAP_PREFIX_LENGTH =
    require('module').wrap('☃').indexOf('☃');

from here.

[bcoe]

It looks like we are setting isModule to true 👍

https://github.com/nodejs/node/blob/master/src/module_wrap.cc#L105

So, I think the only work that would shake out of this is ensuring that ES modules don't have the same wrapper offset issue.

[hashseed]

I guess for the particular use case of code coverage we can get the wrapper header offset directly and subtract it from the coverage offsets. Might still be nice to move towards hiding this implementation detail though, without changing the general semantics of the function scope.

[bmeck]

We should probably check if any existing popular projects are using this: https://github.com/search?l=JavaScript&p=2&q=%22module.wrapper%22+%22require%28.module.%29%22&type=Code&utf8=%E2%9C%93 , a quick look didn't show anything terrible

[jdalton]

a quick look didn't show anything terrible

It shows usage.
(I've used it on more than one occasion too.)

What's wrong with @isaacs' stance?

[bmeck]

Nothing, I am not +/- to it, just interested that no statistics were being brought to the table. None of the uses seemed to be abusing the feature for Ill gotten gains while also being popular enough that we need to replicate such a feature for ESM.

[jdalton]

Okay. FWIW I don't think anyone was suggesting the CJS wrapper APIs be added to ESM, just suggesting they not be removed from CJS if it can be avoided.

[schuay]

I think the same problem applies to CompileNewFunctionInContext. It still adds wrapper code, but then it's wrapper code we can't safely account for.

Currently, yes. But we're discussing possible solutions where the source string would remain unmodified and the wrapper would be created only as AST nodes by the parser. That'd avoid any complications around weird line/column offsets, and we'd have better verification that the given script is actually valid JS.

[bnoordhuis]

^ That seems strictly superior to me to what we currently have. I'm +1 on switching over.

[jdalton]

@schuay

Currently, yes. But we're discussing possible solutions where the source string would remain unmodified and the wrapper would be created only as AST nodes by the parser. That'd avoid any complications around weird line/column offsets, and we'd have better verification that the given script is actually valid JS.

That's pretty cool. It looks like something that could be exposed by vm and used by Module._compile without changing the surrounding APIs (Module.wrap, Module.wrapper, etc.) which is great!

[bmeck]

@jdalton I don't think this would be passed the entire function string, just the body of the function. Which wouldn't be able to be compatible with .wrap/.wrapper since it would return a function instead of a string and would not be accepting the function prefix/suffix.

[jdalton]

@bmeck

I don't think this would be passed the entire function string, just the body of the function. Which wouldn't be able to be compatible with .wrap/.wrapper

Oh, _{ok never mind then} 😢

Exposing it to vm would still be rad on its own though (but that's kinda off topic)

[hashseed]

I landed the patch to v8::ScriptCompiler::CompileFunctionInContext upstream: https://chromium.googlesource.com/v8/v8.git/+/1586f37f2d1f06415dbaf0c03b0f6e0702450ce9

[hashseed]

@bnoordhuis the stack trace line/column issue should be fixed now. The current API does not support code caching, but it would still be interesting to see whether it works for require. It should be already be a bit faster than using wrappers, and I have some ideas to make it faster still.

[hashseed]

In reply to concerns above: return still works.