Make traceur (maybe all traceur-compiled modules) loadable via node 'require'

[arv]

Original author: usrbi...@yahoo.com (February 26, 2013 22:10:37)

I believe it might not be much harder to do this for all modules, so
I've tentatively extended the scope of the bug.

For those just joining: Currently traceur uses an 'eval' hack to
load into node. This loses line numbers in stack traces, and some
flexibility.

Proof-of-concept (way out-of-date currently):

Make bin/runtime.js and bin/traceur.js
run natively in both node and browser.
https://codereview.appspot.com/6745048/

The goal in a nutshell is (as I see it)

$ cat H.js
module H {
export function h(n) {
return 'hello ' + n;
}
}
$ ./traceur --out H.out.js H.js
$ node
> var runtime = require('bin/traceur.js');
> traceur.runtime.setupGlobals(global); // See below.
> var H = require('./H.out.js').H;
> H.h('world');
'hello world'
>
$ cat H.out.html
<script src="runtime.js"></script>
<script src="H.out.js"></script>
<script>console.log(H.h('world'));</script>
$ my-browser H.out.html

Caveats:

Because modules can't affect the loading code's environment, you
need an extra step to install the polyfills for running compiled
traceur code in the current context:

var traceur = require('bin/traceur.js');
traceur.runtime.setupGlobals(global);

There may be other unforseen problems, obviously, so maybe have this
as an optional output format, not default.

Original issue: http://code.google.com/p/traceur-compiler/issues/detail?id=206

[arv]

From arv@chromium.org on February 27, 2013 17:27:43
One thing to add is that modules are about to change. In a nutshell:

module "a/b/c" { ... }

is a named module that is always available under that name, no matter where it is required from.

module "./a/b/c" { ... }

is a relative module and it is resolved like a URL (also "../a/b/c", "/a/b/c", "http://a/b/c")

So when you do import {x} from "a/b/c" it will always get the named module but import {x} from "./a/b/c" will resolve the name based on the current module. (nested modules are not allowed any more)

---

With that in mind, I think a more reasonable approach is to do something like:

module 'abc' {
export var x = 42;
}

// compile

<script src="compiled.js"></script>

var moduleInstance = traceur.loader.get('abc');
var x = moduleInstance.x;
</script>

---

Node.js allows us to hook a transpiler step into the loader. I played around with this before but it broke down on our compilation for external modules being async. Node.js needs sync require.

[arv]

From usrbi...@yahoo.com on February 27, 2013 19:18:40
Just to make things clear, these are no longer legal?

// Illegal. Needs to be:
// module 'M1' { ... }
module M1 { ... }
// Illegal. M2M is a nested module.
module M2 {
export module M2M { ... }
}
// Illegal. M3M is still a nested module.
module M3 {
import M3M from 'M3M.js';
export M3M;
}

And for loading:

// file.js
module file { ... }

// file.html

<script src="file.out.js"></script> <script> // Undefined reference. 'file' no longer magically appears upon // load via script tag. file.func(); </script>

---

With the current implementation, name conflicts are handled
silently, with the last declaration winning (normal JS behavior).

With the new style (reminds me of node or python), does this become
a runtime error?

For node or python, there is a search path for named modules,
eventually searching the importing file's local directory. Is this
similar to how harmony modules is going to be? Or are the directory
separators essentially meaningless semantically?

// So this never means 'c.js' in subdir 'b' in subdir 'a'?
module "a/b/c" { ... }

This seems to run counter to both node and python, which is not to
say it's bad, but that it's counter to common and familiar usage. If
there really is a compelling reason, then it's good, but otherwise,
familiar seems better.

---

Didn't know about the node transpiler hooks. Also have not really
tried traceur's async module loading, though kind of vaguely aware
of it from working on those files.

I'm not sure how invisible we can realistically expect the traceur
layer to be. Currently, I was thinking of explicitly doing 'require'
or direct module access depending on the environment.

But it seems a lot more complicated to do
import 'M' as M;
and have it do the right thing across all environments. And node is
actually 3 environments: direct node, node via 'require', and node
via repl. (Okay, some of the differences might not matter. I haven't
thought this completely through.)

[arv]

From arv@chromium.org on February 28, 2013 17:14:02
I got some details wrong "a/b/c" is resolved relative to the current module loader, not relative to the current file.

[arv]

From usrbi...@yahoo.com on February 28, 2013 22:14:00
The "relative to the loader" makes more sense. Started
wading through harmony:modules.

[some reading...]
http://wiki.ecmascript.org/doku.php?id=harmony:module_loaders

if there is not currently an entry in the module instance
table for its canonical URL, the module must be fetched.

So it looks like

module 'a/b/c' { ... }

probably creates an "entry" in the "module instance table".
But if it doesn't exist, then the Loader prepends its own
baseURL and fetches it.

Being able to specify module pathnames directly is
definitely different from node and python, but kind of makes
sense in the world of CDNs with potentially arbitrary URLs.

---

Since '/be' takes a lot of inspiration from python, it's not
surprising we see a global 'System' object taking the role
of a central registry for modules.

If JS did nothing more than copy python's module system, I
don't see how you could go too wrong, at least in node-style
environments.

On the web, your CDN may go down, and then you need
fail-over, and maybe checksum or signature verification of
your loaded modules just to make sure that no one has
hijacked the backup CDN.

But then again, I guess HTTPS takes care of that --
theoretically. But more of a pain to set up, so not many
do it. And for CDNs, doesn't HTTPS preclude caching?

Maybe I'm being paranoid there. But even if not everybody
verifies to that level, the option to do so should (maybe
is; haven't read the wiki enough to fully understand)
probably be available.

[arv]

From edy.b...@gmail.com on March 27, 2013 13:23:25
Would it be possible and/or spec-conforming to load the modules imported ahead-of-time, using require (instead of loading and inlineing the file), output destructuring code in case import * is used (or make a frozen object with the properties found), and generate a require call (the second call will used the module cached from the ahead-of-time call)?

That would allow things like "module fs from 'fs';" or "import * from 'http';" or "import Command from 'commander';" to work nicely.

It's possible to wrap around the .js module extension handler (I doubt you would want a different extension), to compile with traceur.

[arv]

From usrbi...@yahoo.com on March 27, 2013 17:20:44
There are two possibilities:

• Compile one file that will work unmodified in both browser and node.

  This obviously wouldn't work for native node modules, but should work
  with traceur-compiled modules.

• Have a "node" output target that outputs node modules and turns
  imports into 'require'.

I'd prefer the first possibility if it's possible, because it makes
traceur simpler to use. The output files would Just Work (TM) anywhere.

For source-map, we just wrapped it and used it like a normal traceur
module. That works fine for a few libraries, but isn't scalable.

---

The modules spec requires a lot of things, including a 'System' object
to play the role of 'sys' in Python, and a Loader interface. So a
wrapper around node that uses that interface would probably be
conforming.

This is the best general solution for node. Have a node-specific
Loader. Note that we haven't implemented System yet, and I'm not sure
how spec-conforming our Loader objects are.

The spec is in flux, and I'm not good at being a spec lawyer, so don't
take this as definitive.