Code Splitting

[sokra]

GWT has a cool feature named Code Splitting which loads parts of the code on demand. This is also appliable for webmake.

Here my idea:

// a.js:
var b = require("b");
// ...
bindEventHandlerForSomeRarEvent(function() {
  // the magic keyword "process.nextTick", which behaves normally in node.js
  // but so some special thing in webmake
  process.nextTick(function(err) {
    // Some code loading err handler
    // err is undefined in node.js
    if(err) { /* ... */ }
    var c = require("c"); // really big libary, which is only needed in this rar case
    // ...
  });
});

webmake core would provide the magic process.nextTick function to the compiled code.
process.nextTick do this in the compiled code:

• store the callback function
• adds a script tag to the document
• the script contains JSONP
• callback function (JSONP) adds some new modules to the context
• and calls the stored callback function.
• in case of an error (ex. timeout) the stored callback function is called with some kind of error parameter

Here is a quick hint on the compiled files:

// output.js
( /* core code */
({
  "a": function(/* ... */, process) {
     // a.js:
     var b = require("b");
     // ...
     bindEventHandlerForSomeRarEvent(function() {
       // the magic keyword, which behaves normally in node.js
       // but so some special thing in webmake
       process.nextTick(function(err) {
         // Some code loading err handler
         // err is undefined in node.js
         if(err) { /* ... */ }
         var c = require("c"); // really big libary, 
               // which is only needed in this rar case
         // ...
       });
     });
  },
  "b": function /* content of b.js */
  // here is no "c": !!
  // which saves bandwidth

// may be named "a.output.js":
webmake_magic_jsonp_function( // the jsonp callback function
"a", // origin of code loading (used for identifing the stored function)
{
  "c": function(/* ... */, process) {
    // the content of the big libary
  },
  "d": function(/* ... */, process) {
    // d.js is here a dependency of c.js (optional)
  }
}

This feature may makes the compile process a big complexer.

It may be useful for:

• loading parts of your webapp when accessing them
• loading polyfills only when necessary
• decrease startup time

[medikoo]

Code splitting is indeed most missing major feature and I plan to add it with next big update.
However in first place I imagine it just as a possibility of creating few different bundles that reuse same modules which are loaded once with one of the bundles, that's it.
I don't think it's good idea to change semantics of process.nextTick for client-side for that. If we would really need some function to do async modules bundle load on client side it should be specific to client-side and be used in code written specifically for client-side.

[Phoscur]

Isn't setTimeout(fn, 0) the browser replacement for process.nextTick(fn) ?

[Phoscur]

var c = require("c"); // really big library, which is only needed in this rar case

Maybe this is the problem, you should not include a big library if you don't really need it.

I'm fine with having a little more startup time and then a more fluent interface. I would like to have an option in the opposite direction, so that module code can be preexecuted to be ready when it is required, but that's probably not worth the effort.

This does not mean that I don't like the idea of sending code later to the client, I see usecases for that. But it's not to reduce startup time. It would be nice to be able to rollout new features without a whole app refresh.

[sokra]

@medikoo
i don't think this changes the semantics of process.nextTick. process.nextTick executes some code in the near future in node.js. It you implement Code Splitting as i propossed it has the same semantics. It executes some code in the near future. There is a code loading in between on client side, but this is transparent to the user.

@Phoscur yes setTimeout(fn, 0) is the replacement, but i don't want to use a existing function (on client side) because this could impact exisiting code. I think process.nextTick is the best choice because it has nearly no impact on node.js. The only impact is that the code is executed later. This semantic stays the same on client side (Ok, the delay is bigger on client side)

Consider a big webapp with several features. The features are in subpages, but because you do not want a whole page reload you have all features in one codebase.
Without Code Splitting this has a negative impact on startup time because you compiled your codebase with webmake into a big js file.
To optimize, you want to split your code and you only want to load if you navigate into the subpage. So insert process.nextTick somewhere in your code (i should be easy in ansynchronous functions). Now your compiled codebase will be splitted into pieces and only the first piece is loaded on webapp startup. This decreases startup time in compare to webmake without this feature.

This feature is optional because you don't need to use process.nextTick, but if you decide to use it it is transparent to the programmer.

[Phoscur]

Why wouldn't you want all features in one codebase?
You can pack about 300k codelines into one megabyte minified scriptfile, how big is your webapp going to get? I think the overhead to request a few bytes when you need them is bigger than a little slower loading time in the beginning.
Keep in mind that the code is only loaded and parsed in the beginning, it is lazy-executed when it's needed.

Creating some shims for node specific functions like process.nextTick is not a bad idea, we can probably steal those from somewhere else anyways.

[medikoo]

@sokra I would definitely not mess with process.nextTick it's Node specific and was not intended for asynchrounous background loading.
Anyway as I mentioned, soon we'll have the way to create many separate bundles that share modules and load them on demand. It can be helpful when you have really big application with clearly separated functionalities.

[sokra]

I reimplemented webmake from scratch to reduces some limitations of it and add code splitting.
It is just a start and was made in a 2 day hacking session.

It do need some additional love but the most things are working really good.

See https://github.com/sokra/modules-webpack for some docs
and https://github.com/sokra/modules-webpack/tree/master/example for a example

Here is a list of differences to the original webmake:

Module and file names are replaced with a unique id per file
=> different versions of the same module do not overlap anymore
=> no need for resolving on client side => smaller

It is allowed to require outside to current package scope

Browser replacements
web_modules is preferred to node_modules
xxx.web.js is preferred to xxx.js if require("./xxx") without extension

Code splitting with require.ensure
as in http://wiki.commonjs.org/wiki/Modules/Async/A
see links above

[medikoo]

@sokra thanks for input.

Your approach is very interesting, but you introduced big limitation with that. You didn't take into account specific cases when modules are not required directly.
In code you may pass generated string to require or run require functions indirectly. Such require calls of course cannot be picked statically but in webmake there's include option through which you can force to include such modules, so there's back door to deal with that, in your fork I guess it's not possible.

In projects I currently work with this is the way templates are loaded, indirectly by template engine, I wouldn't be able to use your solution there.

Other goal, when working on webmake, was not to change source code of modules, it's not that major concern as one above, but still it's certainly better for development and debugging to keep module names and paths intact.

If you decide to continue development of your fork please change it's name, as you pointed it's very different solution.

Thanks!

[sokra]

@medikoo thanks for your feedback

The limitation, you pointed out, is a problem, but i think a "back door" can be added with some config options. Do you have a example of a use case? Can you post it?

The changes in the source code are really small so i don't think that this is a big problem.

The point of posting it here is a aspect of code splitting which may can used in your webmake (also without the other changes). It's the same approach as in the inital post, but with require.ensure instead of process.nextTick. (As you argued correctly process.nextTick is more a node specific thing)

I'll rename it if you want so. It was orginally intended as proof of concept for this issue.

[medikoo]

@sokra I'm not sure if any back door is available in your solution. Code can only be scanned statically, but all requires should work programmatically, so e.g. you should provide some sort of back door, so following could work:

var indirect = require;
indirect(pathGeneratedByFn());

When you do all paths resolution on server side, and for client you don't provide any mechanism to do real path resolution then it's no place for back door.

For example here https://github.com/medikoo/domjs/blob/master/lib/domjs.js#L92 I call require indirectly.

[sokra]

Hmm... this really does not seem to work. This stays a limitation.

What do think about the code splitting?

[medikoo]

To have code splitting, in first place we need to add possibility of creating different bundles that share same modules, form that point you can upload such bundles on demand in many different ways.

By example, code of additional bundle that we may need to load, will probably look like:

require.push({
  // extra modules
})
("invoke/one/of/the/added/modules"); // optional

Where require is global variable exposed by bundle that was initially loaded.

I'm still not sure do any load on demand functionality should be included in webmake, or should it be left to external tools how to handle that. We'll figure it out in near future.

[sokra]

I've moved it to https://github.com/webpack/webpack

Hope that is ok...

[medikoo]

That's ok, thanks