Bake-in support for webpack DLLs #1201

amireh · 2016-06-18T22:00:29Z

This patch bases on top of the happypack one although it's not technically related (only in domain) to squeeze out some more build-time improvement. I saw the question of applying DLLs to this project asked a lot so I thought I'd contribute an answer.

Here's a table that shows the build-time gains from using DLLs first, then HappyPack + DLLs:

Elapsed (ms)	Happy	Happy Cache	Using DLLs
9907	No	No	No
7370			Yes
6314	Yes		No
4164			Yes
3119		Yes

The patch may be missing a few things (the ones I know are outlined in the commit message), so feel free to rip it out yourself if you need to as it's very unlikely I'll find the time to address anything around this.

What I tested and found to be functional (you would definitely be the better judge here though):

npm run build
npm run dev
npm start
./node_modules/.bin/webpack --config webpack/dev.config.js

The DLL is allowed in the development build only, and the code gives room to define more DLLs if you guys want later on.

Thanks.

Enables parallel loader application for jsx, json, sass, and less sources using five background threads. Some env notes: - you can disable HappyPack entirely by setting HAPPY=0 in your env - you can disable HappyPack's caching by setting HAPPY_CACHE=0 - you can make HappyPack more verbose by setting HAPPY_VERBOSE=1 (useful to see the state of cache) - HappyPack's state and cache can be reset by `rm -r .happypack/` Other minor changes: - made those loaders only apply to files under `/src` (a whitelist)

One can now benefit from more build-time speed-ups by pre-compiling vendor libraries into a DLL. To compile the DLL: npm run build-dlls To use the DLL: export WEBPACK_DLLS=1 npm run dev What isn't done: - cache busting - webpack isomorphic tools integration (if it's needed anyway, I don't use it/know it)

oyeanuj · 2016-06-18T22:29:41Z

@amireh This looks great, looking forward to integrating it!

I was curious if you ever came across this attempt at DLLs in another boilerplate, and your thoughts on their approach -

react-boilerplate/react-boilerplate#495

amireh · 2016-06-18T22:39:19Z

I haven't actually, thanks for linking. The method applied here was adapted from what we did at work, and it's been working fine for us for months now (but we do not use DLLs in production for a few reasons so I didn't go into it here either but you may want to explore that avenue.)

Btw, the benchmarks above were made on my main box, but I just tried it out on an old macbook and build time went down from 14271ms to 4823ms-5560ms which is actually really nice. Hardly a second or two worth of difference even though the difference in hardware is vast.

oyeanuj · 2016-06-18T23:21:18Z

Thanks for the perspective. One follow-up:

but we do not use DLLs in production for a few reasons

Could you elaborate more on those reasons so that I could inform my decision as well?

amireh · 2016-06-19T00:18:59Z

Hmm, well the only point that would've broken the output is that the DLL contained (for convenience) all our vendor modules, while in production we use several chunks and some libraries are loaded only in certain contexts (aka on-demand), so we'd either have had to maintain multiple versions of the DLL (based on the environment) or cause users to download code they wouldn't run. Obviously, the latter is not an option.

Maintaining multiple versions of the same DLL seems silly simply because the tool is really a development tool - there's no gain that I know of from using a DLL vs using a chunk (at run-time) so it's just useless work.

There's also the extra request overhead...

Just some things I can remember, there's probably more, but I'll leave it at that. 😁 HTH

mmahalwy · 2016-06-20T17:15:42Z

@amireh I think Happypack doesn't work well when doing code splitting - I could be wrong but last time I had to take it out for my js files due to that.

amireh · 2016-06-20T19:38:28Z

@mmahalwy can you open a ticket on happypack's repo with some example?

bertho-zero · 2016-06-22T21:13:29Z

Is there a need to build DLLs manually?

bertho-zero · 2016-06-26T19:28:09Z

I build DLLS on postinstall in my fork and happypack is active by default, I went 8 seconds to 5 seconds when I run npm run dev now.

But the command for generate babel-runtime list doesn't work for me:
egrep -o 'babel-runtime/\S+' | sed 's/\.js$//' | sort | uniq | wc -l, you have an idea for change it ?

amireh · 2016-06-26T20:17:39Z

Oh, that | wc -l shouldn't be there, sorry. The command should be:

egrep -o 'babel-runtime/\S+' | sed 's/\.js$//' | sort | uniq

It might be too aggressive to add babel-runtime to the vendor DLL so maybe you can try removing it from there and just keep it in the main bundle. That should save some of the maintenance burden.

And yes, the DLLs need to be compiled any time the vendor modules change. Doing it in a post-install script should be OK.

bertho-zero · 2016-06-26T20:44:43Z

egrep -o 'babel-runtime/\S+' | sed 's/\.js$//' | sort | uniq

still does not work for me, the command displays nothing and does not close.

amireh · 2016-06-26T23:32:56Z

Err, so maybe the comment wasn't clear, but let me quote:

Generate this list using the following command against the stdout of webpack

So you actually have to run webpack and pipe its output to that command, a la:

./node_modules/.bin/webpack --config webpack/dev.config.js | egrep -o 'babel-runtime/\S+' | sed 's/\.js$//' | sort | uniq

That command basically just tries to extract any babel-runtime module that webpack reports to have included in the build. So if it does report anything, it means it's not being tracked by the vendor DLL. If it reports nothing (and you ARE using the DLL), then everything is OK.

I hope it's clear now! Sorry about the confusion.

bertho-zero · 2016-06-27T17:37:03Z

Thanks, it's clear now

bertho-zero · 2016-06-28T21:57:13Z

I needed to add the --display-modules to webpack for it to work with me, giving me the following command:

webpack --config webpack/dev.config.js --display-modules | egrep -o 'babel-runtime/\S+' | sed 's/\.js$//' | sort | uniq

I think we can also use production as long as there's no lazy loading, it does not stop to remove it easily later.

AndrewRayCode · 2016-06-30T21:10:31Z

webpack/helpers.js

+    var manifest = loadDLLManifest(path.join(root, 'webpack/dlls/manifests/' + dllName + '.json'));
+
+    if (manifest) {
+      console.warn('Webpack: will be using the "%s" DLL.', dllName);


Any particular reason this is a warning instead of a log?

Not a good reason at least!

AndrewRayCode · 2016-07-01T04:09:38Z

I'm trying to create a Dll that has some files in my source code in it, like big json files. However I can't get the webpack server to serve the json file. any ideas? webpack/webpack-dev-middleware#106

msaglietto · 2016-07-04T16:04:12Z

@amireh Impressive, from ~6 to ~3, thanks ...

philipchmalts · 2016-07-12T14:47:10Z

@amireh I followed your changes and have run into a strange issue. Everything bundles correctly and dropped by rebuild time by about half. However, I see the DLL get built from the manifest and it ends up in dist/dlls folder but when I open up any page I get a
Uncaught ReferenceError: DLL_vendor_b8f96ce78b5ec15a89cd is not defined
error in Chrome. The DLL bundle in chrome appears to be corrupted and looks like it contains some html and the beginnings of normalize css? In static/dist/dlls it is the correct file. Here's the relevant part of my webpack dev config.


var webpackConfig = module.exports = {
  devtool: 'inline-eval-cheap-source-map',
  context: path.resolve(__dirname, '..'),
  entry: {
    'main': [
      'webpack-hot-middleware/client?path=http://' + host + ':' + port + '/__webpack_hmr',
      'bootstrap-loader',
      'font-awesome-webpack!./src/theme/font-awesome.config.js',
      './src/client.js'
    ]
  },
  output: {
    path: assetsPath,
    filename: '[name]-[hash].js',
    chunkFilename: '[name]-[chunkhash].js',
    publicPath: 'http://' + host + ':' + port + '/dist/'
  },
  module: {
    loaders: [
      // { test: /\.js?$/,
      //   exclude: /node_modules/,
      //   loader: 'happypack/loader?id=babel'
      // },

      createSourceLoader({
        happy: { id: 'js' },
        test: /\.js?$/,
        loaders: ['babel?' + JSON.stringify(babelLoaderQuery), 'eslint-loader']
      }),
      createSourceLoader({
        happy: { id: 'json'},
        test: /\.json$/,
        loaders: ['json-loader']
      }),
      createSourceLoader({
        happy: { id: 'sass'},
        test: /\.scss$/,
        loader: 'style!css?modules&importLoaders=2&sourceMap&localIdentName=[local]___[hash:base64:5]!autoprefixer?browsers=last 2 version!sass?outputStyle=expanded&sourceMap'
      }),
      // { test: /\.json$/, loader: 'json-loader' },
      // { test: /\.scss$/, loader: 'style!css?modules&importLoaders=2&sourceMap&localIdentName=[local]___[hash:base64:5]!autoprefixer?browsers=last 2 version!sass?outputStyle=expanded&sourceMap' },
      { test: /\.woff(\?v=\d+\.\d+\.\d+)?$/, loader: "url?limit=10000&mimetype=application/font-woff" },
      { test: /\.woff2(\?v=\d+\.\d+\.\d+)?$/, loader: "url?limit=10000&mimetype=application/font-woff" },
      { test: /\.ttf(\?v=\d+\.\d+\.\d+)?$/, loader: "url?limit=10000&mimetype=application/octet-stream" },
      { test: /\.eot(\?v=\d+\.\d+\.\d+)?$/, loader: "file" },
      { test: /\.svg(\?v=\d+\.\d+\.\d+)?$/, loader: "url?limit=10000&mimetype=image/svg+xml" },
      { test: webpackIsomorphicToolsPlugin.regular_expression('images'), loader: 'url-loader?limit=10240' }
    ]
  },
  progress: true,
  resolve: {
    modulesDirectories: [
      'src',
      'node_modules'
    ],
    extensions: ['', '.json', '.js', '.jsx']
  },
  plugins: [
    // HappyPack
    // new HappyPack({
    //   id: 'babel',
    //   cache: false,
    //   loaders: [{
    //     path: path.join( path.resolve(__dirname, '..'), 'node_modules/babel-loader/index.js'),
    //     query: '?' + JSON.stringify(babelLoaderQuery)
    //
    //   }, 'eslint-loader']
    // }),
    // hot reload
    new webpack.HotModuleReplacementPlugin(),
    new webpack.IgnorePlugin(/webpack-stats\.json$/),
    new webpack.DefinePlugin({
      __CLIENT__: true,
      __SERVER__: false,
      __DEVELOPMENT__: true,
      __DEVTOOLS__: true,
      __DLLS__: process.env.WEBPACK_DLLS === '1'  // <-------- DISABLE redux-devtools HERE
    }),
    webpackIsomorphicToolsPlugin.development(),
    createHappyPlugin('js'),
    createHappyPlugin('json'),
    createHappyPlugin('sass')
  ]
};
if (process.env.WEBPACK_DLLS === '1') {
  WebpackHelpers.installVendorDLL(webpackConfig, 'vendor');
}
function createSourceLoader(spec) {
  return Object.keys(spec).reduce(function(x, key) {
    x[key] = spec[key];

    return x;
  },
  {
    include: [path.resolve(__dirname, '../src'), path.resolve(__dirname, '../static/json')]
  });
}

function createHappyPlugin(id) {
  return new HappyPack({
    id: id,
    threadPool: happyThreadPool,

    // disable happy pack with HAPPY=0

    enabled: process.env.HAPPY !== '0',

    // disable caching with HAPPY_CACHE = 0

    cache: process.env.HAPPY_CACHE !== '0',

    // Make happypack more verbose

    verbose: process.env.HAPPY_VERBOSE === '1'
  });
}

Any help would be much appreciated!

AndrewRayCode · 2016-07-12T16:08:04Z

@philipchmalts most likely the DLL file is just 404ing and the HTML you're seeing is from the 404 page.

It's a bit confusing, but Webpack dev server will NOT serve the DLL. I don't fully understand why. Instead you have to load the DLL from the application server (for this project, port 3000), which just serves the file from disk, directly from the dist folder.

In my Html.js file I do this:

{ __DEVELOPMENT__ ? <script src="/dist/vendor.dll.js" charSet="UTF-8" /> : null }

which will just serve that file directly from dist from the main express webapp.

In production you probably don't want to build a dll file, you probably want to use commonchunks or maybe require.ensure if you don't want to build one main file.

philipchmalts · 2016-07-12T20:15:27Z

I have this in my html.js

  {__DLLS__ && __DEVELOPMENT__ &&
            <script
              src="/dist/dlls/dll_vendor.js"
              charSet="UTF-8"
            />
          }

But still getting the same thing. I don't understand at which point the dll file is getting mangled to contain this.

<!doctype html>
<html lang="en-us" data-reactroot="" data-reactid="1" data-react-checksum="1362214140"><head data-reactid="2"><title data-react-helmet="true" data-reactid="3"></title><meta data-react-helmet="true" name="description" content="Invest Forward" data-reactid="4"/><meta data-react-helmet="true" charset="utf-8" data-reactid="5"/><meta data-react-helmet="true" property="og:site_name" content="Invest Forward" data-reactid="6"/><meta data-react-helmet="true" property="og:image" content="https://s3.amazonaws.com/fixed-assets/Icon-IF-200x200-Wht-on-Grn.jpg" data-reactid="7"/><meta data-react-helmet="true" property="og:locale" content="en_US" data-reactid="8"/><meta data-react-helmet="true" property="og:title" content="Online Financial Planning &amp; Advice | Invest Forward" data-reactid="9"/><meta data-react-helmet="true" property="og:description" content="We believe everyone deserves access to high quality personalized financial advice. Invest Forward wants to get you financially healthy " data-reactid="10"/><meta data-react-helmet="true" property="og:card" content="" data-reactid="11"/><meta data-react-helmet="true" property="og:site" content="Invest Forward" data-reactid="12"/><meta data-react-helmet="true" property="og:creator" content="Invest Forward" data-reactid="13"/><meta data-react-helmet="true" property="og:title" content="" data-reactid="14"/><meta data-react-helmet="true" property="og:description" content="All the modern best practices in one example." data-reactid="15"/><meta data-react-helmet="true" property="og:image" content="https://s3.amazonaws.com/fixed-assets/Icon-IF-200x200-Wht-on-Grn.jpg" data-reactid="16"/><meta data-react-helmet="true" property="og:image:width" content="200" data-reactid="17"/><meta data-react-helmet="true" property="og:image:height" content="200" data-reactid="18"/><link rel="shortcut icon" href="/favicon.ico" data-reactid="19"/><meta name="viewport" content="width=device-width, initial-scale=1" data-reactid="20"/><style data-reactid="21">/**
 *  Define scss variables here.
 *
 *  Available options for Bootstrap:
 *  http://getbootstrap.com/customize/
 *
 */
``` ...etc

amireh · 2016-07-12T20:40:43Z

@philipchmalts Try using a fully-qualified URL for the DLL, basically what @delvarworld said above which I suspect is the issue. Instead of src="/dist/vendor.dll.js" try using http://localhost:3000/dist/vendor.dll.js (or 3001, i'm not sure, it was one of these two. This has to do with webpack-isomorphic-tools i assume, I don't really know about this setup)

philipchmalts · 2016-07-12T20:58:34Z

@amireh @delvarworld Thanks!!! that did the trick I think, going to kick the tires on it for a bit but it looks like it works. Down from about ~4-6s to ~1-3s, was pulling my hair out about this.

amireh · 2016-07-12T21:39:06Z

@philipchmalts nice man, glad it works! 😁

bertho-zero · 2016-09-14T21:27:03Z

If ever the DLL is invalid I added a warning and support of the DLL is automatically removed: https://github.com/erikras/react-redux-universal-hot-example/compare/master...bertho-zero:progressive?expand=1#diff-375445b5a84cd461b8571395b5d4a75fR76

amireh added 3 commits June 18, 2016 11:46

Upgrade to HappyPack with SourceMap support

879139a

bertho-zero referenced this pull request in bertho-zero/react-redux-universal-hot-example Jun 25, 2016

Bake-in support for webpack DLLs #1201

6681eee

amireh mentioned this pull request Jun 27, 2016

Webpack incremental builds are slow #616

Open

oyeanuj mentioned this pull request Jun 28, 2016

Implement Webpack DLL Plugin react-boilerplate/react-boilerplate#495

Merged

6 tasks

AndrewRayCode reviewed Jun 30, 2016
View reviewed changes

bertho-zero referenced this pull request in bertho-zero/react-redux-universal-hot-example Jun 30, 2016

#1201 Automation: unable happypack if invalid dll

d7acf98

OkuraSt pushed a commit to OkuraSt/react-redux-universal-hot-example that referenced this pull request Jul 7, 2016

Bake-in support for webpack DLLs erikras#1201

2b3812e

bertho-zero mentioned this pull request Sep 15, 2016

Can we mark this as deprecated? #1281

Open

amireh mentioned this pull request Sep 27, 2016

Example of DLL usage with happypack amireh/happypack#90

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bake-in support for webpack DLLs #1201

Bake-in support for webpack DLLs #1201

amireh commented Jun 18, 2016

oyeanuj commented Jun 18, 2016

amireh commented Jun 18, 2016

oyeanuj commented Jun 18, 2016

amireh commented Jun 19, 2016 •

edited

Loading

mmahalwy commented Jun 20, 2016

amireh commented Jun 20, 2016

bertho-zero commented Jun 22, 2016 •

edited

Loading

bertho-zero commented Jun 26, 2016 •

edited

Loading

amireh commented Jun 26, 2016

bertho-zero commented Jun 26, 2016

amireh commented Jun 26, 2016 •

edited

Loading

bertho-zero commented Jun 27, 2016

bertho-zero commented Jun 28, 2016 •

edited

Loading

AndrewRayCode Jun 30, 2016

amireh Jun 30, 2016

AndrewRayCode commented Jul 1, 2016

msaglietto commented Jul 4, 2016

philipchmalts commented Jul 12, 2016

AndrewRayCode commented Jul 12, 2016 •

edited

Loading

philipchmalts commented Jul 12, 2016

amireh commented Jul 12, 2016

philipchmalts commented Jul 12, 2016

amireh commented Jul 12, 2016

bertho-zero commented Sep 14, 2016

Bake-in support for webpack DLLs #1201

Are you sure you want to change the base?

Bake-in support for webpack DLLs #1201

Conversation

amireh commented Jun 18, 2016

oyeanuj commented Jun 18, 2016

amireh commented Jun 18, 2016

oyeanuj commented Jun 18, 2016

amireh commented Jun 19, 2016 • edited Loading

mmahalwy commented Jun 20, 2016

amireh commented Jun 20, 2016

bertho-zero commented Jun 22, 2016 • edited Loading

bertho-zero commented Jun 26, 2016 • edited Loading

amireh commented Jun 26, 2016

bertho-zero commented Jun 26, 2016

amireh commented Jun 26, 2016 • edited Loading

bertho-zero commented Jun 27, 2016

bertho-zero commented Jun 28, 2016 • edited Loading

AndrewRayCode Jun 30, 2016

Choose a reason for hiding this comment

amireh Jun 30, 2016

Choose a reason for hiding this comment

AndrewRayCode commented Jul 1, 2016

msaglietto commented Jul 4, 2016

philipchmalts commented Jul 12, 2016

AndrewRayCode commented Jul 12, 2016 • edited Loading

philipchmalts commented Jul 12, 2016

amireh commented Jul 12, 2016

philipchmalts commented Jul 12, 2016

amireh commented Jul 12, 2016

bertho-zero commented Sep 14, 2016

amireh commented Jun 19, 2016 •

edited

Loading

bertho-zero commented Jun 22, 2016 •

edited

Loading

bertho-zero commented Jun 26, 2016 •

edited

Loading

amireh commented Jun 26, 2016 •

edited

Loading

bertho-zero commented Jun 28, 2016 •

edited

Loading

AndrewRayCode commented Jul 12, 2016 •

edited

Loading