Discovery: Faster build tool

[jmbowman]

The duration of front end asset bundling and minification has been flagged several times as a source of friction in development and deployment:

• Tutor rebuilds all MFEs relatively often, and this duration is increasing as the number of MFEs grows
• JS minification is a fairly high percentage of setup and deployment time for edx-platform
• MFEs often do less aggressive minification in CI than used in production to keep the test duration short, which means we're testing slightly different code than deployed to production.
• The webpack bundler ecosystem requires a lot of plugins and dependencies, some of which have been abandoned and/or had long-lasting security vulnerabilities.

We currently use webpack for bundling and uglify-js for minification, but there are now much faster alternatives to both tools:

• SWC is a very fast JavaScript/TypeScript compiler/bundler/minifier implemented in Rust which many other tools in the front end ecosystem are starting to utilize.
• Parcel is a fast bundler which adopted SWC and claims to require much less configuration than webpack for most common use cases.
• Next.js is a pretty comprehensive toolkit for using and deploying React which also recently adopted SWC.
• esbuild Is a very fast Go-based bundler & minifier
• I haven't yet found good benchmarks on the recent versions of each of these bundlers, but I did find some good benchmarks of the underlying minifiers.

There have been initial conversations around utilizing SWC in webpack also, but there doesn't seem to be much actual code to that end yet: webpack/webpack#13425 .

As a first step towards optimizing our development feedback loops and deployment time, please do some discovery on which if any of these alternatives could satisfactorily meet our needs and allow us to completely remove our dependency on webpack and/or uglify-js in most MFEs (and ideally edx-platform as well). In particular, we want to know:

• Does the replacement lack any important features our current webpack/uglify-js ecosystem provides?
• Does the replacement gain us any useful features our current ecosystem lacks?
• How much faster is the replacement?
• How much configuration does the replacement need?
• How many dependencies does the replacement have relative to our current stack, and how actively maintained are they?
• How much work is it to replace what we currently use?

We don't need extremely accurate and detailed answers to these questions, but we need enough of an idea to make a call on whether or not to go ahead with a pilot implementation, and if so which tool(s) to use.

[jmbowman]

Already found another contender currently being rewritten in Rust, doesn't seem as widely used as the others but may also be worth a look: https://rome.tools/ . What caught my attention is that it might serve as a faster replacement for eslint.

[kdmccormick]

Tutor rebuilds all MFEs relatively often, and this duration is increasing as the number of MFEs grows

We recently merged overhangio/tutor-mfe#58, which significantly speeds up the dev MFE build (should be a matter of seconds now, no matter how many MFEs are installed). I am also working on https://github.com/overhangio/2u-tutor-adoption/issues/87, which should provide a generic production-ready base MFE image and thus reduce the need to build for production users.

All that being said, I think a faster build tool is still warranted!

[ormsbee]

It's outdated by this point, but I have an old asset compilation audit doc from late 2017 that goes over some of the bottlenecks in edx-platform static asset processing. We've fixed a number of items on that list, but I suspect that at least some of these are still issues.

[kdmccormick]

^ Nice. Tutor works around many of those with a custom re-implemtation of update_assets that is noticeably quicker, but it still hits some of the underlying bottlenecks like xmodule_assets.

[ghassanmas]

There are a lot to unpack here, I have some comments and inquries as well.

My background on this

1. I've been doing some testing on this topic every now and then, and we were able have a workig concept of tutor-mfe prebuilt image that works for everyone (which doesn't need to rebuilt for changing config value only). Also relate to @kdmccormick comment above. See MFE dynamic config overhangio/tutor-mfe#69.

2. I think we have to consider install time as well, in my testing with tutor-mfe I found that npm install might take around 1/3 to 1/2 of total build time¹. (Note this might not quite noticaiable on local dev testing because):

   • In local dev, probably tester focus npm run build beacuse node_modules already exists
   • in local dev, there is a .npm cache which can boost the install while this doesn't exists on tutor base image (or really any fresh node Docker image).

3. During my testing I found some plugins when disabled reduce the built time a bit, about ~2-4 seconds. BundleAnalyzer, SourceMap

4. I think its important to mention SASS/SCSS, files, I think its play a major role in the build time.

Inquirues:

1. Regarding uglify-js, I beleive @jmbowman probably you were referring to babel (and it's set of plugin)? I think uglify-js is used in edx-platform but not in frontend-build.

Webpack Alternative:

1. Do any of the alternaive support module federation, I know we don't use this featuer but we might be near future
2. Do any of alternative support code splitting, dynamic import, bundle chunks...etc Those are important for app perforamnce, while MFEs might not 100% utilize them but I think ultimately we should aim to...
3. Let's take the case of Gatsby.js/Next.js both are competing website building framework, one metric that they both are competeing with is the site build-time, however both use and still rely heavily on using webpack^{2 3 4} probaly bceause of reason relate to the previos point

Footnotes:

Footnotes

1. Related check my discourse post about testing with tutor-mfe https://discuss.openedx.org/t/microfrontends-a-retrospective/6486/20 ↩

2. While Next.js use SWC, but still SWC doesn't really replace weback for them, its more or less replace part of babel (as to my qucik checking). Another point is Next.js last year hired the author of Webpack check his tweet https://twitter.com/wSokra/status/1381860800533528576. ↩

3. Next.js/Wepback Doc https://nextjs.org/docs/messages/webpack5 ↩

4. Gatsby.JS/Webpack Doc https://www.gatsbyjs.com/docs/glossary/webpack/ ↩