There's a lot of interest in improving the performance of the Meteor command-line tool and build system, and if you're interesting in helping, this document is meant to be a jumping off point.
Sooner or later, the tool will be inevitably be substantially rewritten or replaced in order to achieve those sweet sub-second build times everyone loves, because it is currently too complex and reliant on the file system to get there. However, until then, there's work that can be done to make things faster.
For more on the tool, see also the README files in this directory and its sub-directories.
The best way to get information about where time is going in the Meteor tool is with built-in profiler, activated with the environment variable METEOR_PROFILE. The value is interpreted as a threshold in ms, so the value 1 means to report on all calls that take more than 1ms:
METEOR_PROFILE=1 meteor ...You'll then see a report (or a series of reports) printed to the console that might look something like:
| (#1) Profiling: ProjectContext prepareProjectForBuild
| ProjectContext prepareProjectForBuild..........9,207 ms (1)
| _initializeCatalog.............................24 ms (1)
| files.readFile 7 ms (2)
| runJavaScript package.js 2 ms (1)
| files.rm_recursive 4 ms (4)
| other _initializeCatalog 11 ms
| _resolveConstraints.........................6,702 ms (1)
| bundler.readJsImage.........................42 ms (1)
...
| (#1) Total: 9,544 ms (ProjectContext prepareProjectForBuild)
The report is a simple top-down listing, where each entry represents a function call or other instrumented block in the code. A dotted line is drawn for entries that have child entries, and the total time of the child entries should add up to the parent entry (though turning up the profiling threshold may mess with the math). The number in parentheses is the number of times that function or block was called, and the time listed is the total cumulative time of all the calls.
Entries in the profiler come from annotations that instrument code, and unlike what you get with sampling profilers (like node --prof or dtrace), the call counts are exact. All times are measured with high-resolution wall-clock timers and only rounded for display. However, time measurements are quite susceptible to being inflated by random acts of garbage collection; profiler overhead; competition for the CPU; and so on. Also, if the tool were more highly parallelized, these stopwatch-type measurements might stop adding up, but so far they've been pretty accurate as far as we know.
Please see the section "Performance Considerations" before interpreting a profiler report.
Read this section before drawing conclusions or comparing numbers online.
The Meteor build tool is complex and has several layers of caches and modes of operation. It's not apples-to-apples to compare a rebuild time between a Meteor release and a checkout, for example, or to compare a start-up with a cold cache to one with a warm cache. This section lists the various nuances of build tool performance.
Not everything is covered by the reports we generate ("prepare project" and "build app"). For example, you won't see:
- Time spent in app start-up
- Time spent creating file watchers after a rebuild
The tool saves work using various disk caches and in-memory caches. There are caches of:
- Built packages and plugins
- Compiler plugin results - and not just on the most recent version of a file! If you are testing rebuilds by toggling a character back and forth, you may hit cache.
- Linker output
- The entire server program, for client-only changes
- Constraint solver results
Therefore, app start-up will be slow if the disk caches haven't been generated or are invalid because a package needs to be rebuilt, for instance, or the constraint solver needs to run. Rebuilds are much faster than the initial build of the app, and the second rebuild even seems to be faster than the first, at which point the time levels off.
See the section "More About Caches".
The Meteor tool behaves differently depending on whether you are running a Meteor release, or a checked-out branch of a git repository. Most people run a Meteor release, but if you are making any changes to the tool or core packages, you'll be running Meteor in checked-out mode for testing, debugging, and measurement, so it's important to know the discrepancies. (See also "More About Release vs. Branch".)
The following are performance-related differences between running an app with a release of Meteor vs. a checked-out branch of Meteor:
-
Checked-out Meteor recompiles core packages. When you run an app using checked-out Meteor, it will compile core packages into
APP/.meteor/local/isopacks-- that's right, into your app! This process can take a while -- as much as a couple minutes -- but it's only done once per package per app, unless the package changes. When the app starts up, it will read in the built packages and decide if it needs to do anything. However, there's noticeable extra time "preparing" for each rebuild, because the core packages are rescanned, and the large number of files being watched for changes can cause several seconds of delay after a rebuild as well. -
Constraint-solver performance is different. Core packages are constrained by a release and also by running from checkout, but the effect is different, and the constraint solver seems to have an easier time on the constraint problems it gets when running from checkout, because of the way core packages are pinned like local packages.
-
Constraint-solver won't save
APP/.meteor/versionsif there's a release inAPP/.meteor/release. Normally, when running released Meteor,APP/.meteor/versionsprovide stability to your package versions between runs of your app. However, when you runREPO/meteor, core package versions are temporarily overridden by the versions in the repo, but not saved, unlessAPP/.meteor/releaseis set tonone, meaning the app is not written against a released version of Meteor. This difference comes up when testing and measuring the behavior of the constraint solver.
Time spent rebundling and rebuilding the app seems pretty similar across Release and Checkout.
Different apps can strain different parts of the Meteor tool, such as:
- Constraint solver - lots of dependencies; large package graph
- File watchers - lots of total files, e.g. thousands
- Modules - as of Meteor 1.3, imports from packages and
node_modulesmust be scanned and analyzed - CSS processing - e.g. in case of a large LESS or Sass framework in the app
- Community plugins, including legacy plugins without caching between builds
Spinning disks are much slower than SSDs.
Windows has a very different file system from Mac OS X or Linux.
On OS X and Linux, we have all these nice properties:
- Usable symlinks
- Renaming a file is cheap
- Overwriting a file by rename is atomic
- An file can be deleted ("unlinked") while open
- A file can be moved (renamed) while open
Windows doesn't have these properties; for example, renaming a file is a copy operation. In addition, virus scanners and other software will sometimes block or postpone file operations. And, of course, there are the path separator differences. Parts of the tool work really hard to target OS X and Linux efficiently (with symlinks and renames, for example) while doing something that behaves well on Windows, even if it's slower.
If you see a large time next to "other (function)", that's a case where the tool needs further instrumentation. For example, you may see something like this:
| ImportScanner#_readFile...............1,338 ms (329)
| files.readFile 100 ms (329)
| sha1 3 ms (329)
| other ImportScanner#_readFile 1,235 ms
In this example, some function calls under ImportScanner#_readFile just happen to be instrumented (files.readFile and sha1), but no other calls under ImportScanner#_readFile have been annotated yet. Given how much time is spent in this call, we should seek out more calls to annotate. You can determine by experimentation which calls are significant enough to add to the profile.
To annotate a function or block in tool code to be included in the profile, use one of the following forms:
// Profile(.) takes a function and wraps it, preserving arguments
// and this. You can often get away without introducing any extra
// indentation in your text editor.
var doIt = Profile("doIt", function (...) {
...
});
// Wrap a method
MyClass.prototype.doIt =
Profile("MyClass doIt", MyClass.prototype.doIt);
// Profile.time(.) invokes a block immediately and times it,
// passing the return value through.
function doIt() {
...
var result = Profile.time("doIt", () => {
...
});
...
}
The Profiler is activated using Profile.run(.), which will cause a report to be generated. Currently we generate a couple reports when METEOR_PROFILE is set -- one for preparing the project (including constraint solving) and one for building or rebuilding the app.
In addition to tool code, you can also use Profile in compiler plugins. If you want to use Profile in packages that are loaded into the tool (e.g. packages depended on by compiler plugins, or specially loaded into the tool as isopackets), you should always test (typeof Profile !== 'undefined') before accessing Profile, or pass it in from the tool as an option.
These are areas to improve or investigate, along with what's known, in no particular order.
See: meteor#5875.
Meteor's CSS minification is user-pluggable since 1.2. What the standard
minifier does is concatenate your CSS into one file with import statements
hoisted to the top. In production mode, this file is minified before being
served. The benefit of doing the same CSS merging in development and
production is that the meanings of relative paths in CSS files and the order
of rules doesn't change. However, reparsing and merging the CSS for a typical
app can take anywhere from 500ms to a couple seconds or more, depending on
the amount of CSS.
Initial profiling shows the time is split between different operations like parsing CSS, generating CSS, and working with source maps. Approaches could include:
- Inserting a new cache, such as around parsing files
- Only recalculating the CSS if it changes
- Using a faster CSS library
The tool spends noticeable building source maps, as do other tools such as
Webpack, which have evolved creative mechanisms to cut down the time.
There's some question of whether the source-map library is as efficient
as it could be, and whether we're using it as efficiently as we could be.
See: meteor#5818
The "linker cache" on disk at APP/.meteor/local/bundler-cache/linker has some
very large files that take a long time write, and they are apparently not
cleaned up either. Sometimes you will see calls to files.writeFileAtomically
taking many seconds at the top of the "Rebuild App" report, but somehow outside
the top level (probably because they are fired off asynchronously).
Some initial work has been done to make constraint solving fast in the typical
case where a previous solution exists in APP/.meteor/versions, but there is
possible room for improvement.
For example, the previous solution could be checked without
invoking the logic solver at all, though the manual logic would have to
match closely. Implemented!
Time spent reading from the packages database (a SQLite file) could be understood better and improved by batching queries, reading less data, or using SQLite better.
See: meteor#6137
Talking to the package server to download new package metadata seems to take longer than it should; easily 5 seconds, based on watching the progress display, and occasionally much longer.
You'll find scattered through the tool code comments about how it would be more efficient to do certain processing or reading in a way such that it wouldn't have to be done a second time in a different part of the tool, or it wouldn't have to be done on each rebuild. The first one is about the flow of information in the tool, and the second is about making better use of information about what changed, either through caching or file-watching.
It seems like it would be great if file-watching gave the tool perfect information, so it wouldn't have to go to disk and read lots of files even though, in theory, if they had changed it would know. You can imagine a tool that is super fast because it basically reads and writes diffs of the state of the file system. It's not clear how feasible that is, though. Right now, when we're told a file has changed, we immediately go to work, which includes tearing down file-watchers; we don't wait to get the exact list of what files have changed. For the tool to remain receptive to changes would require a different paradigm.
However, it seems like some middle ground would be possible, where the tool does not simply merge all the lists of files it has ("WatchSets") and wait for any change to any of them, but where rather it responds to changes to different sets of files by rebuilding different parts and then putting them together; for example, only processing the CSS if some CSS changed.
It also might be that changing to a different file-watching library would have performance benefits.
Disk caches exist in the following locations, where APP is the app:
APP/.meteor/local/isopacksAPP/.meteor/local/bundler-cacheAPP/.meteor/local/plugin-cache~/.meteor(package storage, when running a release)REPO/.meteor(package storage, when running a checked-out branch)
You can safely remove these directories to clear your caches, but don't remove APP/.meteor/local if you have data in your app's database, because the local Mongo DB resides in APP/.meteor/local/db. (If you don't care about the DB, you can remove all of local for the ultimate clean checkout experience.) The package storage directory holds master copies of packages that will be copied or linked into your project (like Maven's .m2). There's probably no reason to clear them out, except to save some disk space. It's sometimes nice to know where these files are located in case you want to poke around or add console.logs to released code for debugging.
Most people run a released version of the meteor tool, but if you are making any changes to the tool or core packages, you'll be running it in checked-out mode for testing and debugging. Here's how the two modes work:
-
Release: This is when you run the installed
meteortool on your computer. The tool and core packages are part of a release with a name likeMETEOR@1.2.1and were downloaded from the package server and cached in~/.meteoralong with other Atmosphere packages. When the tool is run, the current release is determined fromAPP/.meteor/releasewhen running an app, or the value of--releaseon the command-line. -
Checked-Out Branch: This is when you check out the
meteor/meteorGitHub repository asREPOand runREPO/meteor. The tool is loaded fromREPO/tools, and core packages are loaded fromREPO/packages. The tool takes care of transpiling itself and rebuilding out-of-date core packages. Core packages in the repo behave much like local packages in your app; packagefoois treated as the only known version offoo, and its files are watched for hot-reload.