Vuepress building is too slow ?!

[JimmyVV]

• I confirm that this is an issue rather than a question.

Bug report

My project is large, it maybe contain 6000 markdown files. Once, I meet the dev performance problem, and I solve it by this command node --max_old_space_size=8192 ./node_modules/vuepress/cli.js dev src.

But, when I run building production, the process is too slow. I also modify the building command with node --max_old_space_size=8192 ./node_modules/vuepress/cli.js build src. Unfortunately, it doesn't work.

Version

"vuepress": "^1.0.0-alpha.47",

Steps to reproduce

What is expected?

What is actually happening?

Other relevant information

• Your OS: MAC
• Node.js version: 8.11
• Browser version:
• Is this a global or local install? local
• Which package manager did you use for the install?
• Does this issue occur when all plugins are disabled?

[JimmyVV]

the renderer.renderToString takes up much time.

more than 4 seconds

# analyse code
 try {
      console.time('start');
      html = await this.renderer.renderToString(context)
      console.timeEnd('start');
    } catch (e) {
      console.error(logger.error(chalk.red(`Error rendering ${pagePath}:`), false))
      throw e
    }

[JimmyVV]

after I improve the ssr performance, the time reduce to 60+ms

Rendering page: /dev/api-backend/open-api/express/by-provider/logistics.onGetQuota.htmlstart: 822.013ms

However, the more markdown files it compile, it will need more time.

the result is :

plate-message/templateMessage.getTemplateLibraryList.htmlstart: 808.080ms
Rendering page: /miniprogram/dev/api-backend/open-api/template-message/templateMessage.send.htmlstart: 789.571ms
Rendering page: /miniprogram/dev/api-backend/open-api/template-message/templateMessage.getTemplateList.htmlstart: 807.422ms
Rendering page: /miniprogram/dev/api-backend/open-api/uniform-message/uniformMessage.send.htmlstart: 802.172ms
Rendering page: /miniprogram/dev/api-backend/open-api/updatable-message/updatableMessage.createActivityId.htmlstart: 812.487ms
Rendering page: /miniprogram/dev/api-backend/summary.part.htmlstart: 768.504ms
Rendering page: /miniprogram/dev/api-backend/open-api/updatable-message/updatableMessage.setUpdatableMsg.htmlstart: 834.227ms
Rendering page: /miniprogram/dev/api-backend/open-api/user-info/auth.getPaidUnionId.htmlstart: 848.840ms
Rendering page: /miniprogram/dev/api/ad/InterstitialAd.offError.htmlstart: 783.710ms

[JimmyVV]

@ulivz Help!

[JimmyVV]

I am using the default theme, which causes memory leaks.

[mandaputtra]

I maybe consider doing this with different approach if there are 6000 markdown file. Or maybe different tools. Yes it will slow because rendering/parsing markdown itself are heavy.

if you cant, I consider to make it smaller by separate the docs file.

[ulivz]

First of all, why is it slow? The compilation time depends on the size of the content you build. You have 6000 files to compile, isn't that slow?

Secondly, let's talk about the necessity of this issue. You can try any other static website building tool to build it. If they are very fast, please use accurate data to show that slow construction is a unique issue of vuepress.

[JimmyVV]

?? the number of files doesn't matter, the point is the compilation time. And When I change back 0.11, the renderString time reduce to 50ms per page.

I also analyse the memory usage, that actually indicate memory leaks do happen in vuepress@beta.

[xbill82]

@ulivz I wouldn't wipe this issue out so quickly. Sure @JimmyVV should provide more data but I'm migrating a documentation (5000+ files) from Metalsmith to VuePress and the main issue we have now is that we can't deploy on Netlify because the build time is longer than the maximum build timeout. Metalsmith did build the whole doc within Netlify's timeouts.

Ok, Vuepress' build process is surely more complex than Metalsmith and Vuepress is indeed a better tool, but yet... The build time is incredibly bigger in VP and this is strange.

So this is meant to add some data to the "comparison with other static website building tools" you asked above, @ulivz .

@JimmyVV please, would you post your investigations about memleaks introduced by the default theme?

Another point that cannot be ignored is that when @JimmyVV downgrades, the perfs are better. I definitely think we should look into this.

[xbill82]

UPDATE: I am currently building my documentation (as mentioned above, 5000+ files) using Travis CI and my build job times out after 2h (the same job worked fine with Metalsmith -- not meaning to say Metalsmith was a better tool, just faster).

It seems weird.

[xbill82]

UPDATE: Switching to Node v10.x actually speeds up the build time. We are now able to build in Travis, but an issue is still present.
We noticed that the build time per-file slows down during the build. It looks like there's some kind of leak (more CPU than memory) slowing things down.

[ProTip]

It appears that part of the build process spins up many node processes however at least with the standard build command rendering the static content only utilizes a single node process. Perhaps utilizing available cores for static content generation would speed this up greatly for people?

[swiftone]

Any news on this? Our CI tests are silly-slow (technical term) because of this, not to mention every time we have to rebuild the site locally during development.

[xbill82]

Hi @swiftone we ended up chunking our entire documentation in many different projects that we host on different subfolders of the same S3 bucket.
It sucks, because the SPA mode obviously doesn't work across sub-projects. Not to mention the reconciliation of the Algolia search by forcing the base URL and all the problems that we had to deal with to make developer experience decent with all this...

[JimmyVV]

Recently, I used node workers to speed up the compiling process. the total time decrease from 10min to 2min. Maybe this is another way to improve the compiling procedure, by worker thread.

[kefranabg]

@JimmyVV maybe you could share your work?

[itsxallwater]

Agreed, would be nice to see your solution @JimmyVV!

[itsxallwater]

I've got an update to core/lib/node/build/index.js and an addition of core/lib/node/build/worker.js to leverage Node's worker_threads that I've been testing out.

For background, we've got a repo with ~2250 pages (producing a dist of ~4800 files / 350 MB) that was building in 1 hour, 20 minutes with the current release of Vuepress. With these changes in, I'm now getting the following timings:

1x worker thread = 1 hour
2x worker threads = 26 minutes
4x worker threads = 13.5 minutes (ironically the last minute was console updates from workers on what they were rendering, the actual rendering was already done)
8x worker threads = 8.5 minutes (I added in a verbose flag to toggle the console write of the specific document name being rendered)
16x worker threads = 7.5 minutes

I'll get a PR in for this tomorrow since it seems there could be interest but figured I'd at least leave a note here. Will reference once there's a PR.

[itsxallwater]

We ran this in prod okay today but I've got some work to be done on prepping that PR, including things I'd like to smooth over with my implementation:

1. Make the thread count and verbose flags config driven
2. Extend the worker logging to include progress bars
3. Perhaps even do some Node version sniffing to allow it to build as-is without requiring Node's worker_threads for better backwards compatibility.

Plus things that aren't working for me from https://github.com/vuejs/vuepress/blob/master/.github/CONTRIBUTING.md that I need to chew on:

1. yarn bootstrap is not a defined script and yarn boot throws Error: Cannot find module '@vuepress/shared-utils'
2. The commit message convention link is broken

All that said, if anyone is actively following this and curious, you can see the current state of my work at itsxallwater@fa4c703.

[bretterer]

@itsxallwater I am trying to implement this into my project to help with our build times. I am attempting to replace the build/index.js file and add the worker.js file. I have a script that copies my build and worker script over to the vuepress core library, but when I do it throws an error during build saying that it cannot find modules dist/manifest/client.json

How were you successful in doing the workers in your project. Are you using a fork of vuepress instead of file replacements like I am doing?

My build script looks like:

#!/bin/sh
cp .vuepress/scripts/updateBuildScript.js ../../../node_modules/@bretterer/vuepress-site/node_modules/@vuepress/core/lib/node/build/index.js
cp .vuepress/scripts/addWorkerScript.js ../../../node_modules/@bretterer/vuepress-site/node_modules/@vuepress/core/lib/node/build/worker.js
vuepress build . && cp conductor.yml dist/conductor.yml

[itsxallwater]

@bretterer I'm excited to hear you're trying my changes out! Apologies to all that I've not been able to come back to get an official PR in yet.

Outside of the replacement of ../node_modules/@vuepress/core/lib/node/build/index.js and addition of worker.js, we're not doing anything special. I've run the build two ways with our project package.json npm build script set as:

1. vuepress build
2. node --max_old_space_size=8192 ./node_modules/vuepress/cli.js build

Both have picked up and run the build with my changes with no issue. It's worth calling out that I'm on node 12.14.0 and npm 6.13.0.

[favoyang]

Disable the plugin-last-updated, may reduce half build time. For a repo with less than 200 (dynamic generated) docs:

• turn on: 45.51s
• turn off: 23.76s

themeConfig: {
  lastUpdated: false
}

The plugin need spawn a child process to run a local git query on each file, which can be especially slower on cloud with slow disk.
https://github.com/vuejs/vuepress/blob/master/packages/%40vuepress/plugin-last-updated/index.js

[bretterer]

@itsxallwater it seems that with the release of 1.3.0 today, your fix no longer functions correctly? I am being presented with errors from the build when I try to run it, are you seeing that as well?

[itsxallwater]

Just had a chance to update to 1.3 and try @bretterer and you're correct. Given that @kefranabg has picked this up odds are his path will carry us to victory ;)

[itsxallwater]

Just to call out #2163 (comment), worth mentioning here that our build ran in 24 minutes with version 1.3 + @kefranabg's updates from that PR, without my worker_threads stuff. I may still re-work them in to see if they further help things since rolling with 16x threads was yielding 7.5 minutes, but it seems like there's a good official path forward available here as an option now.

[itsxallwater]

@bretterer if you're so inclined, I updated my fork to 1.3 and made a few adjustments to get my worker_threads version working again. You can find it at https://github.com/itsxallwater/vuepress/tree/feat/implement-node-worker-threads

Haven't had a chance to revisit the contribution guide to work out my issues getting Yarn and the monorepo/workspace stuff sorted out, which would lend itself towards getting this in as a PR. @kefranabg with the work you're putting into perf improvements at the moment, is that a worthwhile thing to pursue? I am seeing additional build runtime improvements in taking advantage of node's worker_threads.

[itsxallwater]

[bretterer]

@bretterer if you're so inclined, I updated my fork to 1.3 and made a few adjustments to get my worker_threads version working again. You can find it at https://github.com/itsxallwater/vuepress/tree/feat/implement-node-worker-threads

@itsxallwater Ill take a look on tuesday! Thank you!

[itsxallwater]

FYI @bretterer latest updates to that fork include allowing a configurable number of worker threads from the CLI. When you run the build command, -w will allow you to set that number. I also took out my verbose flag and have things respecting the --silent option that build has.

I got through all of my other obstacles so I'll get a PR in on these updates formally :)

[tylerbutler]

@itsxallwater I tried your fork on some internal API documentation. It's ~4000 markdown files. Previously our build took 4-5 hours, but with your fork, it's ~20 minutes. Thanks! I'm looking forward to seeing this in master. Your command line additions will be welcome as well.

[kefranabg]

While I'm working on VuePress build time improvement, using @itsxallwater might be a good woarkaround https://github.com/itsxallwater/vuepress/tree/feat/implement-node-worker-threads 👍

[ianwalter]

Can I recommend https://github.com/josdejong/workerpool for a worker threads implementation? It has some nice features that sound relevant from the discussion:

1. Node.js detection to fallback to child processes if worker threads aren't supported
2. Automatically determines thread count based on host's CPU if not specified

Wish I had time to contribute more to this, good luck!

[itsxallwater]

Thanks to @kefranabg we are no longer relying on my previous worker thread workaround. More detail at #2163 (comment).

[itsxallwater]

Just a gentle touch to note I'm thinking about revisiting this. Our docs continue to grow as we add more and more product and our build times (running with GitHub Actions) are creeping towards 40 minutes.

[bretterer]

Thanks @itsxallwater ... We were just talking about this and have not been able to leave v1.2 because of the size of our documentation. There is for sure a memory leak as sites grow bigger. builds start out fast, and then quickly get slower and slower the more pages it works through.

[bretterer]

You can see what I am doing in order to use the workers you suggested here without us having to maintain a fork of vuepress ourselves.

https://github.com/okta/okta-developer-docs/blob/master/packages/%40okta/vuepress-site/.vuepress/scripts/build.sh

[xbill82]

@itsxallwater thanks a lot for your work on the worker threads, we'll surely try it out one of these days (our doc is 6000+ .md files).

@bretterer concerning the memleak, we've tried to take a look at it one year ago and found out that it was not a memleak, but rather vue-ssr that loads all the documents in memory before starting the build. I'm not 100% sure of this information, it should be verified.

[itsxallwater]

@bretterer that's clever, thanks for sharing. I just integrated this into our project and it took our GitHub Action runtime from ~40 minutes down to ~20 minutes. Used to be ~30 minutes build/~10 minutes deploy, now splitting more like ~10 minutes/~10 minutes.

I noticed your code was mostly equivalent to my previous PR. I did set our shouldPrefetch flag back to true in the worker.js file. I also noticed that our custom 404 page was getting dropped on account of this.context.addPage being an async call, so I updated that call in index.js to have an await.

Can see CI/CD runtimes at https://github.com/zumasys/docs/actions and you can see our tweaked scripts at https://github.com/zumasys/docs/tree/master/site/.vuepress/scripts.

[th0r]

Everyone suffering from OOM error may try --max-concurrency option - it helped me reduce memory usage during a build from ~8G to ~3.5G.