Speed up generating speed

[tommy351]

Generating speed is slow when we have mass files. It may take 10 minutes to build a blog with 300+ posts.

Most time is consuming by file rendering. Every file needs 500ms to render on average (depends on the quantity of posts).

• Generate files right after routes are updated
• Optimize rendering process Fragment cache #637
• Optimize database

[chadkoh]

Yep, this is pretty brutal. My blog has 397 posts. It takes 2.5 hours to render!

839 files generated in 8767.078s

[wzpan]

FYI, have you ever tried other renderers? Do they render files slowly, too?

[chadkoh]

I am currently using Octopress and just converted over to hexo this weekend. Octopress takes like 15 seconds to render my site. I started testing hexo on 2.3 and it was fast, so I was excited. I finally got around to doing the conversion this weekend and it was a brutal experience. I hope this issue gets solved... it is a bit of a barrier to entry.

[tommy351]

In commit 80da498, I moved rendering process to workers, which can make use of all cores of CPU. But it wastes lots of memory.

Plus, I found the elapsed time decreased a lot if we don't use categories/tags. The bottleneck may be the data model?

[kamenitxan]

I tried jade&less theme (https://github.com/widatama/hexo-jade-barebone) and it generates my blog in 8 seconds, down from 3 minutes with default theme.

I guess that hexo-renderer-ejs needs optimalization, not Hexo.

[tommy351]

@kamenitxan I think the problem is not the template engine. It's because this theme doesn't have widgets. The default theme has many widgets enabled by default which takes lots of time to query.

[tommy351]

Because it's more difficult to optimize data model, I came up with another idea: Fragment Cache #637

[yimt]

在hexo建立稳定之后，都是往里面添加新的文章，有没有可能在generate的时候只generate一些必要的页面，比如新的文章、archives、sitemap之类的？

[Xuanwo]

每次运行的时候都diff一下，然后生成有变动的网页？

不知道这样搞会不会反而速度更慢= =

[rose1988c]

hexo server & 在后台运行的时候，hexo g 非常卡

[Xuanwo]

@rose1988c 不用hexo server的时候卡不卡呢？

[rose1988c]

@Xuanwo

// 在 hexo server在后台运行的情况，hexo g 卡主了，cpu爆满，直接被系统Killed
root@hexo:/opt/www/rose1988c.github.io# hexo g

INFO  Generated: archives/2014/03/index.html
INFO  Generated: archives/2014/09/index.html
INFO  Generated: archives/2014/10/index.html
......
INFO  Generated: index.html
INFO  Generated: page/2/index.html
Killed

//  下面kill hexo server 再 hexo g
root@hexo:/opt/www/rose1988c.github.io# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  20260     0 ?        Ss+  Nov19   0:00 /bin/bash
root        20  0.0  0.1  20268  1588 ?        Ss   Nov19   0:00 /bin/bash
root       352  5.3 28.9 1295944 293088 ?      Sl   09:42   0:47 hexo                           
root       396  0.0  0.1  17492  1964 ?        R+   09:56   0:00 ps aux

root@hexo:/opt/www/rose1988c.github.io# kill 352

root@hexo:/opt/www/rose1988c.github.io# hexo g

INFO  Files loaded in 24 s
INFO  Generated: tags/index.html
INFO  Generated: link/index.html
......
INFO  Generated: tags/java/index.html
INFO  Generated: tags/maven/index.html
INFO  220 files generated in 20 s
[1]+  Terminated              hexo server
// 成功正常跑完

[tommy351]

Why you run hexo generate and hexo server at the same time?

[rose1988c]

@tommy351 我相信有部分人因为这个hexo server，卡死在hexo g上面。

因为我是 127.0.0.1:4000 和 github gh-pages 是一个负载均衡，所以hexo server 是一直运行着的

[Xuanwo]

@rose1988c 我还是认为各司其职比较好，测试用hexo server，拿来访问&作负载均衡还是用apache或者nginx

[rose1988c]

@Xuanwo 是nginx做的......

俺是看到上面的说hexo g 卡死，我发了这个我碰到的，并提醒类似的小伙伴~

[Xuanwo]

@t3shi 我也不知道- -，或许你可以把分页关掉试试。。

[tl3shi]

@Xuanwo 你那个travis-ci提供了编译环境？免费帮忙编译部署到github？

[Xuanwo]

@tl3shi 持续集成，你可以自行google了解一下~

[leesei]

@tl3shi This PR has been hung for a long time.
Please join #1769 for the latest thread.

[mnlbox]

+1 for this improvement. 👍

[rbarilani]

@tommy351 is this project still alive?

[ghost]

@rnarilani When you say "project" are you referring to Hexo, or this issue specifically?

[rbarilani]

@JHabdas The hexo project, I just post it here since will probably open PR to enhance the project but the amount of open PR and their dates are making us doubt a little bit.

[NoahDragon]

@rbarilani Yes, it is alive. @tommy351 is busy, and I'm the current maintainer for the project. Feel free to submit any issues and PRs. 😄

[ghost]

@rbarilani if ever in doubt just look at the GitHub pulse and other metrics. Or just look at the date of the last closed issue(s) for bigger projects. @NoahDragon has been doing a bang up job triaging. Lucky for him this isn't the Yarn repo! xD

[rbarilani]

@JHabdas thanks and good to know.

[ghost]

@rbarilani My pleasure. Just for you my friend (kidding, it's actually for a blog post) I've put together frequency charts for Hugo, Hexo and Jekyll. And here they are:

[rbarilani]

@JHabdas Can we use gitter chat for questions? I'm building a documentation theme on top of hexo and I have and I will have some questions.

[JLHwung]

@JHabdas I don't mean to close this PR. I apologize if it seems confusing.

What happens is that this PR's original base is dev branch, which is a stale merged branch for a long time. So I deleted this dev branch and Github notice that the dev branch is deleted and closesd this PR.

I just change the base to master and reopen this PR. Thank you for your contributions.

[Zhang21]

我也有这个问题，随着文章数量和文章字数的不断增加，hexo g所需的时间便不断增加。

有没有办法让hexo只生成新的post，因为老的文章已经生成和发布，便不需要再生成，这样也可以节约时间。如加入一个diff之类的命令？

[ghost]

LGTM

[LbhFront-end]

你好 想问一下这个问题解决了吗，我还没有升级到最新的hexo，现在135篇文章,728.3k字数，放到githubActions编译需要5分钟，想问一下是不是需要升级hexo版本可以解决这个编译时间长的问题

[SukkaW]

你好 想问一下这个问题解决了吗，我还没有升级到最新的hexo，现在135篇文章,728.3k字数，放到githubActions编译需要5分钟，想问一下是不是需要升级hexo版本可以解决这个编译时间长的问题

@LbhFront-end

请至少升级到 Hexo > 4.0 版本，推荐升级到 Hexo 6.0.0。