Multicore usage

[improvedk]

Processing ~300 posts takes almost a minute, which is workable, but I'd like it to be much faster.

Looking at the task manager I see 13 node.exe processes (I've got 12 cores). 12 processes running hexo\lib\plugins\console\generate\worker.js, but doing nothing. And then a single process running hexo\bin\hexo g using 100% of the single core.

Am I missing something, or shouldn't hexo be able to utilize those other cores? multi_thread is set to true in my _config.yml, but I'm not sure if it's relevant.

[tommy351]

What multi_thread setting actually do is only writing files. Post rendering tasks are done in main process.

Because Node.js is single thread, I have to create a new Hexo instance if I want to move post rendering tasks to workers.

I wonder if there's a better way to handle this problem.

[improvedk]

I see - thanks for clarifying. Unfortunately my Node.js experience is limited to just a couple of Hexo plugins, as part of migrating my blog here. While the 1 min compile time is OK for now, I'll probably take a stab at it and see if I can figure out a way to distribute the work :)

[improvedk]

Tested a bit more. Loading & processing all the source posts (240) takes 7 seconds. Subsequently, 40 seconds is used on actually rendering the posts. Running process monitor I can see that all of the layouts files are loaded again and again, probably once per source post. Since all files are in memory (by the OS), CPU is the only bottleneck - but at least it narrows the bottleneck down to running the actual EJS templates, rather than processing the posts themselves.

[tommy351]

Maybe we can try to rewrite the theme with other template engines and see if this can make the rendering faster.

[improvedk]

I actually rather like the use of EJS and I think the performance is decent - so I'd much rather come up with a solution to distributing it over multiple processes. As I have 12 cores and only one is used, that'd bring the processing time down from 7+40 seconds to just 7+3 seconds, which is definitely OK.

Right now I'm working around this by just doing a generate and then running the debug server which updates on-demand, so at this point it's bit of a luxury problem, but still, interesting none the less :)

I'm not at my normal machine right now, so I can't check, but doesn't db.json store all of the posts, categories, tags, etc? What if the main process read & processed the files as it does today, wrote it all to db.json and then spun up a new process, per core, that would simply read the json and then render the posts one by one? The main process could give each post a CPU identifier so each render thread would only render posts meant for CPU 1, 2, 3, etc. Not sure if it's too hack'ish, but I'm not too fluent in Node in regards of what other process-data-sharing options there are available .

[stevenjoezhang]

Thanks for your feedback. Please join the discussion here: #4355 :)