Execution timeout when the account has hundreds of posts

[think-in-universe]

Description:

• Execution may timeout when the user has a large number of posts.
• It's a scalability and performance issue
• The speed is too slow to build the blog for some accounts

Solution:

• Add travis_wait to avoid no log in 10 minutes issue
• Incrementally fetch and generate blogs for the user whose blog is already generated.
• speed up the static page generation by hexo
  • disable minifier
  • disable post-level sidebar widgets
  • enable caches by improve cache in theme layout and widgets
  • add tags, categories and archives pagination to 10: Disabling archive, category, tag pagination in 3.0 hexojs/hexo#1334
  • enhance the category and tag page rendering to timeline mode, example
  • add cache for article rendering if possible, with fragment_cache, or partial

[think-in-universe]

fixed by set the smart duration in production mode by checking whether the blog exists or not, and fetch blogs incrementally. This helps to dramatically reduce the time cost in daily update.

[think-in-universe]

hexo incremental build

reference:

• Beginner questions - incremental regeneration & html from data hexojs/hexo#2920
• Suggestion: let hexo support incremental build. hexojs/hexo#3269
• Is it necessary to store file content in db.json for large blog? hexojs/hexo#3271
  • https://github.com/ahuigo/a/blob/master/tool/pre-commit
• Speed up generating speed hexojs/hexo#550
• hexo 能否做到 增量 generate hexojs/hexo#2719
• 请问有没有什么方法提升更新的速度？ hexojs/hexo#823

generators

• examples
  • https://github.com/hexojs/hexo-generator-tag/tree/master/lib
  • https://github.com/dotimpact/hexo-generator-single-page

iframe widgets

• optimize iframe
  • http://www.aaronpeters.nl/blog/iframe-loading-techniques-performance?%3E
  • https://www.ernestojpg.com/2018/05/how-to-load-iframes-faster.html
  • show widgets when rendering is done: https://eager.io/blog/how-to-decide-when-your-code-should-run/
  • speed test tools
    • https://www.webpagetest.org/

hexo plugins

• processors
  • examples:
    • https://github.com/HandshakeAlliance/handshakeacademy.org/tree/b353785496e7f7647c2a6b22149508d9b035b823/scripts
    • https://github.com/distantcam/GithubSandbox/blob/ef2f04e5e22820655c005556c26b0bd1133f7c5e/themes/minimal/scripts/comments.js

find new posts

• git sparse checkout
  • http://jasonkarns.com/blog/subdirectory-checkouts-with-git-sparse-checkout/
• git diff unstaged

[think-in-universe]

hexo performance

hexo performance

• https://hexo.io/zh-cn/docs/generating
• issues
  • hexo generate slow with 400 posts hexojs/hexo#1769, fixed by fragment cache
  • caused by minifier: https://github.com/chenzhutian/hexo-all-minifier

hexo theme performance

• https://github.com/ppoffice/hexo-theme-icarus
  • cache: increase 2x by enabling cache for all widgets

memory

• increase NODE memory limit to 8GB

multi-core

• [Question] Use Hexo for generate more than 10000 posts hexojs/hexo#2579
• Multicore usage hexojs/hexo#520
• Speed up generating speed hexojs/hexo#550

references

• https://oded.blog/2017/07/26/hexo-performance-optimizations/

[think-in-universe]

understand hexo theme creation process

references:

1. http://www.codeblocq.com/2016/03/Create-an-Hexo-Theme-Part-1-Index/
2. http://www.codeblocq.com/2016/03/Create-an-Hexo-Theme-Part-2-Other-Pages/
3. http://www.codeblocq.com/2016/03/Create-an-Hexo-Theme-Part-3-Comments-Analytics-and-Widgets/

[think-in-universe]

performance test log:

• robertyan

  • 22s: normal
  • 36s: try to disable tags and categories, with previous posts by andrewma
  • 21s: try to disable tags and categories, with previous posts by robertyan
  • 24s, 21s: try to disable tags and categories, run hexo clean before build
  • 17s: disable tags and categories pagination
  • 23s, 25s: with pagination of 6
  • 18s, 21s: with pagination of 10
  • 27s: _partial cache removed, with pagination of 10
  • 27s, 28s: _partial cache removed, with pagination of 6
  • 39s: _partial cache removed, with pagination of 6, add sort categories
  • 22s, 31s: pagination = 6, sort categories and tags, add tag and category cache
  • 19s: disable tags and categories pagination, sort categories and tags, add tag and category cache
  • 87s, 50s: when disk and CPU are very busy
  • 17/19/22s: cache={widget: tag/category; layout: head/footer/script}, sort={categories, tags}, pagination=6
  • 25s: cache={widget: none; layout: head/footer/script}, sort={categories, tags}, pagination=6
  • 15/16s: cache={widget: all; layout: head/footer/script}, sort={categories, tags}, pagination=6
  • 12/13s: cache={widget: all; layout: head/footer/script}, format=timeline, pagination=10
  • 12/13s: cache={widget: all; layout: head/footer/script, component:item}, format=timeline, pagination=10
  • 8/10/11s: cache={widget: +profile; as above}
  • 7/8/14s: iframe=true, +above

• andrewma

  • 6.43 min: with cache
  • 8.05 min: remove archive widget
  • 3.98 min: disable tags and categories pagination
  • 3.63 min: cache={widget: all; layout: head/footer/script}, sort={categories, tags}, pagination=6
  • 3.45 min: cache={widget: all; layout: head/footer/script, component:item}, format=timeline, pagination=10
  • 3.4 min: cache={widget: all; layout: head/footer/script}, format=timeline, pagination=10
  • 2.47 min: cache={widget: all; layout: head/footer/script}, format=timeline, pagination=20
  • 2.68 min: cache={widget: all; layout: head/footer/script; component: item}, format=timeline, pagination=20
  • 47/48s: pjax=true, +above
  • 16s: incremental

analysis:

• by checking timestamp of cache, we know that cached_item works

incremental build test:

• 2-days delta: 327 files generated, ~3 min
• 1-day delta: 324 files generated, 3.22 min
• no change: 731 files generated, 2.46 min

[think-in-universe]

incremental build implementaion:

• diff files with previous build
• guess the files to generate based on new state
  • smart generation
    • archives
      • archive of this month: [change: yes]
      • archive of previous month: [change: no] --> quite some
    • index
      • all because the entire list moved: [change: yes]
    • tags
      • tags of the posts: [change: yes]
      • remaining tags: [change: no] --> not much --> widget iframe
    • categories:
      • categories of the posts: [change: yes]
      • remaining categories: [change: no] --> not much --> widget iframe
    • index.html: [change: yes]
    • widgets: [change: yes]
    • posts:
      • the latest posts: [change: yes]
      • the old posts: [change: no] --> quite some
  • use /data
    • need investigation ==> no need
  • widget as iframe
    • reduce regeneration of all the categories and tags
    • generate iframe widgets --> widgets/iframe
    • include iframes
    • display iframes as widgets correctly
  • skip later process if no page loaded or no pages are different
    • what if some posts are updated? ==> diff will find
    • compare the MD files with git diff
  • trigger travis build via command line 'travis'
    • https://docs.travis-ci.com/user/triggering-builds/
• build a dynamic site instead: so it's not hexo based
  • understand steempress /cnstm.org : blog --> steem, no opposite direction
• questions:
  • how to let tags/categories/archives/posts page know that which needs to be changed?
    • how to compare with source and figure out the diff and which pages to change?
    • implementation steps:
      1. create a branch / repo for storing 'source', clone the branch into local before download
      2. use git command to find the markdown files that need to be updated
      3. use hexo APIs to fetch the [posts, tags, categories, and archives] to be updated
      4. update generators for [posts, tags, categories and archives]
      5. write the latest source into the 'source' branch or repo
  • how to write iframe_widget_src into widgets/.html?
  • how to implement robust iframe widgets, by supporting both iframe and ejs widgets?
  • the user expereince issue with the iframe widgets => use pjax widgets
    • load performance / network
      • speed up iframe loading
        • modify the iframe and other sources' loading orders
          • start too late, later than images
          • loading resources started too late, 2s after HTML loaded
        • iframe and bundle.js are loaded at the latest time => trigger rendering at the right time, after #content is rendered
        • remove bundle.js and bundle.css if needed => ignore
    • rendering sytle / auto-resize
      • guess the width of iframe body
      • set the width of iframe body at the earliest possible moment
      • cache iframe at client to speed up rendering
    • replace iframe based solution to pjax/ajax based solution
      • use pjax to build the components
  • how to enalbe multi-core building of hexo
  • user experience issus
    • tag page loading widget is slow ==> ajax is later than image loading
    • post previous / next post button is wrong
    • Netlify build failure