NodeBB Hangs, importing 1M+, no logs, no crash, process still running but not responding, MEM 200MB RAM , CPU 100%

[akhoury]

It could be a data problem, the 1M+ number may not be relevant, but I was not able to reproduce with smaller DBs using the same -smf exporter.

• startup command NODE_ENV=development node app
• start importing 62k users, 83 categories, 73k topics, 1M+ posts
• during the import there are not no obvious memory leaks, around 200MB consistent
• CPU 100% most of the time, but that's normal
• A little while after the posts start importing, NodeBB stops responding, doesn't crash, doesn't log anything out, that's all i see
  
• after that happens
• curl localhost:4567 without a timeout does not return
• node process is still running 100% CPU, 120MB memory, and still listening on :4567

• I do not know why there is a node bcrypt process running, post creation does not call it or get near it.

[akhoury]

TODO:

• could it be process.on("uncaughtException") ?
• Need to add more logging
• Need to isolate the issue with minimal test case, currently reproduction takes over 50min each time :/
• maybe re-implement timemachine to isolate the data

[barisusakli]

The bcrypt proc is fine it gets created on app start and used whenever someone tries to login or register.

Does it still hang when you try to only import the first 700k posts? or the last 700k posts?

[barisusakli]

Also which plugins are installed and active? db search might be a problem.

[akhoury]

Does it still hang when you try to only import the first 700k posts? or the last 700k posts?

I havent tried that yet.. I am trying to find which "posts batch" is causing the issue and Im gonna skip it.

Also which plugins are installed and active? db search might be a problem.

nothing, well just markdown is activated. DB search is not active.

[akhoury]

what's interesting that it always happens at the same exact post _pid

Added more details of the post before getting created

[barisusakli]

What is the data for that post?

[akhoury]

actually it's a batch of 10 posts, so it could any one these guys:

{
    "511038": {
        "_pid": 511038,
        "_tid": 48737,
        "_timestamp": 1187119419000,
        "_subject": "Re: fREE XBOX AND PS3??? CLICK HERE",
        "_content": "____88888888:::::8::::::::::::::::88888888888888888888<br />________888_8:::888888:::::::::::::::::::::::::88888888888___888 ___________88:::::88888888::::::m::::::::::::::88888888888____8 _________888888888888888888:M::::::::::::::::8888888888888 ________88888888888888888888:::::::::::::::::M88888888888888 ________8888888888888888888888:::::::::::::M8888888888888888 _________8888888888888888888888::::::::::M888888888888888888 ________8888888888888888::88888:::::::::M88888888888888888888 ______88888888888888888:::88888::::::::M888888888888888___8888 _____88888888888888888:::88888::::::M:::::;o*M*o;888888888____88 ____88888888888888888:::8888:::::::M::::::::::::::::::88888888____8 ___88888888888888888::::88:::::::::M:;:::::::::::::::::::888888888 __8888888888888888888:::8:::::::::M::::aAa::::::::::::M8888888888_______ 8 __88___8888888888::88::::8::::::::M::::::::::::::::::::888888888888888_8 888 _88__88888888888:::8::::::::::::::M:::::::::::;::::::::88:88888888888888 888 _8__8888888888888::::::::::::::::M::::@@@@::::::::8w8888888888888888 __88888888888:888:::::::::::::::M:::::::@a@:::::::M8i888888888888888 _8888888888::::88:::::::::::::::M888:::::::::::::::::M88z888888888888888 88 8888888888:::::8::::::::::::::::M88888::::::::::::MM88888888888888888888 8 888888888:::::8::::::::::::::::M8888888MAmAMVMM88*88888888___88888888 888888_M::::::::::::::::::::::M888888888:::::::::MM8888888888888___88888 88 8888___M:::::::::::::::::::::M88888888888:::::::MM88888888888888____8888 8 _888___M::::::::::::::::::::M8888888888888M:::::mM888888888888____888 __888__M:::::::::::::::::::M8888:8888888888888:::m::Mm8888_8888___888 ___88__M:::::::::::::::::::8888:8888888888888888:::::::::Mm8___8888___88 8 ___88__M::::::::::::::::8888M::88888::888888888888::::::::::Mm8888____88&#38;#160;<br />___8___MM:::::::::::::8888M::::8888:::::888888888888::::::::::::Mm8_____ 4 _______8M::::::::::::8888M:::::::888::::::::88:::8888888::::::::::::::Mm_____2 ______88MM:::::::::8888M::::::::::88:::::::::8:::::888888:::::::::M::::: :M _____8888M:::::::::888MM::::::::::::8::::::::::::M::::8888:::::::::::M:: ::M ____88888M:::::::::88:M:::::::::::::::8:::::::::::::M:::8888:::::::::::: M::M ___88_888MM::::::888:M:::::::::::::::::::::::::::::::M:8888::::::::::::: :M: ___8_88888M::::::88::M:::::::::::::::::::::::::::::::::MM:88:::::::::::: :::::M _____88888M::::::88::M:::::::::::::::::*88*::::::::::::M:88::::::::::::: :::::::M ____888888M::::::88::M:::::::::::::::88@@88::::::::::M::88:::::::::::::: :::::::M ____888888MM::::88::MM:::::::::::::88@@88:::::::::::M::::8:::::::::::::: :::::::*8 ____88888__M::::::8::MM:::::::::::::::*88*:::::::::::::M:::::::::::::::: ::::::::::::88@<br />____8888___MM:::::::::MM::::::::::::::::::::::::::::::MM:::::::::::::::: :::::::::::88@@<br />_____888____M:::::::::::MM:::::::::::::::::::::::::::MM::M:::::::::::::: ::::::::::::888/ _____888____MM:::::::::::MMM:::::::::::::::::::::MM::::MM::::::::::::::: ::::::::::MM ______88_____M:::::::::::::MMMM::::::::::::MMMM::::::::MM::::::::::::::: :::::MMM _______88____MM:::::::::::::::MMMMMMMMMMM:::::::::::::MMM:::::::MMMMMM ________88____MM:::::::::::::::::::MMMMM::::::::::::::::::::MMMMMMMMMMII<br />_________88___8MM::::::::::::::::::::::::::::::::::::::::::::::::::MMMMM MMM __________8___88MM:::::::::::::::::::::::::::::::::M:::M:::::::::::MMM ______________888MM:::::::::::::::::::::::::::MM::::::::MM::::::MM _____________88888MM::::::::::::::::::::::MMM:::::::::mM:::::MM<br /><br />damn. it fucks up on this forum.......oh well, you get the idea.",
        "_uid": 7915,
        "_approved": 1
    },
    "511042": {
        "_pid": 511042,
        "_tid": 48737,
        "_timestamp": 1187120036000,
        "_subject": "Re: fREE XBOX AND PS3??? CLICK HERE",
        "_content": "<br /><br />[img]http://img161.imageshack.us/img161/2999/k1un4.gif[/img]<br /><br />Kanna says:<br />This forum isn&#039;t properly formatted for ASCII art.<br /><br />[img width=446 height=480]http://img65.imageshack.us/img65/2581/92315342ie3.jpg[/img]",
        "_uid": 21621,
        "_approved": 1
    },
    "511052": {
        "_pid": 511052,
        "_tid": 48737,
        "_timestamp": 1187120648000,
        "_subject": "Re: fREE XBOX AND PS3??? CLICK HERE",
        "_content": "NO U!<br /><br />...screwed up pasting :P",
        "_uid": 14,
        "_approved": 1
    },
    "511098": {
        "_pid": 511098,
        "_tid": 48737,
        "_timestamp": 1187125217000,
        "_subject": "Re: fREE XBOX AND PS3??? CLICK HERE",
        "_content": "[me=Thrawn]wakes up.[/me]<br /><br /><br />[quote author=SpliTT link=topic=48737.msg510942#msg510942 date=1187110309]<br />this is really messed up for Jurassic park <br /><br />http://www.bloody-disgusting.com/news/9638<br />[/quote]<br /><br />O.o Yeah. The best [i]Jurassic Park[/i], by far, is the [url=http://www.michaelcrichton.net/books-jurassicpark.html]book[/url].",
        "_uid": 13151,
        "_approved": 1
    },
    "511105": {
        "_pid": 511105,
        "_tid": 48737,
        "_timestamp": 1187125576000,
        "_subject": "Re: fREE XBOX AND PS3??? CLICK HERE",
        "_content": "Whoa what huh.",
        "_uid": 1733,
        "_approved": 1
    },
    "511115": {
        "_pid": 511115,
        "_tid": 48737,
        "_timestamp": 1187126328000,
        "_subject": "Re: fREE XBOX AND PS3??? CLICK HERE",
        "_content": "Credibility and exchange.",
        "_uid": 23843,
        "_approved": 1
    },
    "511123": {
        "_pid": 511123,
        "_tid": 48737,
        "_timestamp": 1187127214000,
        "_subject": "Re: fREE XBOX AND PS3??? CLICK HERE",
        "_content": "[img]http://img161.imageshack.us/img161/750/lorider1un4.gif[/img]<br />[img]http://img65.imageshack.us/img65/2180/4625772ay9.jpg[/img]<br />[img]http://www.tshirthell.com/shirts/products/a729/a729_thumb.jpg[/img]",
        "_uid": 21621,
        "_approved": 1
    },
    "511151": {
        "_pid": 511151,
        "_tid": 48737,
        "_timestamp": 1187130092000,
        "_subject": "Re: fREE XBOX AND PS3??? CLICK HERE",
        "_content": "Where&#039;s the Mr.T-shaped beef?!",
        "_uid": 23843,
        "_approved": 1
    },
    "511206": {
        "_pid": 511206,
        "_tid": 48737,
        "_timestamp": 1187132459000,
        "_subject": "Re: fREE XBOX AND PS3??? CLICK HERE",
        "_content": "saw simpsons movie today.... shit was alright &quot;sequel&quot;!!",
        "_uid": 12238,
        "_approved": 1
    },
    "511211": {
        "_pid": 511211,
        "_tid": 48737,
        "_timestamp": 1187132632000,
        "_subject": "Re: fREE XBOX AND PS3??? CLICK HERE",
        "_content": "[quote author=UrzaTheOtaku link=topic=48737.msg511123#msg511123 date=1187127214]<br />*shit*<br />[/quote]<br /><br />Stop posting shit.<br /><br />it&#039;s not funny.",
        "_uid": 26079,
        "_approved": 1
    }
}

[barisusakli]

Haha well that was easy I tried posting the first one on my dev env and it locked up the forum.

[barisusakli]

If I disable markdown I was able to post.

[barisusakli]

You are using Posts.create to import right there is no reason for that function to call postTools.parse, i will refactor that so the importer can at least work while we take a look at why md plugin breaks with that certain input.

[barisusakli]

NodeBB/NodeBB@712bd39 should be better and much faster to import as well. Let me know.

[barisusakli]

markedjs/marked#497

[akhoury]

thanks.