Camaro in v8.10.0 Lambda intermittent crashes

[clocked0ne]

Hi,

I have a Lambda executing Node v8.10.0 code which is processing XML responses in near realtime from an API (millions per day). It has been executing perfectly until this past week when the Lambda function has suddenly started intermittently crashing with Process exited before completing request. I doubt the Lambda function is running out of memory: Memory Size: 192 MB Max Memory Used: 75 MB

I have managed to pinpoint this crashing to the Camaro call, which is this in isolation:

const parseXML = require('camaro');
const xmlTemplate = {
	title: '//JD/NeutralForm',
	disciplines: [
		'//JD/Discipline[@language="en"]', '.'
	],
	subDisciplines: [
		'//JD/SubDiscipline[@language="en"]', '.'
	]
};

let result;
try {
	result = parseXML(xml, xmlTemplate);
} catch (err) {
	console.log(err);
}
console.log(result.title, result.disciplines, result.subDisciplines);

Sample of the typical XML when it has failed:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<NormalisationResult>
    <Source><![CDATA[FP&A Accountant]]></Source>
    <NormalisationJDs>
        <JD idGlobalStrikt="1000193383" idnorm="393530" language="en" normbyderivat="0" nosuggest="0" ortho="0" type="JD">
            <NeutralForm><![CDATA[FP&A Accountant]]></NeutralForm>
            <GenderForm gender="M"><![CDATA[FP&A Accountant]]></GenderForm>
            <UsedSource><![CDATA[FP&A Accountant]]></UsedSource>
            <Discipline discipline="Accountancy" id="251169" language="en" subDiscipline="Accountancy"><![CDATA[Accountancy]]></Discipline>
            <SubDiscipline discipline="Accountancy" id="251169" language="en" subDiscipline="Accountancy"><![CDATA[Accountancy]]></SubDiscipline>
            <SynonymJDs>
                <idGlobalStrikt>1000193383</idGlobalStrikt>
            </SynonymJDs>
            <Hypernyms>
                <idGlobalStrikt>1000185261</idGlobalStrikt>
            </Hypernyms>
            <AllHypernyms>
                <idGlobalStrikt>1000185261</idGlobalStrikt>
                <idGlobalStrikt>1000198579</idGlobalStrikt>
                <idGlobalStrikt>170502</idGlobalStrikt>
                <idGlobalStrikt>182350</idGlobalStrikt>
                <idGlobalStrikt>196669</idGlobalStrikt>
            </AllHypernyms>
        </JD>
    </NormalisationJDs>
    <RemainingSource/>
</NormalisationResult>

I cannot see an obvious reason nor why any errors are not being caught by the catch. I have tried switching to manual install as part of Jenkins build instead of using the pre-built binary in Releases, and it has made no difference. Not all requests fail. Any ideas?

[tuananh]

I cannot see an obvious reason nor why any errors are not being caught by the catch.

That's unfortunate. I don't see any reason either. If you can find a way to reproduce the error consistently, maybe i can help.

[tuananh]

i wrote a quick test script to run your xml string with the above script 1 million times but it doesn't crash on my local machine.

[clocked0ne]

Thanks for looking at this so quickly. It doesn't crash for me locally either in my own test suite using the same data in a test fixture, the problem is definitely isolated to something around the Lambda environment :(

I will probably have to switch this out for another library in the interim since it is a production system; until I can get to the bottom of it.

[clocked0ne]

I think the only additional thing I could ask you to look at here is if the errors are being swallowed in a similar way to the post here:
nodejs/node#2380

I don't know enough about how native-addons work to dig into that side of things.

[tuananh]

Maybe it's something to do with N-API. it's experimental for a reason :(

nodejs/node#19437 (comment)

camaro v3 is using N-API. camaro v2 is using NAN. Maybe you can try camaro v2

There's no performance difference between v2 and v3. It's just that I want to learn about N-API

[clocked0ne]

I can give it a go, are there any fundamental differences between v2 and v3 that I should be aware of?

[tuananh]

@clocked0ne iirc no major changes. v3 may add 1 or 2 xpath functions more but that's it. I just want to use N-API that's all :D N-API is no longer experimental in Node 10 which is going into LTS in October. Just wanna update a step ahead.

[clocked0ne]

Great, I will build with v2 and see how I get on, hopefully it works fine aginast all my tests and then I just have to wait for Node 10 to go LTS and AWS to update Lambda support for it before trying v3 again.

[tuananh]

Great. Lemme know if it works / not works for you.

…

On Mon, Aug 20, 2018 at 4:56 PM clocked0ne ***@***.***> wrote: Great, I will build with v2 and see how I get on, hopefully it works fine aginast all my tests and then I just have to wait for Node 10 to go LTS and AWS to update Lambda support for it before trying v3 again. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#43 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAmSTmjmy8COPPTc-m0ucozLgXpzWq2Wks5uSofjgaJpZM4WDjVP> .

[clocked0ne]

Just to keep you updated, I tried v2 and in fact it seemed to crash out even more frequently; though I noticed one single Lambda process was running successfully continuously. So, my suspicion is that its the environment not behaving well when multiple Lambdas are executed for the same function using the native-addon.

I've had to switch to using fast-xml-parser and now all the processes are running in parallel successfully. Hopefully this is something that crops up elsewhere and can be fixed, as I would like to return to camaro for sure, seeing as it is more performant and the templating structure makes for lean, readable code.

I also find it strange that this only became a problem in the last week or so, maybe AWS changed something in their infrastructure, as I have not modified the lambda code itself in some time until now.

[tuananh]

@clocked0ne ah too bad. I don't know how to debug it in Lambda either :(

I'm gonna leave this one issue open as it's unresolved so that others can see it.

[tuananh]

Closing with #64