getting Microsoft.Azure.WebJobs.Script.Workers.WorkerProcessExitException in some cases when the underlying machine starts up

[tonakai]

Original ticket was created here: it is suggested that i should ask here for more details: Azure/azure-functions-host#8438

Investigative information

Please provide the following:

• Timestamp: 2022-06-07T00:11:47.881Z (i've more timestamps but this one is the most recent.)
• Function App version: 4.3.2.18186
• Function App name:
• Function name(s) (as appropriate):
• Invocation ID: 3c07d012-84f0-4437-994c-0b5079e32214
• Region: Central US

Repro steps

Provide the steps required to reproduce the problem:
i don't have exact steps to reproduce this issue, however it seems to happen during startup of each instance.

Expected behavior

Provide a description of the expected behavior.

no error

Actual behavior

Provide a description of the actual behavior observed.

i see an exception with the following call stack:

Microsoft.Azure.WebJobs.Host.FunctionInvocationException:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.33.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:353)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<TryExecuteAsync>d__18.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.33.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:108)
Inner exception Microsoft.Azure.WebJobs.Script.Workers.WorkerProcessExitException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker+<InvokeCore>d__9.MoveNext (Microsoft.Azure.WebJobs.Script, Version=4.3.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs:96)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase+<Invoke>d__24.MoveNext (Microsoft.Azure.WebJobs.Script, Version=4.3.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script/Description/FunctionInvokerBase.cs:82)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator+<Coerce>d__3`1.MoveNext (Microsoft.Azure.WebJobs.Script, Version=4.3.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script/Description/FunctionGenerator.cs:225)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2+<InvokeAsync>d__10.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.33.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs:52)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<InvokeWithTimeoutAsync>d__33.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.33.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:582)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithWatchersAsync>d__32.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.33.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:528)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.33.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:307)

they all seem to have similar pattern like this one:

I also couldn't see any errors being reported using the context since it looks like it happens before my function runs.

Known workarounds

this error happens just once per machine, we have some try logic that handles it but ideally we shouldn't see any errors.
we have about 80k requests and 10 of them failed like this in the last 24 hours.

Related information

Provide any related information

• This is a typescript/node.js project
• HttpBinding

[ejizba]

Sarah mentioned this error node exited with code -1073740791 and that error code converted to hex is 0xC0000409. Based on some googling, seems like that code means STATUS_STACK_BUFFER_OVERRUN or STATUS_FAIL_FAST_EXCEPTION

This is likely caused by something you're doing in your code, as opposed to something on our side. If your code is open source or you're comfortable sharing it here, we can take a look and see if we notice anything.

[ronkot]

We also encountered the node exited with code -1073740791 error on 2022-06-17T09:28:26.3823561Z. This was (probably) the first time we saw this error and it occurred without any changes to our code.

I'm not sure if the error is related to our own NodeJS code or the underlying Azure infra, but here some possible places in our code that could be related to the problem:

• Running deflateSync from zlib module
• Uploading data to blob using @azure/storage-blob
• Writing data to queue output binding

Unfortunately we don't have comprehensive logging so targeting the place exactly is not possible.

We're continuing investigations and will report back if we find something. Also, we can provide more information if it helps your investigation.

[ejizba]

Yeah this kind of error (where node exits quickly) is hard because you don't always get helpful stack traces and whatnot. If you can add more comprehensive logging I'm sure that would help. If the issue is happening consistently, another helpful option is to try to create a sample app with the minimum amount of code needed to reproduce.

As an aside, @hossam-nasr mentioned he has seen this error before when people use axios. Are you using axios by chance?

[ronkot]

I think I can't build a sample app as this problem occurs very rarely, once in a few months or so. We'll need to add more logging and follow this issue for more clues.

Yes, we're using axios in this function app, but not in the failing function. So, the deployment zip package does contain axios, but it's not necessarily loaded to memory in this function.. 🤔

[hossam-nasr]

@ronkot There's a good chance axios might be what's causing your app to crash. What version of axios and node are you using? For some versions, axios is known to cause node to crash with this error code. See axios issue here: axios/axios#3396 and node issue here: nodejs/node#36325

[ronkot]

@hossam-nasr thanks for pointers, those look quite interesting indeed! I just started my 3 week summer holidays but will reach back after that!

[tonakai]

so we are running on node.js 16 LTS and we do have axios dependency "axios": "^0.21.1", not directly but it looks like couple of azure libraries using it. I'll take a look if they updated.

[ronkot]

@hossam-nasr thanks for pointers, those look quite interesting indeed! I just started my 3 week summer holidays but will reach back after that!

Hi again 😎 Just checked our error logs from past 4 months and this problem have occurred only once. The problem is so rare that we won't put more effort to find out the root cause. Thanks a lot for the kind advice, I'll come back if the issue pops up again 🙏

[hossam-nasr]

@tonakai Did you check if the axios dependency was updated? Are you still seeing this error?

[tonakai]

@hossam-nasr sorry i was a bit sidetracked i looked at the last 7 days and it is still there 4 out of 500k requests failed.
one of those reported:
Exception while executing function:<function-name> node exited with code 1 (0x1) LanguageWorkerConsoleLog[error] Worker e7f07bc8-5269-472b-9601-b179deef3a94 uncaught exception (learn more: https://go.microsoft.com/fwlink/?linkid=2097909 ): Error: Connection terminated unexpectedly at Connection.<anonymous> (D:\home\site\wwwroot\node_modules\pg\lib\client.js:132:73) at Object.onceWrapper (node:events:641:28) at Connection.emit (node:events:527:28) at Socket.<anonymous> (D:\home\site\wwwroot\node_modules\pg\lib\connection.js:57:12) at Socket.emit (node:events:539:35) at TCP.<anonymous> (node:net:709:12)

but not sure if it is somehow related since this one mentioned node-pg library, but we didn't observe this error before as well.