Function Triggered while workers are restarting.

[ukandhallu]

hi,
We are running AZ functions with KEDA in aks.
we have 2 function which triggered by EventHub and Http Trigger respectively.

when a function is restarted for any underlying cause, it takes few secs before it is initialized. within that few secs window, we have observed for following:

1. eventhub triggers tried to execute the function for 100s of events. but failed with following exception:
   ---> System.InvalidOperationException: Did not find any initialized language workers
   this errors results in error processing 100 of events in those few seconds.

2. http trigger also has smilar issues while function is restarting, hence resulting in 500 errors.
   Executing 'Functions.function1I' (Reason='This function was programmatically called via the host APIs.', Id=9934f214-71c1-4747-b446-142c)
   Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: Functions.function1
   ---> System.InvalidOperationException: Did not find any initialized language workers
   Executed 'Functions.function1' (Failed, Id=9934f214-71c1-4747-b446-142c, Duration=1ms)

we understand why there are no initialized language worker, its due to functions are initiating. how to precent function trigger while the function is initializing?

[sajidhawk]

@soninaren and @fabiocav can u please look into these behaviors of function, and advise. Thanks

[raushan2998]

Hello @ukandhallu, Thanks for reporting this.

The error happens because triggers fire before the language worker is fully ready during a cold start or restart.
To handle this,

1. EventHub: Use retry policies and dead lettering to avoid data loss during cold starts.
2. KEDA: Set initialScale: 0 in the ScaledObject to avoid preemptive pod starts.
3. Readiness Probe: Add a readiness probe in your pod spec to delay traffic until the function app is fully initialized.

[raushan2998]

Closing the issue as there is no feedback.