Messages from few partitions are getting delayed

[ponsakthi]

We are using Microsoft.Azure.EventHubs.Processor 4.2.0 version on .net core 3.1 running as a pod on Openshift cluster
We have 8 pods listening to eventhub with 32 partitions.

Quite often we are seeing issues where the pod is not able to receive message from few partition where as it is able to pull messages from other partitions. The delay is as high as 10 minutes at times. This gets auto resolved and we get all the messages in burst fashion. But we don't have visibility on why is there a huge delay on few partitions. Is there a trace or log that can show us what is happening behind the scenes while polling the eventhub?

Is this the same issue that is fixed as part of #12691 in the latest version Microsoft.Azure.EventHubs.Processor 4.3.1

[Mohit-Chakraborty]

Thank you for your feedback. Tagging and routing to the team best able to assist.

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @samuelkoppes.

Issue Details

---

We are using Microsoft.Azure.EventHubs.Processor 4.2.0 version on .net core 3.1 running as a pod on Openshift cluster
We have 8 pods listening to eventhub with 32 partitions.

Quite often we are seeing issues where the pod is not able to receive message from few partition where as it is able to pull messages from other partitions. The delay is as high as 10 minutes at times. This gets auto resolved and we get all the messages in burst fashion. But we don't have visibility on why is there a huge delay on few partitions. Is there a trace or log that can show us what is happening behind the scenes while polling the eventhub?

Is this the same issue that is fixed as part of #12691 in the latest version Microsoft.Azure.EventHubs.Processor 4.3.1

Author:	ponsakthi
Assignees:	-
Labels:	Client, Event Hubs, Service Attention, customer-reported, needs-team-attention, question
Milestone:	-

[ponsakthi]

Update:
We upgraded to Microsoft.Azure.EventHubs.Processor 4.3.1 and we are seeing the same issue . But we are using EventProcessorHost (v4) and not EventProcessorClient(v5)

One more thing that we observed: Say pod A is holding the lease to partition 1. The lease ownership changes to Pod B if Pod A is not successfully renewing the least but Pod B is not pulling messages even if the ownership is with it. Later Pod A reclaims the lease back and successfully starts processing from partition 1 again. For the entire time when PodB was holding the lease no messages were processed and hence the latency. Is this a known issue in v4 client and have we addressed this in v5 client?

[serkantkaraca]

I have investigated very similar issue with another customer on K8s and that turned to be a downstream write stuck issue.

Can you please trace in ProcessEventsAsync code as in and out? See if you have a blocking call which is causing the stuck behavior. Each 'in' should have a corresponding 'out' w/ reasonable delay.