ai-k8s is experiencing large ingestion latency into AI component

[SophiaZHANG509]

Hi experts,

We noticed huge ingestion latency issue while ai-k8s App Insights sdk sending telemetry into Application Insights component. The delayed ingestion issue is occurring permanently, not temporary issue. This latency can be up to 24 hours. The version in use is ai-k8s:2.0.3.0

The delayed ingestion issue only occurs on ai-k8s App Insights SDK, while all logs capture by other App Insights SDKs are working properly without latency error.

• ai-k8s App Insights SDK:
  

• other types of App Insights SDK, including dotnet:2.22.0-997, dotnetc:2.20.0-103
  

As per below official document, since time gap is between Timestamp/TimeGenerated and _TimeReceived, and hence, the delay should happen from client side (the server where the app is hosted), there’s no latency issue in Azure App Insights backend.
https://learn.microsoft.com/en-us/azure/azure-monitor/logs/data-ingestion-time#check-ingestion-time

We have already checked the network connection between client and AI endpoints, no network error could be found.

Raising this ticket to seek for advises and insights from your ends to help us look deeper into this huge latency issue and mitigate it.
Any assistance would be really appreciated.

[xiaomi7732]

Hey @SophiaZHANG509 thanks for the reporting.

Do you actually see the delay (like the telemetry entry actually arrives late) or is it a bug on timestamps? (Like we thought it was 24 hours ago)

Is it possible to temporarily pull out application insights Kubernetes enricher and try it again?

To do that, comment out this line at the start of your application:

services.AddApplicationInsightsKubernetesEnricher();

I suspect what you see is a correlation between the issue and this SDK, but not the causation. And that will help us confirm where the issue is and thus find the right expert to investigate it for you.