Skip to content

[BUG] Defender Pods Fail to Read Memory Metrics Due to cgroup v2 Change After Upgrade from Kubernetes 1.27 to 1.30 #4629

Open
@VincentSchmid

Description

@VincentSchmid

Describe the bug
From Sept 4 Defender pods stopped to read memory metrics because of changes cgroup to V2. (upgrade kubernetes from 1.27 to 1.30)

Now microsoft-defender-collector-ds pods spams:

level=error msg="Failed to get max memory usage with error: open /sys/fs/cgroup/memory.peak: no such file or directory"
level=error msg="Failed to get memory usage with error: open /sys/fs/cgroup/memory.current: no such file or directory"
level=info msg="Heartbeat: {\"Performance\":{\"Memory\":\"11Mi\"}}"

microsoft-defender-publisher-ds spams:

level=error msg="Failed to get max memory usage with error: open /sys/fs/cgroup/memory.peak: no such file or directory"
level=info msg="Heartbeat: {\"Cloud Native Identifier\":\"\",\"Events\":[],\"Kernel Version\":\"5.15.0-1073-azure\",\"Performance\":{\"CurrMemoryUsageInMb\":96.44921875,\"MaxMemoryUsageInMb\":-1}}"

Expected behavior
Reading the correct system files

Environment (please complete the following information):

  • Kubernetes version 1.30.5
  • Pod image - mcr.microsoft.com/azuredefender/stable/pod-collector:1.0.110
  • Pod image - mcr.microsoft.com/azuredefender/stable/security-publisher:1.0.148
  • Kubernetes node image - AKSUbuntu-2204gen2containerd-202410.15.0
  • Kubernetes kernel version - 5.15.0-1073-azure

Additional context
Add any other context about the problem here.

Activity

changed the title [BUG] Azure Defender for Containers memory metrics reading error [BUG] Defender Pods Fail to Read Memory Metrics Due to cgroup v2 Change After Upgrade from Kubernetes 1.27 to 1.30 on Nov 6, 2024
microsoft-github-policy-service

microsoft-github-policy-service commented on Nov 6, 2024

@microsoft-github-policy-service
Contributor

@miwithrow, @CocoWang-wql would you be able to assist?

Jon-Hawkins

Jon-Hawkins commented on Nov 11, 2024

@Jon-Hawkins

I am also having this issue. Any ideas?

mmillard0

mmillard0 commented on Dec 11, 2024

@mmillard0

We have the same issue with AKS 1.30.5. Any update ?

microsoft-github-policy-service

microsoft-github-policy-service commented on Dec 16, 2024

@microsoft-github-policy-service
Contributor
microsoft-github-policy-service

microsoft-github-policy-service commented on Dec 31, 2024

@microsoft-github-policy-service
Contributor

Issue needing attention of @Azure/aks-leads

microsoft-github-policy-service

microsoft-github-policy-service commented on Jan 15, 2025

@microsoft-github-policy-service
Contributor

Issue needing attention of @Azure/aks-leads

microsoft-github-policy-service

microsoft-github-policy-service commented on Jan 31, 2025

@microsoft-github-policy-service
Contributor

Issue needing attention of @Azure/aks-leads

microsoft-github-policy-service

microsoft-github-policy-service commented on Feb 15, 2025

@microsoft-github-policy-service
Contributor

Issue needing attention of @Azure/aks-leads

microsoft-github-policy-service

microsoft-github-policy-service commented on Mar 2, 2025

@microsoft-github-policy-service
Contributor

Issue needing attention of @Azure/aks-leads

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      [BUG] Defender Pods Fail to Read Memory Metrics Due to cgroup v2 Change After Upgrade from Kubernetes 1.27 to 1.30 · Issue #4629 · Azure/AKS