[Feature] Create new Azure metriics for Max Node Count in AKS Node Pools

[gouravsw]

Is your feature request related to a problem? Please describe.

When provisioning our AKS cluster, we configure both the minimum and maximum node counts, and rely on Azure's default autoscaler to adjust the node count as needed, up to the maximum limit, as shown in the attached screenshot.

We would like to create a monitor that triggers an alert when the node count reaches 90% (or any threshold mentioned) of the maximum node count. However, I’ve been unable to locate a relevant metric in Datadog.

Right now this metrics is not available. Checked with the Azure support team.

Describe the solution you'd like

Create a metrics for max and min node count.

azure.compute_virtualmachinescalesets.vm_max_count
azure.compute_virtualmachinescalesets.vm_min_count

Describe alternatives you've considered
For AWS clusters we are using the cluster autoscaler and it's giving the max node count (cluster_autoscaler_max_nodes_count). We have not used external autoscaler in Azure cluster as we wanted to use the default autoscaling of Azure.

Additional context
Add any other context or screenshots about the feature request here.

[gouravsw]

Few more details

What problem are you trying to solve with this feature?

We manage multiple clusters where customers run their workloads. These clusters are provisioned via Terraform with defined node counts (minimum, maximum, and desired). For example, a customer’s cluster might have the following initial configuration:

Max nodes = 25
Min nodes = 3
Desired nodes = 3

When customers add workloads that require more resources, Azure’s default autoscaler will automatically scale the nodes, allocating more capacity for workload scheduling. If the autoscaler scales to the maximum limit (e.g., Max = 25, Desired = 25, Min = 3), but the customer continues to add workloads that need more resources, the cluster will hit the max node count. As a result, additional workloads cannot be scheduled, which negatively impacts the customer experience. Customers often involve our team for troubleshooting.

To address this, we want to shift from a reactive to an active monitoring approach. Our goal is to set up a monitor that triggers a P2 alert when node capacity reaches 90% or a specified threshold, preventing workload scheduling issues before they occur.

What is the desired node count?
The minimum node count is set to 3 when the cluster is initially provisioned. As the autoscaler increases the node count, the desired node count will automatically increase by the number of nodes added by the autoscaler.
[Note:] To achieve this we need the max node count metrics for Azure clusters.

How would this feature improve your or your team’s workflow?

· Enhanced customer experience: Proactively monitoring and addressing capacity issues will improve customer satisfaction and our team's reputation.
· Active monitoring: Shifting from reactive to active monitoring reduces the need for manual intervention and troubleshooting.
· Fewer investigations: By automating alerts when node capacity is close to reaching its max, customers and our team will no longer need to investigate node scheduling issues.

How do you workaround not having this feature?

Currently, without this feature, we rely on escalations from customers when their workloads fail to schedule. Once the issue is identified as a capacity problem, we take a reactive approach, manually adjusting the maximum node count to resolve the issue.

What impact does not having this feature have on your team or organization?

The lack of this feature results in:
Damage to team reputation: Frequent customer escalations and delays in addressing issues can harm the reputation of our team and the organization.
Increased escalations: Without proactive monitoring, we receive more customer complaints and troubleshooting requests.
Poor customer experiences: This leads to negative feedback and dissatisfaction from customers, which can affect retention and trust.