Skip to content

Commit

Permalink
Add the blog post for the beta graduation of HPA ContainerResource ty…
Browse files Browse the repository at this point in the history
…pe metric (#39822)

* Add the blog post for the beta graduation of HPA ContainerResource type metric

* fix based on reviews

* fix based on the suggestion

* fix based on review

* update date

* Apply changes from code review

Co-authored-by: Nate W. <natew@cncf.io>

* Push back publication date and timezone

---------

Co-authored-by: Tim Bannister <tim@scalefactory.com>
Co-authored-by: Nate W. <natew@cncf.io>
  • Loading branch information
3 people authored May 2, 2023
1 parent f787489 commit 73437c2
Showing 1 changed file with 108 additions and 0 deletions.
108 changes: 108 additions & 0 deletions content/en/blog/_posts/2023-05-02-hpa-container-resource-metric.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
---
layout: blog
title: "Kubernetes 1.27: HorizontalPodAutoscaler ContainerResource type metric moves to beta"
date: 2023-05-02T12:00:00+0800
slug: hpa-container-resource-metric
---

**Author:** [Kensei Nakada](https://github.com/sanposhiho) (Mercari)

Kubernetes 1.20 introduced the [`ContainerResource` type metric](/docs/tasks/run-application/horizontal-pod-autoscale/#container-resource-metrics)
in HorizontalPodAutoscaler (HPA).

In Kubernetes 1.27, this feature moves to beta and the corresponding feature gate (`HPAContainerMetrics`) gets enabled by default.

## What is the ContainerResource type metric

The ContainerResource type metric allows us to configure the autoscaling based on resource usage of individual containers.

In the following example, the HPA controller scales the target
so that the average utilization of the cpu in the application container of all the pods is around 60%.
(See [the algorithm details](/docs/tasks/run-application/horizontal-pod-autoscale/#algorithm-details)
to know how the desired replica number is calculated exactly)

```yaml
type: ContainerResource
containerResource:
name: cpu
container: application
target:
type: Utilization
averageUtilization: 60
```
## The difference from the Resource type metric
HPA already had a [Resource type metric](/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-resource-metrics).
You can define the target resource utilization like the following,
and then HPA will scale up/down the replicas based on the current utilization.
```yaml
type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 60
```
But, this Resource type metric refers to the average utilization of the **Pods**.
In case a Pod has multiple containers, the utilization calculation would be:
```
sum{the resource usage of each container} / sum{the resource request of each container}
```

The resource utilization of each container may not have a direct correlation or may grow at different rates as the load changes.

For example:
- A sidecar container is only providing an auxiliary service such as log shipping.
If the application does not log very frequently or does not produce logs in its hotpath
then the usage of the log shipper will not grow.
- A sidecar container which provides authentication. Due to heavy caching
the usage will only increase slightly when the load on the main container increases.
In the current blended usage calculation approach this usually results in
the HPA not scaling up the deployment because the blended usage is still low.
- A sidecar may be injected without resources set which prevents scaling
based on utilization. In the current logic the HPA controller can only scale
on absolute resource usage of the pod when the resource requests are not set.

And, in such case, if only one container's resource utilization goes high,
the Resource type metric may not suggest scaling up.

So, for the accurate autoscaling, you may want to use the ContainerResource type metric for such Pods instead.

## What's new for the beta?

For Kubernetes v1.27, the ContainerResource type metric is available by default as described at the beginning
of this article.
(You can still disable it by the `HPAContainerMetrics` feature gate.)

Also, we've improved the observability of HPA controller by exposing some metrics from the kube-controller-manager:
- `metric_computation_total`: Number of metric computations.
- `metric_computation_duration_seconds`: The time that the HPA controller takes to calculate one metric.
- `reconciliations_total`: Number of reconciliation of HPA controller.
- `reconciliation_duration_seconds`: The time that the HPA controller takes to reconcile a HPA object once.

These metrics have labels `action` (`scale_up`, `scale_down`, `none`) and `error` (`spec`, `internal`, `none`).
And, in addition to them, the first two metrics have the `metric_type` label
which corresponds to `.spec.metrics[*].type` for a HorizontalPodAutoscaler.

All metrics are useful for general monitoring of HPA controller,
you can get deeper insight into which part has a problem, where it takes time, how much scaling tends to happen at which time on your cluster etc.

Another minor stuff, we've changed the `SuccessfulRescale` event's messages
so that everyone can check whether the events came from the resource metric or
the container resource metric (See [the related PR](https://github.com/kubernetes/kubernetes/pull/116045)).

## Getting involved

This feature is managed by [SIG Autoscaling](https://github.com/kubernetes/community/tree/master/sig-autoscaling).
Please join us and share your feedback. We look forward to hearing from you!

## How can I learn more?

- [The official document of the ContainerResource type metric](/docs/tasks/run-application/horizontal-pod-autoscale/#container-resource-metrics)
- [KEP-1610: Container Resource based Autoscaling](https://github.com/kubernetes/enhancements/tree/master/keps/sig-autoscaling/1610-container-resource-autoscaling)

0 comments on commit 73437c2

Please sign in to comment.