HorizontalPodAutoscaler for envoy proxy

[gitanuj]

Description:
Does envoy gateway have any design to scale out envoy proxy pods? Are there any best practices on how to handle increased number of requests? I'm mostly looking at deploying this to clouds like AWS/GCP/Azure which will allocate application/network Load Balancers but then how can I make sure that envoy proxy pod is not the bottleneck in the gateway and can scale out when needed.

[danehans]

HPA is currently not supported by Envoy Gateway. It's a feature that would be beneficial to the project, so I've added it to the backlog. Let me know if you're interested in working on this and I can assign it to you.

I'm mostly looking at deploying this to clouds like AWS/GCP/Azure which will allocate application/network Load Balancers...

Envoy Gateway currently creates an ELB when running in AWS. #648 intends to design an API for providing configurability to the Envoy service, PTAL, and comment as needed.

[gitanuj]

1. Is there any design doc explaining which pods are involved in the data path for EG? I'm trying to understand if there are any other components other than EnvoyProxy which need to be scaled out as well.
2. Before implementing HPA I wanted to understand if I can simply update the number of replicas here. Is there an API through which this can be exposed?

[danehans]

Is there any design doc explaining which pods are involved in the data path for EG? I'm trying to understand if there are any other components other than EnvoyProxy which need to be scaled out as well.

Only the Envoy proxy deployment needs to be scaled. The design docs are the best place for this type of info, specifically the system design and config API design docs.

Before implementing HPA I wanted to understand if I can simply update the number of replicas here. Is there an API through which this can be exposed?

You can update this field but EG will revert it back to the desired state. This field should be configurable using the EnvoyProxy config API. If you would like to make this field configurable, please create an issue that describes your use case and xref this issue.

[arkodg]

cc @qicz in case you're interested in adding this to EnvoyProxy resource :)

[qicz]

cc @qicz in case you're interested in adding this to EnvoyProxy resource :)

I am sorry for the late reply.

I think we just need to provide the same docs about HPA. current envoy deployment we can configure more fields and #1398 support affinity etc settings.

gateway/api/config/v1alpha1/shared_types.go

Lines 45 to 119 in 4e43170

	// KubernetesDeploymentSpec defines the desired state of the Kubernetes deployment resource.
	type KubernetesDeploymentSpec struct {
	// Replicas is the number of desired pods. Defaults to 1.
	//
	// +optional
	Replicas *int32 `json:"replicas,omitempty"`
	// Pod defines the desired annotations and securityContext of container.
	//
	// +optional
	Pod *KubernetesPodSpec `json:"pod,omitempty"`
	// Container defines the resources and securityContext of container.
	//
	// +optional
	Container *KubernetesContainerSpec `json:"container,omitempty"`
	// TODO: Expose config as use cases are better understood, e.g. labels.
	}
	// KubernetesPodSpec defines the desired state of the Kubernetes pod resource.
	type KubernetesPodSpec struct {
	// Annotations are the annotations that should be appended to the pods.
	// By default, no pod annotations are appended.
	//
	// +optional
	Annotations map[string]string `json:"annotations,omitempty"`
	// SecurityContext holds pod-level security attributes and common container settings.
	// Optional: Defaults to empty. See type description for default values of each field.
	//
	// +optional
	SecurityContext *corev1.PodSecurityContext `json:"securityContext,omitempty"`
	// NodeSelector is a selector which must be true for the pod to fit on a node.
	// Selector which must match a node's labels for the pod to be scheduled on that node.
	// More info: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
	// +optional
	// +mapType=atomic
	NodeSelector map[string]string `json:"nodeSelector,omitempty"`
	// If specified, the pod's scheduling constraints.
	// +optional
	Affinity *corev1.Affinity `json:"affinity,omitempty"`
	// If specified, the pod's tolerations.
	// +optional
	Tolerations []corev1.Toleration `json:"tolerations,omitempty"`
	}
	// KubernetesContainerSpec defines the desired state of the Kubernetes container resource.
	type KubernetesContainerSpec struct {
	// List of environment variables to set in the container.
	//
	// +optional
	Env []corev1.EnvVar `json:"env,omitempty"`
	// Resources required by this container.
	// More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
	//
	// +optional
	Resources *corev1.ResourceRequirements `json:"resources,omitempty"`
	// SecurityContext defines the security options the container should be run with.
	// If set, the fields of SecurityContext override the equivalent fields of PodSecurityContext.
	// More info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
	//
	// +optional
	SecurityContext *corev1.SecurityContext `json:"securityContext,omitempty"`
	// Image specifies the EnvoyProxy container image to be used, instead of the default image.
	//
	// +optional
	Image *string `json:"image,omitempty"`
	}

[qicz]

example

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: envoy-demo
    app.kubernetes.io/name: envoy-demo
    app.kubernetes.io/part-of: envoy-demo-ns
  name: envoy-demo
  namespace: envoy-demo-ns
spec:
  progressDeadlineSeconds: 600
  replicas:  1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/component: envoy-demo
      app.kubernetes.io/name: envoy-demo
      app.kubernetes.io/part-of: envoy-demo-ns
  template:
    metadata:
      labels:
        app.kubernetes.io/component: envoy-demo
        app.kubernetes.io/name: envoy-demo
        app.kubernetes.io/part-of: envoy-demo-ns
    spec:
      serviceAccountName: envoy-demo-sa
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node-role.kubernetes.io/master
                    operator: In
                    values:
                      - ""
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/component: envoy-demo
                    app.kubernetes.io/name: envoy-demo
                    app.kubernetes.io/part-of: envoy-demo-ns
                namespaces:
                  - envoy-demo-ns
                topologyKey: kubernetes.io/hostname
              weight: 100
      containers:
        - name: envoy-demo
          image: envoy-demo:latest
          imagePullPolicy: IfNotPresent
          resources:
            requests:
              cpu: 10m
              memory: 64Mi
            limits:
              cpu: 1
              memory: 1024Mi
      tolerations:
      - effect: "NoSchedule"
        key: "node-role.kubernetes.io/master"
        operator: "Exists"

---
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: test-hpa
  namespace: envoy-demo-ns
spec:
  maxReplicas: 3
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: envoy-demo
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 80

[arkodg]

thanks for outlining this @qicz, looks like we just need to outline this using docs, and no API changes are needed

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[arkodg]

@qicz are you planning on working on this one for v0.6.0-rc1 ?

[ardikabs]

hi @arkodg @qicz

is there any update on this matter? I could help to work on this. However, I need some advice on how we should support HPA for Envoy Proxy deployment.

Just for a context, I've applied a patch to Envoy Gateway at my organization to make it compatible with HPA when enabled externally. This patch allows EG to determine when the EnvoyProxy spec sets the number of replicas to 0, effectively making EG disregard the replicas field in the Envoy Proxy Deployment. However, we think that this implicit solution might not align with the community's goals.

[arkodg]

hey @ardikabs assigned this issue to you for now, thanks for volunteering !
I think disabling the replica field by default in the deployment is fine

the question I have is - should EG create the HPA resource or should the end user create one and link it to the generated Envoyproxy deployment

[ardikabs]

IMO, it would be better if EG also managed the HPA config with introduce another field such as hpaSpec like what Istio did on controlling their gateway through IstioOperator API. So that the existence of the field will override the replicas field on EnvoyProxy API for kubernetes provider. Thoughts @arkodg?

[arkodg]

ptal @envoyproxy/gateway-maintainers, please share your thoughts on adding envoyHpa field into the EnvoyProxy resource

[zirain]

keep eye on kubernetes-sigs/gateway-api#1355

[gazal-k]

That part that confuses me is that I see it is possible to control the envoy deployment using the EnvoyProxy object, like: https://github.com/envoyproxy/gateway/blob/main/examples/kubernetes/envoy-proxy-config.yaml. However, the helm chart does not seem to do this, instead defining minimal config;

gateway/charts/gateway-helm/values.tmpl.yaml

Lines 26 to 34 in dcc0ee0

	config:
	envoyGateway:
	gateway:
	controllerName: gateway.envoyproxy.io/gatewayclass-controller
	provider:
	type: Kubernetes
	logging:
	level:
	default: info

and defining the Deployment object separately; https://github.com/envoyproxy/gateway/blob/main/charts/gateway-helm/templates/envoy-gateway-deployment.yaml

This makes me think, we can introduce opt-in autoscaling configuration similar to: https://github.com/kubernetes/ingress-nginx/blob/7f723c59855e82614582ff7b2efd1783b1afc2ee/charts/ingress-nginx/values.yaml#L367-L373

[shawnh2]

hi @gazal-k, seems you have a little misunderstanding here.

the deployment in helm chart is what we used to start envoy-gateway controller, the EnvoyProxy deployment is reconciled by this controller:

gateway/internal/infrastructure/kubernetes/deployment.go

Line 120 in 7c6c37a

deployment := &appsv1.Deployment{

the values in this yaml is used by the controller:

gateway/charts/gateway-helm/values.tmpl.yaml

Lines 27 to 29 in dcc0ee0

	envoyGateway:
	gateway:
	controllerName: gateway.envoyproxy.io/gatewayclass-controller

---

but https://github.com/kubernetes/ingress-nginx/blob/7f723c59855e82614582ff7b2efd1783b1afc2ee/charts/ingress-nginx/values.yaml#L367-L373 seems a good example to define hpaSpec.

[gazal-k]

Thanks for that. So, the Deployment object in the chart is the controller. The controller then creates another Deployment object for the envoy proxy fleet?

[shawnh2]

Thanks for that. So, the Deployment object in the chart is the controller. The controller then creates another Deployment object for the envoy proxy fleet?

correct

[arkodg]

hey @ardikabs are you planning on working on this one ?

[ardikabs]

Yes @arkodg, I am still working on it

[shawnh2]

closed in favor of #2257