Autoscaler modifies ASG resources that it shouldn't

[orkhanM]

On a kube-aws ( version v0.9.8 ) cluster with cluster autoscaler addon enabled I noticed the autoscaler was scaling down ASG's unrelated to the cluster.

The autoscaler manifest looks like:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: cluster-autoscaler
  namespace: kube-system
  labels: 
    app: cluster-autoscaler
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cluster-autoscaler
  template:
    metadata:
      labels: 
        app: cluster-autoscaler
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:   
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: "kube-aws.coreos.com/cluster-autoscaler-supported"
                operator: "In"
                values: 
                - "true"
      tolerations:
      - key: "node.alpha.kubernetes.io/role"
        operator: "Equal" 
        value: "master"
        effect: "NoSchedule"
      - key: "CriticalAddonsOnly"
        operator: "Exists"
      containers:
        - image: gcr.io/google_containers/cluster-autoscaler:v0.6.0
          name: cluster-autoscaler
          resources:
            limits: 
              cpu: 100m
              memory: 300Mi
            requests:
              cpu: 100m
              memory: 300Mi
          command:
            - ./cluster-autoscaler
            - --v=4 
            - --stderrthreshold=info
            - --cloud-provider=aws
            - --skip-nodes-with-local-storage=false
            - --expander=least-waste
            - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,kubernetes.io/cluster/production-k8s-02
          env:    
            - name: AWS_REGION
              value: us-east-1
          imagePullPolicy: "Always"

An excerpt from the cluster-autoscaler logs:

I1009 22:20:48.400952       1 leaderelection.go:204] successfully renewed lease kube-system/cluster-autoscaler\
I1009 22:20:48.662164       1 auto_scaling_groups.go:94] Regenerating ASG information for CoreUpdate-ASG-1TFRMDY7ZBDXB\
I1009 22:20:48.814109       1 auto_scaling_groups.go:94] Regenerating ASG information for Jenkins-Linux-HCM-ASG-CXZPKHJBVBMC\
I1009 22:20:48.848760       1 auto_scaling_groups.go:94] Regenerating ASG information for Jenkins-Linux-HCM-ASG-Spot\
I1009 22:20:48.893549       1 auto_scaling_groups.go:94] Regenerating ASG information for Jenkins-Linux-Slaves-ASG-KW2EKO1HQGSY\
I1009 22:20:48.939883       1 auto_scaling_groups.go:94] Regenerating ASG information for Jenkins-Windows-OPS01-ASG-KA5DIDGLDP5B\
I1009 22:20:49.000438       1 auto_scaling_groups.go:94] Regenerating ASG information for Production-vpc-00fd2379-Services01-ASG-UAFQVWLSRY6Z\
I1009 22:20:49.064799       1 auto_scaling_groups.go:94] Regenerating ASG information for Production-vpc-00fd2379-Services02-ASG-1VMX2FTPJ1ZT5\
I1009 22:20:49.115805       1 auto_scaling_groups.go:94] Regenerating ASG information for Production-vpc-00fd2379-Web01-ASG-1N4JRPQU0MYZ2\
I1009 22:20:49.161246       1 auto_scaling_groups.go:94] Regenerating ASG information for Production-vpc-00fd2379-Web02-ASG-1ANYJ6OIK5C2\
I1009 22:20:49.196010       1 auto_scaling_groups.go:94] Regenerating ASG information for Registry-Example-Com-AutoScalingGroup-FG40W31U9ZHL\
I1009 22:20:49.236423       1 auto_scaling_groups.go:94] Regenerating ASG information for Registry-Example-Com-CLAIR-AutoScalingGroup-Y0JX9W3VA29A\
I1009 22:20:49.275025       1 auto_scaling_groups.go:94] Regenerating ASG information for looker-nodes.production-k8s-01.example.com\
I1009 22:20:49.316710       1 auto_scaling_groups.go:94] Regenerating ASG information for looker-nodes.production-k8s.example.com\
I1009 22:20:49.360140       1 auto_scaling_groups.go:94] Regenerating ASG information for master-us-east-1a.masters.production-k8s-01.example.com\
I1009 22:20:49.433501       1 auto_scaling_groups.go:94] Regenerating ASG information for master-us-east-1a.masters.production-k8s.example.com\
I1009 22:20:49.471324       1 auto_scaling_groups.go:94] Regenerating ASG information for master-us-east-1b.masters.production-k8s-01.example.com\
I1009 22:20:49.501557       1 auto_scaling_groups.go:94] Regenerating ASG information for master-us-east-1b.masters.production-k8s.example.com\
I1009 22:20:49.569452       1 auto_scaling_groups.go:94] Regenerating ASG information for master-us-east-1c.masters.production-k8s-01.example.com\
I1009 22:20:49.707034       1 auto_scaling_groups.go:94] Regenerating ASG information for master-us-east-1c.masters.production-k8s.example.com\
I1009 22:20:49.746283       1 auto_scaling_groups.go:94] Regenerating ASG information for master-us-east-1d.masters.production-k8s-01.example.com\
I1009 22:20:49.778591       1 auto_scaling_groups.go:94] Regenerating ASG information for master-us-east-1d.masters.production-k8s.example.com\
I1009 22:20:49.830307       1 auto_scaling_groups.go:94] Regenerating ASG information for master-us-east-1e.masters.production-k8s-01.example.com\
I1009 22:20:49.873971       1 auto_scaling_groups.go:94] Regenerating ASG information for master-us-east-1e.masters.production-k8s.example.com\
I1009 22:20:49.919504       1 auto_scaling_groups.go:94] Regenerating ASG information for nodes-core.production-k8s.example.com\
I1009 22:20:50.414791       1 leaderelection.go:204] successfully renewed lease kube-system/cluster-autoscaler\
I1009 22:20:50.536753       1 auto_scaling_groups.go:94] Regenerating ASG information for nodes.production-k8s-01.example.com\
I1009 22:20:50.581501       1 auto_scaling_groups.go:94] Regenerating ASG information for production-k8s-02-Controlplane-1EQ9EQZSGTVAN-Controllers-1MJB0LU5KF4TO\
I1009 22:20:50.627064       1 auto_scaling_groups.go:94] Regenerating ASG information for production-k8s-02-Controlplane-1EQ9EQZSGTVAN-Etcd0-1XUNPV1PHJF9G\
I1009 22:20:50.659466       1 auto_scaling_groups.go:94] Regenerating ASG information for production-k8s-02-Controlplane-1EQ9EQZSGTVAN-Etcd1-WE30HM714T51\
I1009 22:20:50.699992       1 auto_scaling_groups.go:94] Regenerating ASG information for production-k8s-02-Controlplane-1EQ9EQZSGTVAN-Etcd2-R2SAF483UQAJ\
I1009 22:20:50.742502       1 auto_scaling_groups.go:94] Regenerating ASG information for production-k8s-02-Controlplane-1EQ9EQZSGTVAN-Etcd3-DVGPKGKIHKH7\
I1009 22:20:50.773041       1 auto_scaling_groups.go:94] Regenerating ASG information for production-k8s-02-Controlplane-1EQ9EQZSGTVAN-Etcd4-LOZU27BQ9UFO\
I1009 22:20:50.806447       1 auto_scaling_groups.go:94] Regenerating ASG information for production-k8s-02-Spotworkers-LJSGSQAREX7R-Workers-ZNY07LBGNIQI\
I1009 22:20:50.857753       1 auto_scaling_groups.go:94] Regenerating ASG information for production-k8s-02-Workers-1650RNVJX2B6F-Workers-NEGMM0Y4SX9G\
I1009 22:20:50.935470       1 auto_scaling_groups.go:94] Regenerating ASG information for spot-nodes-core.production-k8s.example.com\
I1009 22:20:50.974711       1 auto_scaling_groups.go:94] Regenerating ASG information for spot-nodes.production-k8s-01.example.com\
I1009 22:20:52.430606       1 leaderelection.go:204] successfully renewed lease kube-system/cluster-autoscaler\
I1009 22:20:54.444916       1 leaderelection.go:204] successfully renewed lease kube-system/cluster-autoscaler\
I1009 22:20:56.364412       1 polling_autoscaler.go:105] Poll finished\

All ASGs referenced in the above log with the prefix production-k8s-02-Controlplane are the ASG's I expected to be effected and they provision the EC2 instances on which the cluster-autoscaler runs. All other ASG's mentioned are not part of the cluster and should not be touched by autoscaler.

[MaciekPytel]

@mumoshu Can you take a look?

@orkhanM This looks like a problem with node group autodiscovery. There is also another open bug related to it: #252. My suggestion would be to use static node group configuration (specifying each group you want CA to manage explicitly with --nodes flag as described in https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md) until the issues with --node-group-auto-discovery are fixed.

[geezyx]

Hey there, I just reproduced this issue last night and actually found that CA scaled down some master ASGs from a different cluster in the same region! I was using v0.6.2 and also began seeing API rate limit exceeded errors from AWS. Very odd... I was using the --node-group-auto-discovery as well. Let me know if you want me to try reproducing any further, I'm using k8s 1.6.7 and have multiple clusters where I am running CA and can test this out.

[bismarck]

We ran into this issue also. We modified our IAM policy for the CA to protect against this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeLaunchConfigurations",
                "autoscaling:DescribeTags"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:TerminateInstanceInAutoScalingGroup",
                "autoscaling:SetDesiredCapacity"
            ],
            "Resource": "*",
            "Condition": {
                "ForAnyValue:StringEqualsIgnoreCase": {
                    "autoscaling:ResourceTag/KubernetesCluster": [
                        "k8s.your-domain.com"    
                    ]
                }
            }
        }
    ]
}

[mumoshu]

Hi, sorry for the trouble.

@geezy Are you sure you're using CA 0.6.2? The fix for the problem on CA side is #280, but it seems like the only release including it is CA 1.0.0 as of today.

@geezy @orkhanM As @bismarck has kindly suggested, you can prevent the problem that unintended node being killed by modifying the IAM policy like kubernetes-retired/kube-aws#903. Btw, the modification to the IAM policy is included since kube-aws v0.9.9-rc.1.

Also, in case your cluster is provisioned with kube-aws, probably you're misconfiguring cluster.yaml as explained in kubernetes-retired/kube-aws#905. Would you mind revising your cluster.yaml?

FYI, kubernetes-retired/kube-aws#800 would help you understand what was going on in your cluster - if you're interested.

[geezyx]

Thanks @bismarck !

@mumoshu sorry for the delayed response! For now, we removed the --node-group-auto-discovery flag and are defining node groups manually. We decided against the IAM policy changes because we were also running into #252. These are not kube-aws deployed clusters, they are tectonic clusters. If a 0.6.x release comes out that takes care of this and #252 we'll give it another go.

Thank you very much!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close