PSP Rollout is broken #5505

dschunack · 2023-02-13T15:55:39Z

Hi,

the last Chart update breaks the rollout of the PSP. API Version v1 is not available on AWS EKS 1.22, 1.23 and 1.24.

Error: resource mapping not found for name: "cluster-autoscaler-aws-cluster-autoscaler" namespace: "" from "": no matches for kind "PodSecurityPolicy" in version "policy/v1"

autoscaler/charts/cluster-autoscaler/templates/_helpers.tpl

Lines 73 to 74 in a484713

    
           {{- else if semverCompare ">1.21-0" $kubeTargetVersion -}} 
        
           {{- print "policy/v1" -}}

EKS 1.22

kubectl api-resources --api-group='policy'   
NAME                   SHORTNAMES   APIVERSION       NAMESPACED   KIND
poddisruptionbudgets   pdb          policy/v1        true         PodDisruptionBudget
podsecuritypolicies    psp          policy/v1beta1   false        PodSecurityPolicy

kubectl version 
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:25:17Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.16-eks-ffeb93d", GitCommit:"52e500d139bdef42fbc4540c357f0565c7867a81", GitTreeState:"clean", BuildDate:"2022-11-29T18:41:42Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

EKS 1.24

kubectl api-resources --api-group='policy'   
NAME                   SHORTNAMES   APIVERSION       NAMESPACED   KIND
poddisruptionbudgets   pdb          policy/v1        true         PodDisruptionBudget
podsecuritypolicies    psp          policy/v1beta1   false        PodSecurityPolicy

kubectl version                       
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:25:17Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.8-eks-ffeb93d", GitCommit:"abb98ec0631dfe573ec5eae40dc48fd8f2017424", GitTreeState:"clean", BuildDate:"2022-11-29T18:45:03Z", GoVersion:"go1.18.8", Compiler:"gc", Platform:"linux/amd64"}

v1 doesn't exist in the API Documentation for PSP [API ref 1.24] (https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#podsecuritypolicy-v1beta1-policy) and PSP are removed in 1.25. It make sense to revert #5357 and #5500 .

autoscaler/charts/cluster-autoscaler/templates/_helpers.tpl

Lines 66 to 76 in b57d917

    
           {{/* 
        
           Return the appropriate apiVersion for podsecuritypolicy. 
        
           */}} 
        
           {{- define "podsecuritypolicy.apiVersion" -}} 
        
           {{- $kubeTargetVersion := default .Capabilities.KubeVersion.GitVersion .Values.kubeTargetVersionOverride }} 
        
           {{- if semverCompare "<1.10-0" $kubeTargetVersion -}} 
        
           {{- print "extensions/v1beta1" -}} 
        
           {{- else -}} 
        
           {{- print "policy/v1beta1" -}} 
        
           {{- end -}} 
        
           {{- end -}}

I will create a PR to fix this in the next minutes.

The text was updated successfully, but these errors were encountered:

gjtempleton · 2023-02-13T16:46:50Z

#5480 should resolve this once merged.

gjtempleton · 2023-02-13T21:44:00Z

#5480 has now been merged, if you can confirm that's fixed this, we can close this off. Thanks for raising it.

llamahunter · 2023-04-27T03:04:51Z

tried to update to 1.25 today, and got this:

Error: resource mapping not found for name: "aws-cluster-autoscaler" namespace: "" from "": no matches for kind "PodSecurityPolicy" in version "policy/v1beta1"
ensure CRDs are installed first

Using helm chart 9.28.0 and image tag v1.25.1

josecsotomorales · 2023-05-11T16:42:22Z

Upgraded to k8s 1.25 ... getting this issue with helm chart 9.28.0 as well

josecsotomorales · 2023-05-11T20:42:01Z

This is the error I'm getting on my side: no matches for kind "PodDisruptionBudget" in version "policy/v1beta1"

josecsotomorales · 2023-05-11T22:07:29Z

From my research it's a Helm issue actually, tested helm template and it produces the expected policy, more details here: helm/helm#7219

4sudiptodas · 2023-06-13T07:07:00Z

Upgraded to aws k8s 1.27 and getting this issue with helm chart 9.28.0, unable to build kubernetes objects from current release manifest: resource mapping not found for name: "aws-cluster-autoscaler" namespace: "" from "": no matches for kind "PodDisruptionBudget" in version "policy/v1beta1" ensure CRDs are installed first

JCBSLMN · 2023-06-21T21:04:11Z

i'm having the same issue as @4sudiptodas

xiaodong-xie · 2023-07-04T13:27:41Z

Got inspirations from this comment: helm/helm#7219 (comment)

This works for me:

helm plugin install https://github.com/helm/helm-mapkubeapis
helm mapkubeapis -n kube-system cluster-autoscaler

Then upgrading cluster-autoscaler using helm upgrade --install succeeded.

jd-sandk · 2023-07-07T19:04:19Z

Same issue with Helm Chart: 9.29.1 when using Terraform to apply - we're were getting Error:

╷
│ Error: unable to build kubernetes objects from current release manifest: resource mapping not found for name: "cluster-autoscaler-aws-cluster-autoscaler" namespace: "" from "": no matches for kind "PodDisruptionBudget" in version "policy/v1beta1"
│ ensure CRDs are installed first
│ 
│   on modules/cluster/helm.tf line 45, in resource "helm_release" "cluster-autoscaler":
│   45: resource "helm_release" "cluster-autoscaler" {
│ 
╵

Exited with code exit status 1
CircleCI received exit code 1

We've even tried to explicitly set the following in the Helm Chart:

  set {
    name  = "kubeTargetVersionOverride"
    value = "1.27.0"
    type  = "string"
  }

As we are on v1.27 and due to seeing:

{{- define "podDisruptionBudget.apiVersion" -}}
{{- $kubeTargetVersion := default .Capabilities.KubeVersion.GitVersion .Values.kubeTargetVersionOverride }}
{{- if semverCompare "<1.21-0" $kubeTargetVersion -}}
{{- print "policy/v1beta1" -}}
{{- else -}}
{{- print "policy/v1" -}}
{{- end -}}
{{- end -}}

https://github.com/kubernetes/autoscaler/blob/master/charts/cluster-autoscaler/templates/_helpers.tpl#L78-L88

Also later based upon our looking of the Template code (https://github.com/kubernetes/autoscaler/blob/master/charts/cluster-autoscaler/templates/podsecuritypolicy.yaml#L1C3-L1C3) a member of our team thought the following might help trying to set the following on the Helm Chart:

  set {
    name  = "rbac.create"
    value = "true"
    type  = "string"
  }

  set {
    name  = "rbac.pspEnabled"
    value = "true"
    type  = "string"
  }

but then that also only leads to another similiar issue:

╷
│ Error: resource mapping not found for name: "cluster-autoscaler-aws-cluster-autoscaler" namespace: "" from "": no matches for kind "PodSecurityPolicy" in version "policy/v1beta1"
│ ensure CRDs are installed first
│ 
│   on modules/cluster/helm.tf line 45, in resource "helm_release" "cluster-autoscaler":
│   45: resource "helm_release" "cluster-autoscaler" {
│ 
╵

Exited with code exit status 1
CircleCI received exit code 1

Why is the Helm chart trying to force us to usev1beta1, when we're on EKS Cluster/Node Group (Kubernetes) Version: 1.27, when it should be trying to use v1 instead.

We have those Kinds available in our Cluster:

# kubectl api-resources --api-group='policy'   
NAME                   SHORTNAMES   APIVERSION   NAMESPACED   KIND
poddisruptionbudgets   pdb          policy/v1    true         PodDisruptionBudget

😞

Idan-Lazar · 2023-08-29T12:16:45Z

@jd-sandk Do you find any solutions?

jd-sandk · 2023-08-29T14:52:40Z

@Idan-Lazar We don't remember fully, since it was awhile ago, but we tried to recall this morning, and we want to say: that once we upgraded the Kubernetes version, then semverCompare started to play nice, and everything just worked from that point on (I think after removing and readding the HELM Chart).

Idan-Lazar · 2023-08-29T18:17:59Z

@jd-sandk
What is your Helm Chart version now?
And cluster-autoscaler tag?

vtrenton · 2023-12-06T22:22:41Z

Considering <1.21 has been out of support for years now. I dont really understand the reasoning of tempting the api-version for the pdb. How likely is it that it will change soon/ever from policy/v1? I'm looking at the chart and see apiVersion: {{ template "podDisruptionBudget.apiVersion" . }} per

autoscaler/charts/cluster-autoscaler/templates/pdb.yaml

Line 2 in 8f75e9c

apiVersion: {{ template "podDisruptionBudget.apiVersion" . }}

yet the values.yaml doesn't specify the version:
https://github.com/kubernetes/autoscaler/blob/master/charts/cluster-autoscaler/values.yaml#L258-L262

I'm of the opinion we just hardcode policy/v1 and simply state that this chart requires 1.21+. I dont think that's a huge ask.

marinator86 · 2024-01-16T08:36:06Z

Hi,
does anyone have a workaround for this? Would be great to share.
Thanks

gxpd-jjh · 2024-02-02T00:53:42Z

FWIW, here is what i had to override as a parameter (using the Chart) to get autoscaler deployed on 1.25.

        - name: podDisruptionBudget
          value: 'false'
        - name: rbac.pspEnabled
          value: 'false'

The first one required because

autoscaler/charts/cluster-autoscaler/templates/pdb.yaml

Line 1 in a2f4cac

returns True always due to chart's Values.yaml.

The second one is required because

autoscaler/charts/cluster-autoscaler/templates/podsecuritypolicy.yaml

Line 1 in a2f4cac

and then there is still an issue from #5364 where they just gave up, so it somehow returns "policy/v1beta1" still which is gone in 1.25.

cc @marinator86

sappusaketh · 2024-04-03T23:10:21Z

is this function even working I m on 9.34.1 and even after setting kubeTargetVersionOverride: 1.28.6 still its using "policy/v1beta1" tried kubeTargetVersionOverride: v1.28.6 same error
no matches for kind "PodDisruptionBudget" in version "policy/v1beta1"

I m using TF, deleting chart and recreating helped in my case

shaksing · 2024-07-12T13:10:14Z

Any udpate on this issue ??

dschunack changed the title ~~PSP Rollout on AWS EKS is broken~~ PSP Rollout is broken Feb 13, 2023

dschunack added a commit to dschunack/autoscaler that referenced this issue Feb 13, 2023

Fix issue kubernetes#5505

1cfa977

dschunack mentioned this issue Feb 13, 2023

Fix issue #5505 #5506

Closed

jbartosik added the area/cluster-autoscaler label Feb 13, 2023

gjtempleton added the help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. label Jun 13, 2023

towca added the area/helm-charts label Mar 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PSP Rollout is broken #5505

PSP Rollout is broken #5505

dschunack commented Feb 13, 2023 •

edited

Loading

gjtempleton commented Feb 13, 2023

gjtempleton commented Feb 13, 2023

llamahunter commented Apr 27, 2023

josecsotomorales commented May 11, 2023

josecsotomorales commented May 11, 2023

josecsotomorales commented May 11, 2023

4sudiptodas commented Jun 13, 2023 •

edited

Loading

JCBSLMN commented Jun 21, 2023

xiaodong-xie commented Jul 4, 2023

jd-sandk commented Jul 7, 2023 •

edited

Loading

Idan-Lazar commented Aug 29, 2023

jd-sandk commented Aug 29, 2023 •

edited

Loading

Idan-Lazar commented Aug 29, 2023

vtrenton commented Dec 6, 2023 •

edited

Loading

marinator86 commented Jan 16, 2024

gxpd-jjh commented Feb 2, 2024 •

edited

Loading

sappusaketh commented Apr 3, 2024 •

edited

Loading

shaksing commented Jul 12, 2024

PSP Rollout is broken #5505

PSP Rollout is broken #5505

Comments

dschunack commented Feb 13, 2023 • edited Loading

EKS 1.22

EKS 1.24

gjtempleton commented Feb 13, 2023

gjtempleton commented Feb 13, 2023

llamahunter commented Apr 27, 2023

josecsotomorales commented May 11, 2023

josecsotomorales commented May 11, 2023

josecsotomorales commented May 11, 2023

4sudiptodas commented Jun 13, 2023 • edited Loading

JCBSLMN commented Jun 21, 2023

xiaodong-xie commented Jul 4, 2023

jd-sandk commented Jul 7, 2023 • edited Loading

Idan-Lazar commented Aug 29, 2023

jd-sandk commented Aug 29, 2023 • edited Loading

Idan-Lazar commented Aug 29, 2023

vtrenton commented Dec 6, 2023 • edited Loading

marinator86 commented Jan 16, 2024

gxpd-jjh commented Feb 2, 2024 • edited Loading

sappusaketh commented Apr 3, 2024 • edited Loading

shaksing commented Jul 12, 2024

dschunack commented Feb 13, 2023 •

edited

Loading

4sudiptodas commented Jun 13, 2023 •

edited

Loading

jd-sandk commented Jul 7, 2023 •

edited

Loading

jd-sandk commented Aug 29, 2023 •

edited

Loading

vtrenton commented Dec 6, 2023 •

edited

Loading

gxpd-jjh commented Feb 2, 2024 •

edited

Loading

sappusaketh commented Apr 3, 2024 •

edited

Loading