[ordering] Wait for clusterRole creation before creating clusterRoleBinding

[galamiram]

kapp try to create clusterRoleBinding before clusterRole has been created:

Changes                                                                                                                                                                                                                                                                                                                  
                                                                                                                                                                                                                                                                                                                         
Namespace    Name                Kind                Conds.  Age  Op      Op st.  Wait to    Rs  Ri                                                                                                                                                                                                                      
(cluster)    kube-state-metrics  ClusterRole         -       -    create  -       reconcile  -   -                                                                                                                                                                                                                       
^            kube-state-metrics  ClusterRoleBinding  -       -    create  -       reconcile  -   -                                                                                                                                                                                                                       
kube-system  kube-state-metrics  Deployment          -       -    create  -       reconcile  -   -                                                                                                                                                                                                                       
^            kube-state-metrics  Service             -       -    create  -       reconcile  -   -                                                                                                                                                                                                                       
^            kube-state-metrics  ServiceAccount      -       -    create  -       reconcile  -   -                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                                                                                         
Op:      5 create, 0 delete, 0 update, 0 noop                                                                                                                                                                                                                                                                            
Wait to: 5 reconcile, 0 delete, 0 noop                                                                                                                                                                                                                                                                                   
                                                                                                                                                                                                                                                                                                                         
3:10:51PM: ---- applying 3 changes [0/5 done] ----                                                                                                                                                                                                                                                                       
3:10:51PM: create clusterrolebinding/kube-state-metrics (rbac.authorization.k8s.io/v1) cluster                                                                                                                                                                                                                           
                                                                                                                                                                                                                                                                                                                         
kapp: Error: Applying create clusterrolebinding/kube-state-metrics (rbac.authorization.k8s.io/v1) cluster:                                                                                                                                                                                                               
  Creating resource clusterrolebinding/kube-state-metrics (rbac.authorization.k8s.io/v1) cluster: clusterroles.rbac.authorization.k8s.io "kube-state-metrics" not found (reason: NotFound)   

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this worked on as soon as possible"
👎 "There are more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help fix this issue.

[cppforlife]

@galamiram we are having trouble replicating this error. we changed kube-state-metrics to order ClusterRoleBinding creation before ClusterRole to hit this error message...

Target cluster 'https://192.168.99.101:8443' (nodes: minikube)

Changes

Namespace    Name                Kind                Conds.  Age  Op      Op st.  Wait to    Rs  Ri
(cluster)    kube-state-metrics  ClusterRole         -       -    create  -       reconcile  -   -
^            kube-state-metrics  ClusterRoleBinding  -       -    create  -       reconcile  -   -
kube-system  kube-state-metrics  Deployment          -       -    create  -       reconcile  -   -
^            kube-state-metrics  Service             -       -    create  -       reconcile  -   -
^            kube-state-metrics  ServiceAccount      -       -    create  -       reconcile  -   -

Op:      5 create, 0 delete, 0 update, 0 noop
Wait to: 5 reconcile, 0 delete, 0 noop

Continue? [yN]: y

5:12:17PM: ---- applying 2 changes [0/5 done] ----
5:12:17PM: create clusterrolebinding/kube-state-metrics (rbac.authorization.k8s.io/v1) cluster
5:12:17PM: create serviceaccount/kube-state-metrics (v1) namespace: kube-system
5:12:17PM: ---- waiting on 2 changes [0/5 done] ----
5:12:17PM: ok: reconcile serviceaccount/kube-state-metrics (v1) namespace: kube-system
... continues on successfully ...

as you can see ClusterRoleBinding gets created successfully before ClusterRole is in the cluster.

im actually surprised you are running into a problme here since *Role and *RoleBinding resources should not care about ordering when they get created in the cluster.

anything else you can provide for us to repro this problem? what k8s version are you running? do you have some custom webhooks installed that add some special validations, etc?

[galamiram]

@cppforlife we are running GKE cluster 1.15.12-gke.2 and we don't have any custom webhooks nor special validation IIRC. What additional information I can provide to help with the repro?

[galamiram]

BTW, we're using kustomize and pipe its output to kapp: kustomize build . | kapp deploy --namespace <namespace> --diff-changes --yes --app <app_name> -f -

[cppforlife]

asked about this error in sig-api-machinery in https://kubernetes.slack.com/archives/C0EG7JC6T/p1601043259006300

[cppforlife]

@galamiram could you try following two things:

• try applying cluster role binding and nothing else to see if you get the same error. one would imagine that you should receive same error since cluster role wouldnt be there.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: test-rbac-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: test-rbac-metrics
subjects:
- kind: ServiceAccount
  name: test-rbac-metrics
  namespace: kube-system

• i just happened to have v1.16.11-gke.5 cluster myself where i cannot repro this problem. can you try a higher version of gke and see if it still happens? (use standalone config above just to ensure that there is no cluster role present)

[galamiram]

@cppforlife It was working for me...

@@ create clusterrolebinding/test-rbac-metrics (rbac.authorization.k8s.io/v1) cluster @@
      0 + apiVersion: rbac.authorization.k8s.io/v1
      1 + kind: ClusterRoleBinding
      2 + metadata:
      3 +   labels:
      4 +     kapp.k14s.io/app: "1601201925676890633"
      5 +     kapp.k14s.io/association: v1.3c0fcf39a5bc3e771fc9ac9f196c1c83
      6 +   name: test-rbac-metrics
      7 + roleRef:
      8 +   apiGroup: rbac.authorization.k8s.io
      9 +   kind: ClusterRole
     10 +   name: test-rbac-metrics
     11 + subjects:
     12 + - kind: ServiceAccount
     13 +   name: test-rbac-metrics
     14 +   namespace: kube-system
     15 + 

Changes

Namespace  Name               Kind                Conds.  Age  Op      Op st.  Wait to    Rs  Ri  
(cluster)  test-rbac-metrics  ClusterRoleBinding  -       -    create  -       reconcile  -   -  

Op:      1 create, 0 delete, 0 update, 0 noop
Wait to: 1 reconcile, 0 delete, 0 noop

8:41:14PM: ---- applying 1 changes [0/1 done] ----
8:41:15PM: create clusterrolebinding/test-rbac-metrics (rbac.authorization.k8s.io/v1) cluster
8:41:15PM: ---- waiting on 1 changes [0/1 done] ----
8:41:15PM: ok: reconcile clusterrolebinding/test-rbac-metrics (rbac.authorization.k8s.io/v1) cluster
8:41:15PM: ---- applying complete [1/1 done] ----
8:41:15PM: ---- waiting complete [1/1 done] ----

Succeeded

[cppforlife]

@galamiram is the original problem consistently reproducible?

[galamiram]

@cppforlife Nope AFAIK. I'm not the one who experience this issue first-place and was not able to repro as well. Let's close this issue for now and if anyone will encounter this problem in the future, let's re-open it.

[Velyks]

This is also happening to me with the following resources, applying them with just kubectl works fine.
Trying to apply only the RoleBinding and not the role produces the same error as well.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: catalog
  labels:
    app.kubernetes.io/name: catalog
    app.kubernetes.io/instance: catalog
    app.kubernetes.io/managed-by: eks-deploy
    app.kubernetes.io/part-of: catalog
    app.kubernetes.io/version: "1d8ad1e7d8af1894eb56f293223d1b2b3ae9c6c3"
    helm.sh/chart: lyst-app-0.1.9
    application_group: catalog
    app_name: catalog
    team: productdata
rules:
# pod get/watch is used to identify the container IDs of the current pod
# pod patch is used to annotate the step's outputs back to controller (e.g. artifact location)
- apiGroups:
    - ""
  resources:
    - pods
  verbs:
    - get
    - watch
    - patch
# logs get/watch are used to get the pods logs for script outputs, and for log archival
- apiGroups:
    - ""
  resources:
    - pods/log
  verbs:
    - get
    - watch
---
# Source: lyst-app/templates/role-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: catalog
  labels:
    app.kubernetes.io/name: catalog
    app.kubernetes.io/instance: catalog
    app.kubernetes.io/managed-by: eks-deploy
    app.kubernetes.io/part-of: catalog
    app.kubernetes.io/version: "1d8ad1e7d8af1894eb56f293223d1b2b3ae9c6c3"
    helm.sh/chart: lyst-app-0.1.9
    application_group: catalog
    app_name: catalog
    team: productdata
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: catalog
subjects:
  - kind: ServiceAccount
    name: catalog

$ kapp deploy -a catalog -n catalog -f roles.yaml
Target cluster '\<cluster url\>'

Changes

Namespace  Name     Kind         Conds.  Age  Op      Op st.  Wait to    Rs  Ri  
catalog    catalog  Role         -       -    create  -       reconcile  -   -  
^          catalog  RoleBinding  -       -    create  -       reconcile  -   -  

Op:      2 create, 0 delete, 0 update, 0 noop
Wait to: 2 reconcile, 0 delete, 0 noop

Continue? [yN]: y

5:57:38PM: ---- applying 2 changes [0/2 done] ----
5:57:38PM: create rolebinding/catalog (rbac.authorization.k8s.io/v1) namespace: catalog

kapp: Error: Applying create rolebinding/catalog (rbac.authorization.k8s.io/v1) namespace: catalog:
  Creating resource rolebinding/catalog (rbac.authorization.k8s.io/v1) namespace: catalog: roles.rbac.authorization.k8s.io "catalog" not found (reason: NotFound)

kapp version 0.34.0
kubectl v1.19.3
EKS 1.18.8

[danielhelfand]

Thanks for reporting this @Velyks.

I am unfortunately not able to reproduce this, but I am not using EKS so it might be worth looking into it there. Would you be able to see if you could run this on a local cluster like Minikube or KinD to see if you still run into the problem? It will help narrow down if we need to look specifically into EKS.

As far as just running with creating the RoleBinding, I am also seeing no error.

I will reopen this for now though.

[cirocosta]

@danielhelfand I'm currently facing something really similar running kapp deploy on an eduk8s workshop session in a kind cluster:

Namespace                   Name                     Kind              Conds.  Age  Op      Op st.  Wait to    Rs  Ri  
demo-xxxxx-xxxxxx-w01-s001  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 
...
^                           kapp-permissions         Role              -       -    create  -       reconcile  -   -  
^                           kapp-permissions         RoleBinding       -       -    create  -       reconcile  -   -  
...

Op:      10 create, 0 delete, 0 update, 0 noop
Wait to: 10 reconcile, 0 delete, 0 noop

Continue? [yN]: y

2:52:20PM: ---- applying 3 changes [0/10 done] ----
2:52:20PM: create rolebinding/kapp-permissions (rbac.authorization.k8s.io/v1) namespace: demo-xxxxx-xxxxxx-w01-s001

kapp: Error: Applying create rolebinding/kapp-permissions (rbac.authorization.k8s.io/v1) namespace: demo-xxxxx-xxxxxx-w01-s001:
  Creating resource rolebinding/kapp-permissions (rbac.authorization.k8s.io/v1) namespace: demo-xxxxx-xxxxxx-w01-s001: roles.rbac.authorization.k8s.io "kapp-permissions" not found (reason: NotFound)

where

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kapp-permissions
rules:
  - apiGroups:
      - service.binding
      - serving.knative.dev
    resources: ['*']
    verbs: ['*']
  - apiGroups: [""]
    resources: ['configmaps', 'secrets']
    verbs: ['*']

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kapp-permissions
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kapp-permissions
subjects:
  - kind: ServiceAccount
    name: service-account

After the failure, I can see that kapp was able to create the Role though (which makes me think that yes, it's an ordering issue).

(naturallly, kubectl apply -f ... works with no problems)

also:

$ kapp --version
kapp version 0.36.0

and

$ k version
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-03-11T06:23:38Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

ps.: having the Role created before and then kapp deploying just the RoleBinding works.
pps.: running kapp deploy twice ends up working

debug logs

03:23:09PM: debug: LabeledResources: findNonLabeledResources: end (7.527585ms)
03:23:09PM: debug: LabeledResources: AllAndMatching: end (1.136277528s)
03:23:09PM: debug: ClusterChangeSet: ChangeGraph: NewChangeGraph: start
03:23:09PM: debug: ClusterChangeSet: ChangeGraph: buildEdges: start
03:23:09PM: debug: ClusterChangeSet: ChangeGraph: buildEdges: end (90.877µs)
03:23:09PM: debug: ClusterChangeSet: ChangeGraph: checkCycles: start
03:23:09PM: debug: ClusterChangeSet: ChangeGraph: checkCycles: end (4.695µs)
03:23:09PM: debug: ClusterChangeSet: ChangeGraph: buildEdges: start
03:23:09PM: debug: ClusterChangeSet: ChangeGraph: buildEdges: end (3.863µs)
03:23:09PM: debug: ClusterChangeSet: ChangeGraph: checkCycles: start
03:23:09PM: debug: ClusterChangeSet: ChangeGraph: checkCycles: end (2.795µs)
03:23:09PM: debug: ClusterChangeSet: ChangeGraph: NewChangeGraph: end (123.373µs)
Changes
Namespace                   Name              Kind            Conds.  Age  Op      Op st.  Wait to    Rs  Ri

demo-steel-thread-w01-s002  kapp-permissions  Role            -       -    create  -       reconcile  -   -

^                           kapp-permissions  RoleBinding     -       -    create  -       reconcile  -   -

^                           service-account   ServiceAccount  -       -    create  -       reconcile  -   -
Op:      3 create, 0 delete, 0 update, 0 noop

Wait to: 3 reconcile, 0 delete, 0 noop
Continue? [yN]: y
3:23:44PM: ---- applying 3 changes [0/3 done] ----

03:23:44PM: debug: IdentifiedResources: Create(rolebinding/kapp-permissions (rbac.authorization.k8s.io/v1) namespace: demo-steel-thread-w01-s002): start

03:23:44PM: debug: IdentifiedResources: Create(serviceaccount/service-account (v1) namespace: demo-steel-thread-w01-s002): start

03:23:44PM: debug: IdentifiedResources: Create(role/kapp-permissions (rbac.authorization.k8s.io/v1) namespace: demo-steel-thread-w01-s002): start

03:23:44PM: debug: IdentifiedResources: Create(rolebinding/kapp-permissions (rbac.authorization.k8s.io/v1) namespace: demo-steel-thread-w01-s002): end (6.169487ms)

03:23:44PM: debug: IdentifiedResources: Create(serviceaccount/service-account (v1) namespace: demo-steel-thread-w01-s002): end (6.164437ms)

3:23:44PM: create rolebinding/kapp-permissions (rbac.authorization.k8s.io/v1) namespace: demo-steel-thread-w01-s002

03:23:44PM: debug: IdentifiedResources: Update(serviceaccount/service-account (v1) namespace: demo-steel-thread-w01-s002): start

03:23:44PM: debug: IdentifiedResources: Create(role/kapp-permissions (rbac.authorization.k8s.io/v1) namespace: demo-steel-thread-w01-s002): end (26.772177ms)

03:23:44PM: debug: IdentifiedResources: Update(serviceaccount/service-account (v1) namespace: demo-steel-thread-w01-s002): end (15.988096ms)

03:23:44PM: debug: IdentifiedResources: Update(role/kapp-permissions (rbac.authorization.k8s.io/v1) namespace: demo-steel-thread-w01-s002): start

03:23:44PM: debug: CommandRun: end (35.836923091s)