fix(uninstall):remove kubearmor annotations from kubernetes resources

[Prateeknandle]

fixes: #434

1. If annotations are added at the owner level then we remove annotations from deployments, replicasets, statefulsets, daemonsets, job, cronjob.
2. If annotations are added at pod level then all the pods with annotations will be restarted(deleted).
3. Lists all the pods with karmor uninstall that will be restarted when --force flag will be used

• karmor uninstall output:

prateek@bot:~ $ ~/go/bin/karmor uninstall 
ℹ️   Resources not managed by helm/Global Resources are not cleaned up. Please use karmor uninstall --force if you want complete cleanup.
ℹ️   Following pods will get restarted with karmor uninstall --force: 
+-----+--------------------------+-----------+
| NO  |         POD NAME         | NAMESPACE |
+-----+--------------------------+-----------+
|   1 | example-replicaset-6k4z4 | default   |
|   2 | example-replicaset-6m7nq | default   |
|   3 | example-replicaset-727ds | default   |
|   4 | example-replicaset-7bskf | default   |
|   5 | example-replicaset-7ckcn | default   |
|   6 | example-replicaset-9npvb | default   |
|   7 | example-replicaset-bshxd | default   |
|   8 | example-replicaset-cjxm9 | default   |
|   9 | example-replicaset-fmc58 | default   |
|  10 | example-replicaset-ltwl7 | default   |
|  11 | example-replicaset-mpxvc | default   |
|  12 | example-replicaset-n4xsw | default   |
|  13 | example-replicaset-pn4mb | default   |
|  14 | example-replicaset-s774m | default   |
|  15 | example-replicaset-sqzm4 | default   |
|  16 | example-statefulset-0    | default   |
|  17 | example-statefulset-1    | default   |
|  18 | example-statefulset-10   | default   |
|  19 | example-statefulset-11   | default   |
|  20 | example-statefulset-12   | default   |
|  21 | example-statefulset-13   | default   |
|  22 | example-statefulset-14   | default   |
|  23 | example-statefulset-15   | default   |
|  24 | example-statefulset-16   | default   |
|  25 | example-statefulset-17   | default   |
|  26 | example-statefulset-18   | default   |
|  27 | example-statefulset-19   | default   |
|  28 | example-statefulset-2    | default   |
|  29 | example-statefulset-3    | default   |
|  30 | example-statefulset-4    | default   |
|  31 | example-statefulset-5    | default   |
|  32 | example-statefulset-6    | default   |
|  33 | example-statefulset-7    | default   |
|  34 | example-statefulset-8    | default   |
|  35 | example-statefulset-9    | default   |
+-----+--------------------------+-----------+
❌  KubeArmor resources removed
🔄  Checking if KubeArmor pods are stopped...
🔴  Done Checking; all services are stopped!             
⌚️  Termination Time: 5.294239144s 

[DelusionalOptimist]

@Prateeknandle let's add a warning as well that policies and annotations will be removed when running karmor uninstall
cc @daemon1024

[DelusionalOptimist]

LGTM

[DelusionalOptimist]

KubeArmor v1.4.0 stable, BPF LSM node - pods belonging to deployments are also getting restarted with --force and being presented with this warning.

IMO pods which don't have the apparmor annotation should not be restarted?

$ ./karmor uninstall
ℹ️   Resources not managed by helm/Global Resources are not cleaned up. Please use karmor uninstall --force if you want complete cleanup.
ℹ️   Following pods will get restarted with karmor uninstall --force:

+-----+-----------------------------------------+-------------+
| NO  |                POD NAME                 |  NAMESPACE  |
+-----+-----------------------------------------+-------------+
|   1 | nginx-bf5d5cf98-99lcw                   | default     |
|   2 | coredns-576bfc4dc7-55xmp                | kube-system |
|   3 | local-path-provisioner-6795b5f9d8-7vt46 | kube-system |
|   4 | metrics-server-557ff575fb-r67vv         | kube-system |
+-----+-----------------------------------------+-------------+
❌  KubeArmor resources removed
🔄  Checking if KubeArmor pods are stopped...
🔴  Done Checking; all services are stopped!
⌚️  Termination Time: 4.329732048s

kubectl describe pod after uninstall

Name:             nginx-bf5d5cf98-99lcw
Namespace:        default
Priority:         0
Service Account:  default
Node:             kubearmor-dev-next/10.0.2.15
Start Time:       Tue, 06 Aug 2024 11:44:33 +0000
Labels:           app=nginx
                  pod-template-hash=bf5d5cf98
Annotations:      kubearmor-policy: enabled
                  kubearmor-visibility: process,file,network,capabilities
Status:           Running
IP:               10.42.0.57
IPs:
  IP:           10.42.0.57
Controlled By:  ReplicaSet/nginx-bf5d5cf98
Containers:
  nginx:
    Container ID:   docker://4ac64f9c6e035a814d1ff752745c36a479f7cedf4a187df81018e56bbb7ad439
    Image:          nginx
    Image ID:       docker-pullable://nginx@sha256:6af79ae5de407283dcea8b00d5c37ace95441fd58a8b1d2aa1ed93f5511bb18c
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Tue, 06 Aug 2024 11:44:38 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2ph48 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  kube-api-access-2ph48:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

[DelusionalOptimist]

Even with --force some resource are left, just like the current behavior.
cc @daemon1024

$ kubectl api-resources | grep kubearmor
kubearmorconfigs                                 operator.kubearmor.com/v1         true         KubeArmorConfig
kubearmorclusterpolicies            csp          security.kubearmor.com/v1         false        KubeArmorClusterPolicy

[daemon1024]

Run legacy uninstall after uninstall regardless of installation type

[rootxrishabh]

CRDs are cleaning successfully -

rootxrishabh@fedora:~/kubearmor-client$ ./karmor uninstall --force
❌  Removing CR kubearmorconfig-default
❌  Removing CRD kubearmorconfigs.operator.kubearmor.com
❌  Removing CRD kubearmorpolicies.security.kubearmor.com
❌  Removing CRD kubearmorclusterpolicies.security.kubearmor.com
❌  Removing CRD kubearmorhostpolicies.security.kubearmor.com
Force removing the annotations. Deployments might be restarted.
❌  KubeArmor resources removed
🔄  Checking if KubeArmor pods are stopped...
🔴  Done Checking; all services are stopped!             
⌚️  Termination Time: 31.036573999s 


rootxrishabh@fedora:~/kubearmor-client$ k get all -n kubearmor
No resources found in kubearmor namespace.

Do we plan to keep clusterrole and clusterrolebindings even after force uninstall?

rootxrishabh@fedora:~/kubearmor-client$ k get clusterrole | grep kubearmor
kubearmor-clusterrole                                                  2024-11-05T08:16:25Z
kubearmor-controller-clusterrole                                       2024-11-15T08:09:59Z
kubearmor-controller-proxy-role                                        2024-11-15T08:09:59Z
kubearmor-relay-clusterrole                                            2024-11-15T08:09:59Z
kubearmor-snitch                                                       2024-11-15T08:09:59Z
rootxrishabh@fedora:~/kubearmor-client$ k get clusterrolebinding | grep kubearmor
kubearmor-clusterrolebinding                                    ClusterRole/kubearmor-clusterrole                                           10d
kubearmor-controller-clusterrolebinding                         ClusterRole/kubearmor-controller-clusterrole                                8m20s
kubearmor-controller-proxy-rolebinding                          ClusterRole/kubearmor-controller-proxy-role                                 8m19s
kubearmor-relay-clusterrolebinding                              ClusterRole/kubearmor-relay-clusterrole                                     8m20s
kubearmor-snitch-binding                                        ClusterRole/kubearmor-snitch                                                8m19s

@Prateeknandle @DelusionalOptimist