Suggesting more restrictive k8s RBAC and PSP

[bygui86]

Describe the bug:
I think that RBAC and PSP applied to logging-operator, fluent-bit and fluentd are too permissive.

Proposal:
I propose following RBAC and PSP to restrict as much as possible permissions to components, without jeopardizing or influencing normal functionalities:

operator RBAC + PSP:

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: logging-restricted
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: runtime/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName:  runtime/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName:  runtime/default
spec:
  privileged: false
  allowPrivilegeEscalation: false
  requiredDropCapabilities:
    - ALL
  volumes:
    - configMap
    - secret
    - persistentVolumeClaim
    - emptyDir
    - projected
    - downwardAPI
  hostPID: false
  hostIPC: false
  hostNetwork: false
  runAsUser:
    rule: MustRunAsNonRoot
  runAsGroup:
    rule: MustRunAs
    ranges:
      - min: 100
        max: 65535
  supplementalGroups:
    rule: MustRunAs
    ranges:
      - min: 1
        max: 65535
  fsGroup:
    rule: MustRunAs
    ranges:
      - min: 1
        max: 65535
  seLinux:
    rule: RunAsAny
  readOnlyRootFilesystem: true
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: logging-operator
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: logging-operator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: logging-operator
subjects:
  - kind: ServiceAccount
    name: logging-operator
    namespace: logging
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: logging-operator
rules:
  - apiGroups:
      - policy
    resources:
      - podsecuritypolicies
    verbs:
      - use
    resourceNames:
      - logging-restricted
  - apiGroups:
      - batch
    resources:
      - jobs
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: logging-operator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: logging-operator
subjects:
  - kind: ServiceAccount
    name: logging-operator
    namespace: logging
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: logging-operator
rules:
  - apiGroups:
      - policy
    resources:
      - podsecuritypolicies
    verbs:
      - use
    resourceNames:
      - logging-restricted
  - apiGroups:
      - monitoring.coreos.com
    resources:
      - servicemonitors
      - podmonitors
      - prometheusrules
    verbs:
      - "*"
  - apiGroups:
      - apiextensions.k8s.io
    resources:
      - customresourcedefinitions
    verbs:
      - list
      - watch
  - apiGroups:
      - logging.banzaicloud.io
    resources:
      - "*"
    verbs:
      - "*"
  - apiGroups:
      - ""
    resources:
      - nodes
      - namespaces
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - pods
      - services
      - endpoints
      - configmaps
      - secrets
      - persistentvolumeclaims
    verbs:
      - "*"
  - apiGroups:
      - apps
    resources:
      - deployments
      - daemonsets
      - statefulsets
    verbs:
      - "*"
  - apiGroups:
      - ""
      - rbac.authorization.k8s.io
    resources:
      - serviceaccounts
      - roles
      - rolebindings
      - clusterroles
      - clusterrolebindings
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - policy
    resources:
      - podsecuritypolicies
    verbs:
      - get
      - list
      - watch

fluent-bit RBAC + PSP:

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: logging-fluentbit-restricted
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: runtime/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName:  runtime/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName:  runtime/default
spec:
  privileged: false
  allowPrivilegeEscalation: false
  requiredDropCapabilities:
    - ALL
  hostPID: false
  hostIPC: false
  hostNetwork: false
  runAsGroup:
    rule: MustRunAs
    ranges:
      - min: 100
        max: 65535
  supplementalGroups:
    rule: MustRunAs
    ranges:
      - min: 1
        max: 65535
  fsGroup:
    rule: MustRunAs
    ranges:
      - min: 1
        max: 65535
  seLinux:
    rule: RunAsAny
  readOnlyRootFilesystem: true
  # DIFFERENCES from logging-restricted
  volumes:
    - configMap
    - secret
    - emptyDir
    - hostPath
  allowedHostPaths:
    # why this folder even considering it's empty? anyway read only...
    - pathPrefix: /var/lib/docker/containers
      readOnly: true
    - pathPrefix: /var/log
      readOnly: false
  runAsUser:
    rule: RunAsAny
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: logging-fluentbit
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: logging-fluentbit
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: logging-fluentbit
subjects:
- kind: ServiceAccount
  name: logging-fluentbit
  namespace: logging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: logging-fluentbit
rules:
  - apiGroups:
      - policy
    resources:
      - podsecuritypolicies
    verbs:
      - use
    resourceNames:
      - logging-fluentbit-restricted
  - apiGroups:
    - ""
    resources:
    - pods
    - namespaces
    verbs:
    - get
    - list
    - watch

fluentd RBAC + PSP:

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: logging-fluentd-restricted
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: runtime/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName:  runtime/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName:  runtime/default
spec:
  privileged: false
  allowPrivilegeEscalation: false
  requiredDropCapabilities:
    - ALL
  hostPID: false
  hostIPC: false
  hostNetwork: false
  runAsUser:
    rule: MustRunAsNonRoot
  runAsGroup:
    rule: MustRunAs
    ranges:
      - min: 100
        max: 65535
  supplementalGroups:
    rule: MustRunAs
    ranges:
      - min: 1
        max: 65535
  fsGroup:
    rule: MustRunAs
    ranges:
      - min: 1
        max: 65535
  seLinux:
    rule: RunAsAny
  # DIFFERENCES from logging-restricted
  volumes:
    - configMap
    - secret
    - persistentVolumeClaim
    - emptyDir
    - hostPath
  # readOnlyRootFilesystem: true
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: logging-fluentd
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: logging-fluentd
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: logging-fluentd
subjects:
  - kind: ServiceAccount
    name: logging-fluentd
    namespace: logging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: logging-fluentd
rules:
  - apiGroups:
      - policy
    resources:
      - podsecuritypolicies
    verbs:
      - use
    resourceNames:
      - logging-fluentd-restricted
  - apiGroups:
      - ""
    resources:
      - configmaps
      - secrets
    verbs:
      - "*"

logging usage example:

apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  name: logging-components
spec:
  enableRecreateWorkloadOnImmutableFieldChange: true
  controlNamespace: logging
  fluentbit:
    metrics:
      serviceMonitor: true
    tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - operator: Exists
        effect: NoExecute
      - operator: Exists
        effect: NoSchedule
    security:
      roleBasedAccessControlCreate: false
      podSecurityPolicyCreate: false
      serviceAccount: logging-fluentbit
      podSecurityContext:
        fsGroup: 101
  fluentd:
    scaling:
      replicas: 3
    metrics:
      serviceMonitor: true
    livenessDefaultCheck: true
    security:
      roleBasedAccessControlCreate: false
      podSecurityPolicyCreate: false
      serviceAccount: logging-fluentd
      securityContext:
        readOnlyRootFilesystem: false
      podSecurityContext:
        runAsUser: 100
        runAsGroup: 100
        fsGroup: 101

Environment details:
Above proposal were tested on an environment with following details:

• Kubernetes version: v1.17.14
• Cloud-provider/provisioner: GKE
• logging-operator version: 3.8.3
• Install method: static manifests

Looking forward your feedback!!

/kind bug