RBAC warnings when using custom service account/rbac

[andrewdinunzio]

Component(s)

No response

What happened?

Description

Might be related to #2685. Not totally convinced that it's not just something I'm doing wrong though.

Steps to Reproduce

$ minikube create
$ helm install my-opentelemetry-operator open-telemetry/opentelemetry-operator --set "manager.collectorImage.repository=otel/opentelemetry-collector-k8s" --set admissionWebhooks.certManager.enabled=false --set admissionWebhooks.autoGenerateCert.enabled=true
$ kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-custom-sa
---
# collector rbac
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-role
rules:
- apiGroups: [""]
  resources: ["pods", "namespaces", "nodes", "nodes/metrics", "services", "endpoints", "pods"]
  verbs: ["get", "watch", "list"]
- apiGroups: ["apps"]
  resources: ["replicasets"]
  verbs: ["get", "watch", "list"]
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get"]
- apiGroups: ["discovery.k8s.io"]
  resources: ["endpointslices"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["networking.k8s.io"]
  resources: ["ingresses"]
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
- apiGroups: ["monitoring.coreos.com"]
  resources: ["servicemonitors", "podmonitors"]
  verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: otel-metrics-rolebinding
subjects:
- kind: ServiceAccount
  name: metrics-custom-sa
  namespace: default
roleRef:
  kind: ClusterRole
  name: otel-role
  apiGroup: rbac.authorization.k8s.io
---
# target allocator rbac
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: target-allocator-cluster-role
rules:
- apiGroups: [""]
  resources: ["nodes", "nodes/metrics", "services", "endpoints", "pods", "namespaces"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get"]
- apiGroups: ["discovery.k8s.io"]
  resources: ["endpointslices"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["networking.k8s.io"]
  resources: ["ingresses"]
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
- apiGroups: ["monitoring.coreos.com"]
  resources: ["servicemonitors", "podmonitors"]
  verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: target-allocator-clusterrolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: target-allocator-cluster-role
subjects:
- kind: ServiceAccount
  name: metrics-targetallocator        # <-- bind to service account created by operator
  namespace: default
---
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: metrics
spec:
  serviceAccount: metrics-custom-sa    # <-- Use a custom service account
  args:
    feature-gates: -component.UseLocalHostAsDefaultHost
  config:
    exporters:
      nop: {}
    receivers:
      prometheus:
        config:
          global:
            scrape_interval: 10s
          scrape_configs:
          - job_name: otel-collector
            scrape_interval: 10s
            static_configs:
            - targets:
              - 0.0.0.0:8888
    service:
      pipelines:
        metrics:
          exporters:
          - nop
          receivers:
          - prometheus
  mode: statefulset
  observability:
    metrics:
      enableMetrics: true
  targetAllocator:
    allocationStrategy: consistent-hashing
    enabled: true
    filterStrategy: relabel-config
    observability:
      metrics: {}
    prometheusCR:
      enabled: true
      podMonitorSelector: {}
      scrapeInterval: 30s
      serviceMonitorSelector: {}
  upgradeStrategy: automatic
EOF

Expected Result

No warnings/errors.

Actual Result

serviceaccount/metrics-custom-sa created
clusterrole.rbac.authorization.k8s.io/otel-role created
clusterrolebinding.rbac.authorization.k8s.io/otel-metrics-rolebinding created
clusterrole.rbac.authorization.k8s.io/target-allocator-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/target-allocator-clusterrolebinding created
Warning: missing the following rules for monitoring.coreos.com/servicemonitors: [*]
Warning: missing the following rules for services: [get,list,watch]
Warning: missing the following rules for configmaps: [get]
Warning: missing the following rules for discovery.k8s.io/endpointslices: [get,list,watch]
Warning: missing the following rules for networking.k8s.io/ingresses: [get,list,watch]
Warning: missing the following rules for monitoring.coreos.com/podmonitors: [*]
Warning: missing the following rules for nodes: [get,list,watch]
Warning: missing the following rules for nodes/metrics: [get,list,watch]
Warning: missing the following rules for endpoints: [get,list,watch]
Warning: missing the following rules for pods: [get,list,watch]
Warning: missing the following rules for namespaces: [get,list,watch]
Warning: missing the following rules for nonResourceURL: /metrics: [get]
opentelemetrycollector.opentelemetry.io/metrics created

Also any other edits to the metrics OpenTelemetryCollector results in the same warnings.

Kubernetes Version

1.30.0

Operator version

v0.108.0

Collector version

v0.108.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04") on WSL, running in minikube (using docker driver).

Log output

No response

Additional context

$ kubectl get pods -o custom-columns=NAME:.metadata.name,SA:.spec.serviceAccountName
NAME                                        SA
metrics-collector-0                         metrics-custom-sa
metrics-targetallocator-7d9758598f-v99c6    metrics-targetallocator
my-opentelemetry-operator-cd87d8bdc-r5t64   opentelemetry-operator

Permissions of Collector service account:

$ kubectl auth can-i --as system:serviceaccount:default:metrics-custom-sa --list
Resources                                       Non-Resource URLs                      Resource Names   Verbs
podmonitors.monitoring.coreos.com               []                                     []               [*]
servicemonitors.monitoring.coreos.com           []                                     []               [*]
selfsubjectreviews.authentication.k8s.io        []                                     []               [create]
selfsubjectaccessreviews.authorization.k8s.io   []                                     []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                                     []               [create]
endpointslices.discovery.k8s.io                 []                                     []               [get list watch]
ingresses.networking.k8s.io                     []                                     []               [get list watch]
endpoints                                       []                                     []               [get watch list]
namespaces                                      []                                     []               [get watch list]
nodes/metrics                                   []                                     []               [get watch list]
nodes                                           []                                     []               [get watch list]
pods                                            []                                     []               [get watch list]
services                                        []                                     []               [get watch list]
replicasets.apps                                []                                     []               [get watch list]
                                                [/.well-known/openid-configuration/]   []               [get]
                                                [/.well-known/openid-configuration]    []               [get]
                                                [/api/*]                               []               [get]
                                                [/api]                                 []               [get]
                                                [/apis/*]                              []               [get]
                                                [/apis]                                []               [get]
                                                [/healthz]                             []               [get]
                                                [/healthz]                             []               [get]
                                                [/livez]                               []               [get]
                                                [/livez]                               []               [get]
                                                [/metrics]                             []               [get]
                                                [/openapi/*]                           []               [get]
                                                [/openapi]                             []               [get]
                                                [/openid/v1/jwks/]                     []               [get]
                                                [/openid/v1/jwks]                      []               [get]
                                                [/readyz]                              []               [get]
                                                [/readyz]                              []               [get]
                                                [/version/]                            []               [get]
                                                [/version/]                            []               [get]
                                                [/version]                             []               [get]
                                                [/version]                             []               [get]
configmaps                                      []                                     []               [get]

Permissions of Target Allocator service account:

$ kubectl auth can-i --as system:serviceaccount:default:metrics-targetallocator --list
Resources                                       Non-Resource URLs                      Resource Names   Verbs
podmonitors.monitoring.coreos.com               []                                     []               [*]
servicemonitors.monitoring.coreos.com           []                                     []               [*]
selfsubjectreviews.authentication.k8s.io        []                                     []               [create]
selfsubjectaccessreviews.authorization.k8s.io   []                                     []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                                     []               [create]
endpoints                                       []                                     []               [get list watch]
namespaces                                      []                                     []               [get list watch]
nodes/metrics                                   []                                     []               [get list watch]
nodes                                           []                                     []               [get list watch]
pods                                            []                                     []               [get list watch]
services                                        []                                     []               [get list watch]
endpointslices.discovery.k8s.io                 []                                     []               [get list watch]
ingresses.networking.k8s.io                     []                                     []               [get list watch]
                                                [/.well-known/openid-configuration/]   []               [get]
                                                [/.well-known/openid-configuration]    []               [get]
                                                [/api/*]                               []               [get]
                                                [/api]                                 []               [get]
                                                [/apis/*]                              []               [get]
                                                [/apis]                                []               [get]
                                                [/healthz]                             []               [get]
                                                [/healthz]                             []               [get]
                                                [/livez]                               []               [get]
                                                [/livez]                               []               [get]
                                                [/metrics]                             []               [get]
                                                [/openapi/*]                           []               [get]
                                                [/openapi]                             []               [get]
                                                [/openid/v1/jwks/]                     []               [get]
                                                [/openid/v1/jwks]                      []               [get]
                                                [/readyz]                              []               [get]
                                                [/readyz]                              []               [get]
                                                [/version/]                            []               [get]
                                                [/version/]                            []               [get]
                                                [/version]                             []               [get]
                                                [/version]                             []               [get]
configmaps                                      []                                     []               [get]

Permissions of the operator itself:

$ kubectl auth can-i --as system:serviceaccount:default:opentelemetry-operator --list
Resources                                             Non-Resource URLs                      Resource Names   Verbs
configmaps                                            []                                     []               [create delete get list patch update watch]
persistentvolumeclaims                                []                                     []               [create delete get list patch update watch]
persistentvolumes                                     []                                     []               [create delete get list patch update watch]
pods                                                  []                                     []               [create delete get list patch update watch]
serviceaccounts                                       []                                     []               [create delete get list patch update watch]
services                                              []                                     []               [create delete get list patch update watch]
daemonsets.apps                                       []                                     []               [create delete get list patch update watch]
deployments.apps                                      []                                     []               [create delete get list patch update watch]
statefulsets.apps                                     []                                     []               [create delete get list patch update watch]
horizontalpodautoscalers.autoscaling                  []                                     []               [create delete get list patch update watch]
podmonitors.monitoring.coreos.com                     []                                     []               [create delete get list patch update watch]
servicemonitors.monitoring.coreos.com                 []                                     []               [create delete get list patch update watch]
ingresses.networking.k8s.io                           []                                     []               [create delete get list patch update watch]
opampbridges.opentelemetry.io                         []                                     []               [create delete get list patch update watch]
poddisruptionbudgets.policy                           []                                     []               [create delete get list patch update watch]
routes.route.openshift.io/custom-host                 []                                     []               [create delete get list patch update watch]
routes.route.openshift.io                             []                                     []               [create delete get list patch update watch]
leases.coordination.k8s.io                            []                                     []               [create get list update]
events                                                []                                     []               [create patch]
selfsubjectreviews.authentication.k8s.io              []                                     []               [create]
tokenreviews.authentication.k8s.io                    []                                     []               [create]
selfsubjectaccessreviews.authorization.k8s.io         []                                     []               [create]
selfsubjectrulesreviews.authorization.k8s.io          []                                     []               [create]
subjectaccessreviews.authorization.k8s.io             []                                     []               [create]
instrumentations.opentelemetry.io                     []                                     []               [get list patch update watch]
opentelemetrycollectors.opentelemetry.io              []                                     []               [get list patch update watch]
replicasets.apps                                      []                                     []               [get list watch]
jobs.batch                                            []                                     []               [get list watch]
infrastructures.config.openshift.io/status            []                                     []               [get list watch]
infrastructures.config.openshift.io                   []                                     []               [get list watch]
replicasets.extensions                                []                                     []               [get list watch]
opampbridges.opentelemetry.io/status                  []                                     []               [get patch update]
opentelemetrycollectors.opentelemetry.io/finalizers   []                                     []               [get patch update]
opentelemetrycollectors.opentelemetry.io/status       []                                     []               [get patch update]
configmaps/status                                     []                                     []               [get update patch]
                                                      [/.well-known/openid-configuration/]   []               [get]
                                                      [/.well-known/openid-configuration]    []               [get]
                                                      [/api/*]                               []               [get]
                                                      [/api]                                 []               [get]
                                                      [/apis/*]                              []               [get]
                                                      [/apis]                                []               [get]
                                                      [/healthz]                             []               [get]
                                                      [/healthz]                             []               [get]
                                                      [/livez]                               []               [get]
                                                      [/livez]                               []               [get]
                                                      [/openapi/*]                           []               [get]
                                                      [/openapi]                             []               [get]
                                                      [/openid/v1/jwks/]                     []               [get]
                                                      [/openid/v1/jwks]                      []               [get]
                                                      [/readyz]                              []               [get]
                                                      [/readyz]                              []               [get]
                                                      [/version/]                            []               [get]
                                                      [/version/]                            []               [get]
                                                      [/version]                             []               [get]
                                                      [/version]                             []               [get]
namespaces                                            []                                     []               [list watch]
opampbridges.opentelemetry.io/finalizers              []                                     []               [update]

[davidhaja]

When I tried this scenario on my end and changed this warning message to include the user, it turned out the operator checks the system:serviceaccount:default:<EMPTY SA NAME> serviceaccount instead of the generated serviceaccount name.

Warning: missing the following rules for networking.k8s.io/ingresses: [get,system:serviceaccount:default:,list,system:serviceaccount:default:,watch,system:serviceaccount:default:]
Warning: missing the following rules for nodes: [get,system:serviceaccount:default:,list,system:serviceaccount:default:,watch,system:serviceaccount:default:]
Warning: missing the following rules for nodes/metrics: [get,system:serviceaccount:default:,list,system:serviceaccount:default:,watch,system:serviceaccount:default:]
Warning: missing the following rules for endpoints: [get,system:serviceaccount:default:,list,system:serviceaccount:default:,watch,system:serviceaccount:default:]
Warning: missing the following rules for pods: [get,system:serviceaccount:default:,list,system:serviceaccount:default:,watch,system:serviceaccount:default:]
Warning: missing the following rules for discovery.k8s.io/endpointslices: [get,system:serviceaccount:default:,list,system:serviceaccount:default:,watch,system:serviceaccount:default:]
Warning: missing the following rules for nonResourceURL: /metrics: [get,system:serviceaccount:default:]
Warning: missing the following rules for monitoring.coreos.com/servicemonitors: [*,system:serviceaccount:default:]
Warning: missing the following rules for monitoring.coreos.com/podmonitors: [*,system:serviceaccount:default:]
Warning: missing the following rules for services: [get,system:serviceaccount:default:,list,system:serviceaccount:default:,watch,system:serviceaccount:default:]
Warning: missing the following rules for namespaces: [get,system:serviceaccount:default:,list,system:serviceaccount:default:,watch,system:serviceaccount:default:]
Warning: missing the following rules for configmaps: [get,system:serviceaccount:default:]

I can take care of this and try to implement a fix.