Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubernetes client logging not wrapped in otel collector logging #30928

Closed
ringerc opened this issue Jan 31, 2024 · 7 comments
Closed

kubernetes client logging not wrapped in otel collector logging #30928

ringerc opened this issue Jan 31, 2024 · 7 comments
Labels
bug Something isn't working needs triage New item requiring triage processor/k8sattributes k8s Attributes processor processor/resourcedetection Resource detection processor receiver/k8scluster receiver/kubeletstats receiver/prometheus Prometheus receiver

Comments

@ringerc
Copy link

ringerc commented Jan 31, 2024

Component(s)

processor/k8sattributes, processor/resourcedetection, receiver/k8scluster, receiver/kubeletstats, receiver/prometheus

What happened?

Description

Error logs from kubernetes/client-go library not wrapped and captured in otel collector logger, so they are logged as plaintext when when json logging is on.

Steps to Reproduce

Deploy an OpenTelemetry collector in Kubernetes with

  • receivers..prometheus configured with k8s pod discovery
  • receivers.kubletstats configured
  • receivers.k8s_cluster configured
  • processors.k8sattributes configured

... but omit the ClusterRole and ClusterRoleBinding required for kube RBAC, so as to deliberately induce error logging on k8s apiserver accesses.

Expected Result

Errors from k8s apiserver access are logged through the OpenTelemetry Collector's logging stream, respecting the service.telemetry.logs.encoding setting (json logging) and service.telemetry.logs.level setting (log level).

Actual Result

Errors from k8s apiserver are logged in plaintext even when service.telemetry.logs.encoding is set to json, ignoring log formatting configuration.

E.g.

W1220 20:43:06.585812       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:p-6sbq4blfkb:otel-collector" cannot list resource "replicasets" in API group "apps" at the cluster scope

I suspect this issue involves multiple components that interact with the client-go library, but have not yet isolated exactly which ones.

Edit: Per ticket comments below

  • k8s_cluster - does not wrap client-go logs
  • k8sattributes - does not wrap client-go logs
  • prometheus - k8s discover does not wrap client-go logs
  • kubeletstats - doesn't seem to use k8sclient calls that emit logs directly, all its logs are properly json

Collector version

0.92.0

Environment information

Environment

Kubernetes 1.29 (Ubuntu nodes)

OpenTelemetry Collector configuration

# requires K8S_NODE_NAME injected via kube downward API from DaemonSet
#
receivers:
  prometheus/discovery:
    config:
      scrape_configs:
      - job_name: 'postgres-pod'
        scrape_interval: 1m
        kubernetes_sd_configs:
        - role: pod
        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_node_name]
          action: keep
          regex: ${env:K8S_NODE_NAME}

  kubeletstats:
    collection_interval: 1m
    auth_type: 'serviceAccount'
    endpoint: '${env:K8S_NODE_NAME}:10250'
    insecure_skip_verify: true
    metric_groups:
      - node
      - pod
      - container
      - volume
    extra_metadata_labels:
      - container.id
      - k8s.volume.type
    k8s_api_config:
      auth_type: serviceAccount

  k8s_cluster:
    collection_interval: 1m
    auth_type: serviceAccount
    node_conditions_to_report:
      # https://kubernetes.io/docs/reference/node/node-status/#condition
      - Ready
      - MemoryPressure
    allocatable_types_to_report:
      - cpu
      - memory
    metrics:
      k8s.node.condition:
         enabled: true
      k8s.pod.status_reason:
         enabled: true

exporters:
  debug/basic:
    verbosity: basic

processors:
  resourcedetection/k8snode:
    detectors: ["k8snode"]
    k8snode:
      node_from_env_var: K8S_NODE_NAME
    system:
      hostname_sources: ["os"]

  k8sattributes:
    passthrough: false
    auth_type: "serviceAccount"
    filter:
      node_from_env_var: K8S_NODE_NAME
    pod_association:
      - sources:
        - from: resource_attribute
          name: k8s.pod.uid
      - sources:
        - from: resource_attribute
          name: k8s.pod.ip
      - sources:
        - from: connection
    extract:
      metadata:
        - container.id
        - container.image.name
        - container.image.tag
        - k8s.container.name
        - k8s.cronjob.name
        - k8s.daemonset.name
        - k8s.daemonset.uid
        - k8s.deployment.name
        - k8s.job.name
        - k8s.job.uid
        - k8s.namespace.name
        - k8s.node.name
        - k8s.pod.name
        - k8s.pod.start_time
        - k8s.pod.uid
        - k8s.replicaset.name
        - k8s.replicaset.uid
        - k8s.statefulset.name
        - k8s.statefulset.uid
      labels:
      - tag_name: "app.kubernetes.io/name"
        key: app.kubernetes.io/name
        from: pod
      - tag_name: "app.kubernetes.io/instance"
        key: app.kubernetes.io/instance
        from: pod
      - tag_name: "app.kubernetes.io/version"
        key: app.kubernetes.io/version
        from: pod
      - tag_name: "app.kubernetes.io/component"
        key: app.kubernetes.io/component
        from: pod
      - tag_name: "app.kubernetes.io/part-of"
        key: app.kubernetes.io/part-of
        from: pod
      - tag_name: "app.kubernetes.io/managed-by"
        key: app.kubernetes.io/managed-by
        from: pod

  batch:

service:
  telemetry:
    logs:
      encoding: "json"
      level: "info"
  pipelines:
    metrics:
      receivers:
        - prometheus/discovery
        - k8s_cluster
        - kubeletstats
      processors:
        - k8sattributes
        - batch
      exporters:
        - debug/basic

Log output

E0201 23:39:52.564923       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
W0201 23:39:52.697503       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
E0201 23:39:52.697525       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
W0201 23:39:52.711159       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.DaemonSet: daemonsets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "daemonsets" in API group "apps" at the cluster scope
E0201 23:39:52.711172       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.DaemonSet: failed to list *v1.DaemonSet: daemonsets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "daemonsets" in API group "apps" at the cluster scope
W0201 23:39:52.750020       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
E0201 23:39:52.750045       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
W0201 23:39:52.763790       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "nodes" in API group "" at the cluster scope
E0201
...

(more complete log will be attached)

Additional context

(Extracted from my real config, which is much larger, and as yet untested because it omits various specifics that mean it can no longer be deployed in my real running environment)

Use a kustomization.yaml like:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
configMapGenerator:
  - name: otel-collector-config
    files:
      - otel-collector-config.yaml
    options:
      disableNameSuffixHash: true
resources:
  - daemonset.yaml
  - serviceaccount.yaml

with otel-collector-config.yaml containing the config listed above.

with serviceaccount.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: otel-collector

with daemonset.yaml:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-collector-node-agent
  annotations:
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: otel-collector-node-agent
      app.kubernetes.io/component: otel-collector
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: otel-collector-node-agent
        app.kubernetes.io/component: otel-collector
      annotations:
        kubectl.kubernetes.io/default-container: opentelemetry-collector
    spec:
      containers:
        - name: opentelemetry-collector
          command:
            - "/otelcol-contrib"
            - "--config=/etc/otel-collector-config/otel-collector-config.yaml"
            - "--feature-gates=+k8sattr.rfc3339"
          image: "otel/opentelemetry-collector-contrib:0.93.0"
          securityContext:
            allowPrivilegeEscalation: false
            privileged: false
            readOnlyRootFilesystem: true
            seccompProfile:
              type: RuntimeDefault
          env:
          - name: K8S_NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          volumeMounts:
            - name: otel-collector-config
              mountPath: /etc/otel-collector-config
      serviceAccountName: otel-collector
      volumes:
        - name: otel-collector-config
          configMap:
            name: otel-collector-config
            items:
              - key: otel-collector-config.yaml
                path: otel-collector-config.yaml
@ringerc ringerc added bug Something isn't working needs triage New item requiring triage labels Jan 31, 2024
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@ringerc
Copy link
Author

ringerc commented Feb 1, 2024

Here's a set of logs generated by running the above workload on kind

sample-logs.txt

and here's an archive with a prepared set of mainfests:

otel-30928.zip

that you can apply with

unzip otel-30928.zip
cd otel-30928
kubectl build . | kubectl apply -f -

@ringerc
Copy link
Author

ringerc commented Feb 2, 2024

If I turn off everything but the prometheus receiver, here are the logs

{"level":"info","ts":1706839124.2536318,"caller":"prometheusreceiver@v0.93.0/metrics_receiver.go:282","msg":"Starting scrape manager","kind":"receiver","name":"prometheus/discovery","data_type":"metrics"}
W0202 01:58:44.258533       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
E0202 01:58:44.258567       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
W0202 01:58:45.216484       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
E0202 01:58:45.216509       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
W0202 01:58:47.429298       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
E0202 01:58:47.429332       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
W0202 01:58:50.900448       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
E0202 01:58:50.900473       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope

so it looks like the prometheus scraper's k8s discovery is one source of the log events.

@ringerc
Copy link
Author

ringerc commented Feb 2, 2024

Here's logs when only k8sattributes receiver enabled (I added hostmetrics.scraper.cpu to provide a data source)

➜  otel-30928 kubectl --context=kind-kind logs $(kubectl --context=kind-kind get pod -o name)
{"level":"info","ts":1706839652.09378,"caller":"service@v0.93.0/telemetry.go:76","msg":"Setting up own telemetry..."}
{"level":"info","ts":1706839652.0938776,"caller":"service@v0.93.0/telemetry.go:146","msg":"Serving metrics","address":":8888","level":"Basic"}
{"level":"info","ts":1706839652.0939312,"caller":"exporter@v0.93.0/exporter.go:275","msg":"Development component. May change in the future.","kind":"exporter","data_type":"metrics","name":"debug/basic"}
{"level":"info","ts":1706839652.0951843,"caller":"service@v0.93.0/service.go:139","msg":"Starting otelcol-contrib...","Version":"0.93.0","NumCPU":16}
{"level":"info","ts":1706839652.0951989,"caller":"extensions/extensions.go:34","msg":"Starting extensions..."}
{"level":"info","ts":1706839652.0955966,"caller":"kube/client.go:113","msg":"k8s filtering","kind":"processor","name":"k8sattributes","pipeline":"metrics","labelSelector":"","fieldSelector":"spec.nodeName=kind-control-plane"}
{"level":"info","ts":1706839652.0957146,"caller":"service@v0.93.0/service.go:165","msg":"Everything is ready. Begin running and processing data."}
W0202 02:07:32.101281       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "replicasets" in API group "apps" at the cluster scope
W0202 02:07:32.101283       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
W0202 02:07:32.101289       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Namespace: namespaces "kube-system" is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "namespaces" in API group "" at the cluster scope
E0202 02:07:32.101308       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0202 02:07:32.101308       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Namespace: failed to list *v1.Namespace: namespaces "kube-system" is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "namespaces" in API group "" at the cluster scope
E0202 02:07:32.101309       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
{"level":"info","ts":1706839653.2965133,"msg":"MetricsExporter","kind":"exporter","data_type":"metrics","name":"debug/basic","resource metrics":1,"metrics":1,"data points":128}
W0202 02:07:33.466503       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Namespace: namespaces "kube-system" is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "namespaces" in API group "" at the cluster scope
E0202 02:07:33.466593       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Namespace: failed to list *v1.Namespace: namespaces "kube-system" is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "namespaces" in API group "" at the cluster scope
W0202 02:07:33.526776       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0202 02:07:33.526838       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "replicasets" in API group "apps" at the cluster scope
W0202 02:07:33.598605       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
E0202 02:07:33.598719       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
W0202 02:07:35.217218       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Namespace: namespaces "kube-system" is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "namespaces" in API group "" at the cluster scope
E0202 02:07:35.217246       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Namespace: failed to list *v1.Namespace: namespaces "kube-system" is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "namespaces" in API group "" at the cluster scope
W0202 02:07:35.524130       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
E0202 02:07:35.524160       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
W0202 02:07:36.529253       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0202 02:07:36.529283       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "replicasets" in API group "apps" at the cluster scope

@ringerc
Copy link
Author

ringerc commented Feb 2, 2024

Here's logs when I only have the k8s_cluster receiver enabled

➜  otel-30928 kubectl --context=kind-kind logs $(kubectl --context=kind-kind get pod -o name)
{"level":"info","ts":1706839992.299362,"caller":"service@v0.93.0/telemetry.go:76","msg":"Setting up own telemetry..."}
{"level":"info","ts":1706839992.2995026,"caller":"service@v0.93.0/telemetry.go:146","msg":"Serving metrics","address":":8888","level":"Basic"}
{"level":"info","ts":1706839992.2995825,"caller":"exporter@v0.93.0/exporter.go:275","msg":"Development component. May change in the future.","kind":"exporter","data_type":"metrics","name":"debug/basic"}
{"level":"info","ts":1706839992.3012202,"caller":"service@v0.93.0/service.go:139","msg":"Starting otelcol-contrib...","Version":"0.93.0","NumCPU":16}
{"level":"info","ts":1706839992.3012378,"caller":"extensions/extensions.go:34","msg":"Starting extensions..."}
{"level":"warn","ts":1706839992.3150103,"caller":"k8sclusterreceiver@v0.93.0/watcher.go:144","msg":"Server doesn't support any of the group versions defined for the kind","kind":"receiver","name":"k8s_cluster","data_type":"metrics","kind":"HorizontalPodAutoscaler"}
{"level":"info","ts":1706839992.3161683,"caller":"service@v0.93.0/service.go:165","msg":"Everything is ready. Begin running and processing data."}
{"level":"info","ts":1706839992.3162425,"caller":"k8sclusterreceiver@v0.93.0/receiver.go:53","msg":"Starting shared informers and wait for initial cache sync.","kind":"receiver","name":"k8s_cluster","data_type":"metrics"}
W0202 02:13:12.317216       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.DaemonSet: daemonsets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "daemonsets" in API group "apps" at the cluster scope
W0202 02:13:12.317216       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.ResourceQuota: resourcequotas is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "resourcequotas" in API group "" at the cluster scope
E0202 02:13:12.317266       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.ResourceQuota: failed to list *v1.ResourceQuota: resourcequotas is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "resourcequotas" in API group "" at the cluster scope
E0202 02:13:12.317265       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.DaemonSet: failed to list *v1.DaemonSet: daemonsets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "daemonsets" in API group "apps" at the cluster scope
W0202 02:13:12.317385       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0202 02:13:12.317412       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
W0202 02:13:12.317411       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0202 02:13:12.317438       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "replicasets" in API group "apps" at the cluster scope
W0202 02:13:12.317439       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "services" in API group "" at the cluster scope
E0202 02:13:12.317459       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "services" in API group "" at the cluster scope
W0202 02:13:12.317560       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0202 02:13:12.317582       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "statefulsets" in API group "apps" at the cluster scope
W0202 02:13:12.317595       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "namespaces" in API group "" at the cluster scope
W0202 02:13:12.317597       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Job: jobs.batch is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "jobs" in API group "batch" at the cluster scope
E0202 02:13:12.317611       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Namespace: failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "namespaces" in API group "" at the cluster scope
E0202 02:13:12.317614       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Job: failed to list *v1.Job: jobs.batch is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "jobs" in API group "batch" at the cluster scope
W0202 02:13:12.317664       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.CronJob: cronjobs.batch is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "cronjobs" in API group "batch" at the cluster scope
E0202 02:13:12.317684       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.CronJob: failed to list *v1.CronJob: cronjobs.batch is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "cronjobs" in API group "batch" at the cluster scope
W0202 02:13:12.317752       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "nodes" in API group "" at the cluster scope
E0202 02:13:12.317769       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "nodes" in API group "" at the cluster scope
W0202 02:13:12.317929       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
E0202 02:13:12.317942       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
W0202 02:13:12.317987       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "deployments" in API group "apps" at the cluster scope
E0202 02:13:12.318000       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Deployment: failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "deployments" in API group "apps" at the cluster scope
W0202 02:13:13.139579       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "nodes" in API group "" at the cluster scope
E0202 02:13:13.139615       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "nodes" in API group "" at the cluster scope
W0202 02:13:13.185262       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
E0202 02:13:13.185288       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
W0202 02:13:13.259740       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0202 02:13:13.259770       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
W0202 02:13:13.288686       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0202 02:13:13.288713       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "statefulsets" in API group "apps" at the cluster scope
W0202 02:13:13.481825       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Job: jobs.batch is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "jobs" in API group "batch" at the cluster scope
E0202 02:13:13.481849       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Job: failed to list *v1.Job: jobs.batch is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "jobs" in API group "batch" at the cluster scope
W0202 02:13:13.506380       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "namespaces" in API group "" at the cluster scope
E0202 02:13:13.506426       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Namespace: failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "namespaces" in API group "" at the cluster scope
W0202 02:13:13.566242       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.CronJob: cronjobs.batch is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "cronjobs" in API group "batch" at the cluster scope
E0202 02:13:13.566269       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.CronJob: failed to list *v1.CronJob: cronjobs.batch is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "cronjobs" in API group "batch" at the cluster scope
W0202 02:13:13.581264       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0202 02:13:13.581288       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "replicasets" in API group "apps" at the cluster scope
W0202 02:13:13.623465       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.ResourceQuota: resourcequotas is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "resourcequotas" in API group "" at the cluster scope
E0202 02:13:13.623491       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.ResourceQuota: failed to list *v1.ResourceQuota: resourcequotas is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "resourcequotas" in API group "" at the cluster scope
W0202 02:13:13.627191       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "services" in API group "" at the cluster scope
E0202 02:13:13.627212       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "services" in API group "" at the cluster scope
W0202 02:13:13.718669       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.DaemonSet: daemonsets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "daemonsets" in API group "apps" at the cluster scope
E0202 02:13:13.718692       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.DaemonSet: failed to list *v1.DaemonSet: daemonsets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "daemonsets" in API group "apps" at the cluster scope
W0202 02:13:13.859413       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "deployments" in API group "apps" at the cluster scope
E0202 02:13:13.859435       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Deployment: failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "deployments" in API group "apps" at the cluster scope
W0202 02:13:15.312294       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
E0202 02:13:15.312360       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
W0202 02:13:15.357289       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.CronJob: cronjobs.batch is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "cronjobs" in API group "batch" at the cluster scope
E0202 02:13:15.357357       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.CronJob: failed to list *v1.CronJob: cronjobs.batch is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "cronjobs" in API group "batch" at the cluster scope
W0202 02:13:15.552681       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0202 02:13:15.552721       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "replicasets" in API group "apps" at the cluster scope
W0202 02:13:15.556914       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "nodes" in API group "" at the cluster scope
E0202 02:13:15.556960       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "nodes" in API group "" at the cluster scope
W0202 02:13:15.566828       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0202 02:13:15.566855       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
W0202 02:13:15.775685       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Job: jobs.batch is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "jobs" in API group "batch" at the cluster scope
E0202 02:13:15.775749       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Job: failed to list *v1.Job: jobs.batch is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "jobs" in API group "batch" at the cluster scope
W0202 02:13:15.819500       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0202 02:13:15.819576       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "statefulsets" in API group "apps" at the cluster scope
W0202 02:13:15.840311       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "namespaces" in API group "" at the cluster scope
E0202 02:13:15.840343       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Namespace: failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "namespaces" in API group "" at the cluster scope
W0202 02:13:16.365898       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.ResourceQuota: resourcequotas is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "resourcequotas" in API group "" at the cluster scope
E0202 02:13:16.365928       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.ResourceQuota: failed to list *v1.ResourceQuota: resourcequotas is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "resourcequotas" in API group "" at the cluster scope
W0202 02:13:16.385089       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.DaemonSet: daemonsets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "daemonsets" in API group "apps" at the cluster scope
E0202 02:13:16.385119       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.DaemonSet: failed to list *v1.DaemonSet: daemonsets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "daemonsets" in API group "apps" at the cluster scope
W0202 02:13:16.625468       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "services" in API group "" at the cluster scope
E0202 02:13:16.625491       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "services" in API group "" at the cluster scope
W0202 02:13:16.951344       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "deployments" in API group "apps" at the cluster scope
E0202 02:13:16.951372       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Deployment: failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "deployments" in API group "apps" at the cluster scope
W0202 02:13:18.967102       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0202 02:13:18.967178       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "replicasets" in API group "apps" at the cluster scope
W0202 02:13:19.925992       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.DaemonSet: daemonsets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "daemonsets" in API group "apps" at the cluster scope
E0202 02:13:19.926016       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.DaemonSet: failed to list *v1.DaemonSet: daemonsets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "daemonsets" in API group "apps" at the cluster scope
W0202 02:13:20.233197       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "deployments" in API group "apps" at the cluster scope
E0202 02:13:20.233220       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Deployment: failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "deployments" in API group "apps" at the cluster scope
W0202 02:13:20.626028       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0202 02:13:20.626055       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
W0202 02:13:20.662873       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "namespaces" in API group "" at the cluster scope
E0202 02:13:20.662895       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Namespace: failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "namespaces" in API group "" at the cluster scope
W0202 02:13:20.847044       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Job: jobs.batch is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "jobs" in API group "batch" at the cluster scope
E0202 02:13:20.847093       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Job: failed to list *v1.Job: jobs.batch is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "jobs" in API group "batch" at the cluster scope
W0202 02:13:20.902142       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0202 02:13:20.902169       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "statefulsets" in API group "apps" at the cluster scope
W0202 02:13:21.395899       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
E0202 02:13:21.395926       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
W0202 02:13:21.602410       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "nodes" in API group "" at the cluster scope
E0202 02:13:21.602435       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "nodes" in API group "" at the cluster scope
W0202 02:13:21.637460       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.CronJob: cronjobs.batch is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "cronjobs" in API group "batch" at the cluster scope
E0202 02:13:21.637486       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.CronJob: failed to list *v1.CronJob: cronjobs.batch is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "cronjobs" in API group "batch" at the cluster scope
W0202 02:13:21.739878       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.ResourceQuota: resourcequotas is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "resourcequotas" in API group "" at the cluster scope
E0202 02:13:21.739908       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.ResourceQuota: failed to list *v1.ResourceQuota: resourcequotas is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "resourcequotas" in API group "" at the cluster scope
W0202 02:13:22.437512       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "services" in API group "" at the cluster scope
E0202 02:13:22.437563       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "services" in API group "" at the cluster scope
W0202 02:13:28.633974       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.ResourceQuota: resourcequotas is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "resourcequotas" in API group "" at the cluster scope
E0202 02:13:28.634040       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.ResourceQuota: failed to list *v1.ResourceQuota: resourcequotas is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "resourcequotas" in API group "" at the cluster scope
W0202 02:13:28.775272       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0202 02:13:28.775355       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "statefulsets" in API group "apps" at the cluster scope
W0202 02:13:29.226287       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
E0202 02:13:29.226356       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "pods" in API group "" at the cluster scope
W0202 02:13:29.300658       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.DaemonSet: daemonsets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "daemonsets" in API group "apps" at the cluster scope
E0202 02:13:29.300743       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.DaemonSet: failed to list *v1.DaemonSet: daemonsets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "daemonsets" in API group "apps" at the cluster scope
W0202 02:13:29.322166       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0202 02:13:29.322194       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "replicasets" in API group "apps" at the cluster scope
W0202 02:13:29.644323       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Job: jobs.batch is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "jobs" in API group "batch" at the cluster scope
E0202 02:13:29.644346       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Job: failed to list *v1.Job: jobs.batch is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "jobs" in API group "batch" at the cluster scope
W0202 02:13:30.526893       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "deployments" in API group "apps" at the cluster scope
E0202 02:13:30.526919       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Deployment: failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "deployments" in API group "apps" at the cluster scope
W0202 02:13:31.250049       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.CronJob: cronjobs.batch is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "cronjobs" in API group "batch" at the cluster scope
E0202 02:13:31.250077       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.CronJob: failed to list *v1.CronJob: cronjobs.batch is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "cronjobs" in API group "batch" at the cluster scope
W0202 02:13:31.356021       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0202 02:13:31.356092       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
W0202 02:13:32.358216       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "namespaces" in API group "" at the cluster scope
E0202 02:13:32.358242       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Namespace: failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "namespaces" in API group "" at the cluster scope
W0202 02:13:32.592057       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "services" in API group "" at the cluster scope
E0202 02:13:32.592122       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "services" in API group "" at the cluster scope
W0202 02:13:33.048051       1 reflector.go:535] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "nodes" in API group "" at the cluster scope
E0202 02:13:33.048117       1 reflector.go:147] k8s.io/client-go@v0.28.4/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:default:otel-collector" cannot list resource "nodes" in API group "" at the cluster scope

@ringerc
Copy link
Author

ringerc commented Feb 2, 2024

Logs when only kubeletstats is enabled

➜  otel-30928 kubectl --context=kind-kind logs $(kubectl --context=kind-kind get pod -o name)
{"level":"info","ts":1706840068.3006518,"caller":"service@v0.93.0/telemetry.go:76","msg":"Setting up own telemetry..."}
{"level":"info","ts":1706840068.30074,"caller":"service@v0.93.0/telemetry.go:146","msg":"Serving metrics","address":":8888","level":"Basic"}
{"level":"info","ts":1706840068.300833,"caller":"exporter@v0.93.0/exporter.go:275","msg":"Development component. May change in the future.","kind":"exporter","data_type":"metrics","name":"debug/basic"}
{"level":"warn","ts":1706840068.30634,"caller":"metadata/generated_metrics.go:2894","msg":"[WARNING] `container.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric container.cpu.usage instead.","kind":"receiver","name":"kubeletstats","data_type":"metrics"}
{"level":"warn","ts":1706840068.306368,"caller":"metadata/generated_metrics.go:2897","msg":"[WARNING] `k8s.node.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric k8s.node.cpu.usage instead.","kind":"receiver","name":"kubeletstats","data_type":"metrics"}
{"level":"warn","ts":1706840068.3063755,"caller":"metadata/generated_metrics.go:2900","msg":"[WARNING] `k8s.pod.cpu.utilization` should not be enabled: This metric will be disabled in a future release. Use metric k8s.pod.cpu.usage instead.","kind":"receiver","name":"kubeletstats","data_type":"metrics"}
{"level":"warn","ts":1706840068.3064184,"caller":"metadata/generated_metrics.go:2894","msg":"[WARNING] `container.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric container.cpu.usage instead.","kind":"receiver","name":"kubeletstats","data_type":"metrics"}
{"level":"warn","ts":1706840068.3064265,"caller":"metadata/generated_metrics.go:2897","msg":"[WARNING] `k8s.node.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric k8s.node.cpu.usage instead.","kind":"receiver","name":"kubeletstats","data_type":"metrics"}
{"level":"warn","ts":1706840068.3064313,"caller":"metadata/generated_metrics.go:2900","msg":"[WARNING] `k8s.pod.cpu.utilization` should not be enabled: This metric will be disabled in a future release. Use metric k8s.pod.cpu.usage instead.","kind":"receiver","name":"kubeletstats","data_type":"metrics"}
{"level":"warn","ts":1706840068.3064516,"caller":"metadata/generated_metrics.go:2894","msg":"[WARNING] `container.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric container.cpu.usage instead.","kind":"receiver","name":"kubeletstats","data_type":"metrics"}
{"level":"warn","ts":1706840068.3064585,"caller":"metadata/generated_metrics.go:2897","msg":"[WARNING] `k8s.node.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric k8s.node.cpu.usage instead.","kind":"receiver","name":"kubeletstats","data_type":"metrics"}
{"level":"warn","ts":1706840068.3064637,"caller":"metadata/generated_metrics.go:2900","msg":"[WARNING] `k8s.pod.cpu.utilization` should not be enabled: This metric will be disabled in a future release. Use metric k8s.pod.cpu.usage instead.","kind":"receiver","name":"kubeletstats","data_type":"metrics"}
{"level":"warn","ts":1706840068.3064795,"caller":"metadata/generated_metrics.go:2894","msg":"[WARNING] `container.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric container.cpu.usage instead.","kind":"receiver","name":"kubeletstats","data_type":"metrics"}
{"level":"warn","ts":1706840068.3064837,"caller":"metadata/generated_metrics.go:2897","msg":"[WARNING] `k8s.node.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric k8s.node.cpu.usage instead.","kind":"receiver","name":"kubeletstats","data_type":"metrics"}
{"level":"warn","ts":1706840068.3064873,"caller":"metadata/generated_metrics.go:2900","msg":"[WARNING] `k8s.pod.cpu.utilization` should not be enabled: This metric will be disabled in a future release. Use metric k8s.pod.cpu.usage instead.","kind":"receiver","name":"kubeletstats","data_type":"metrics"}
{"level":"info","ts":1706840068.3074691,"caller":"service@v0.93.0/service.go:139","msg":"Starting otelcol-contrib...","Version":"0.93.0","NumCPU":16}
{"level":"info","ts":1706840068.3074841,"caller":"extensions/extensions.go:34","msg":"Starting extensions..."}
{"level":"info","ts":1706840068.3075335,"caller":"service@v0.93.0/service.go:165","msg":"Everything is ready. Begin running and processing data."}
{"level":"error","ts":1706840069.3327367,"caller":"kubeletstatsreceiver@v0.93.0/scraper.go:77","msg":"call to /stats/summary endpoint failed","kind":"receiver","name":"kubeletstats","data_type":"metrics","error":"kubelet request GET https://kind-control-plane:10250/stats/summary failed - \"403 Forbidden\", response: \"Forbidden (user=system:serviceaccount:default:otel-collector, verb=get, resource=nodes, subresource=stats)\"","stacktrace":"github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kubeletstatsreceiver.(*kubletScraper).scrape\n\tgithub.com/open-telemetry/opentelemetry-collector-contrib/receiver/kubeletstatsreceiver@v0.93.0/scraper.go:77\ngo.opentelemetry.io/collector/receiver/scraperhelper.ScrapeFunc.Scrape\n\tgo.opentelemetry.io/collector/receiver@v0.93.0/scraperhelper/scraper.go:20\ngo.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport\n\tgo.opentelemetry.io/collector/receiver@v0.93.0/scraperhelper/scrapercontroller.go:197\ngo.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1\n\tgo.opentelemetry.io/collector/receiver@v0.93.0/scraperhelper/scrapercontroller.go:172"}
{"level":"error","ts":1706840069.3327897,"caller":"scraperhelper/scrapercontroller.go:200","msg":"Error scraping metrics","kind":"receiver","name":"kubeletstats","data_type":"metrics","error":"kubelet request GET https://kind-control-plane:10250/stats/summary failed - \"403 Forbidden\", response: \"Forbidden (user=system:serviceaccount:default:otel-collector, verb=get, resource=nodes, subresource=stats)\"","scraper":"kubeletstats","stacktrace":"go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport\n\tgo.opentelemetry.io/collector/receiver@v0.93.0/scraperhelper/scrapercontroller.go:200\ngo.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1\n\tgo.opentelemetry.io/collector/receiver@v0.93.0/scraperhelper/scrapercontroller.go:172"}

@ringerc
Copy link
Author

ringerc commented Feb 2, 2024

Turns out I filed a previous bug for this open-telemetry/opentelemetry-collector#8765 but only just found it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working needs triage New item requiring triage processor/k8sattributes k8s Attributes processor processor/resourcedetection Resource detection processor receiver/k8scluster receiver/kubeletstats receiver/prometheus Prometheus receiver
Projects
None yet
Development

No branches or pull requests

1 participant