-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kubernetes client logging not wrapped in otel collector logging #30928
Comments
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
Here's a set of logs generated by running the above workload on and here's an archive with a prepared set of mainfests: that you can apply with
|
If I turn off everything but the
so it looks like the |
Here's logs when only
|
Here's logs when I only have the
|
Logs when only
|
Turns out I filed a previous bug for this open-telemetry/opentelemetry-collector#8765 but only just found it. |
Component(s)
processor/k8sattributes, processor/resourcedetection, receiver/k8scluster, receiver/kubeletstats, receiver/prometheus
What happened?
Description
Error logs from
kubernetes/client-go
library not wrapped and captured in otel collector logger, so they are logged as plaintext when when json logging is on.Steps to Reproduce
Deploy an OpenTelemetry collector in Kubernetes with
receivers..prometheus
configured with k8s pod discoveryreceivers.kubletstats
configuredreceivers.k8s_cluster
configuredprocessors.k8sattributes
configured... but omit the
ClusterRole
andClusterRoleBinding
required for kube RBAC, so as to deliberately induce error logging on k8s apiserver accesses.Expected Result
Errors from k8s apiserver access are logged through the OpenTelemetry Collector's logging stream, respecting the
service.telemetry.logs.encoding
setting (json logging) andservice.telemetry.logs.level
setting (log level).Actual Result
Errors from k8s apiserver are logged in plaintext even when
service.telemetry.logs.encoding
is set tojson
, ignoring log formatting configuration.E.g.
I suspect this issue involves multiple components that interact with the
client-go
library, but have not yet isolated exactly which ones.Edit: Per ticket comments below
k8s_cluster
- does not wrapclient-go
logsk8sattributes
- does not wrapclient-go
logsprometheus
- k8s discover does not wrapclient-go
logskubeletstats
- doesn't seem to use k8sclient calls that emit logs directly, all its logs are properly jsonCollector version
0.92.0
Environment information
Environment
Kubernetes 1.29 (Ubuntu nodes)
OpenTelemetry Collector configuration
Log output
(more complete log will be attached)
Additional context
(Extracted from my real config, which is much larger, and as yet untested because it omits various specifics that mean it can no longer be deployed in my real running environment)
Use a
kustomization.yaml
like:with
otel-collector-config.yaml
containing the config listed above.with
serviceaccount.yaml
:with
daemonset.yaml
:The text was updated successfully, but these errors were encountered: