FluentBit - Collector - Loki : Failed to transform logs into Loki log streams #16368
Labels
Comments
nlamirault
added
bug
|
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
|
This is happening because there are no labels for the items that were exported. I see you had two batches, one with 1000 records and one with 829. Only the smaller batch seems to have failed. Perhaps that one didn't contain the expected log attributes? The Loki exporter has been changed recently to export data even if no labels exist. This requires configuring the exporter in a slightly different way, like the following: receivers:
otlp:
protocols:
grpc:
http:
fluentforward:
endpoint: 0.0.0.0:24224
processors:
attributes:
actions:
- action: insert
key: loki.attribute.labels
value: [container_name, container_id]
resource:
attributes:
- action: insert
key: collector.name
value: "${KUBE_POD_NAME}"
batch:
send_batch_max_size: 1000
timeout: 15s
send_batch_size : 800
# Data sources: traces, metrics, logs
memory_limiter:
limit_percentage: 90
spike_limit_percentage: 30
check_interval: 5s
exporters:
logging:
loglevel: info
# https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/lokiexporter/README.md
loki:
endpoint: http://loki-gateway.logging.svc.cluster.local:80/loki/api/v1/push
headers:
X-Scope-OrgID: homelab
tls:
insecure: true
extensions:
health_check:
memory_ballast:
size_in_percentage: 20
pprof:
endpoint: :1888
zpages:
endpoint: :55679
service:
telemetry:
logs:
level: info
metrics:
level: detailed
address: 0.0.0.0:8888
extensions:
- health_check
- memory_ballast
- pprof
- zpages
pipelines:
logs:
receivers:
- fluentforward
# - otlp
processors:
- batch
- memory_limiter
- resource
# - attributes
exporters:
- logging
- lokiWith your current configuration, you see this in the logs: With the new configuration, you should see something different. Note that we do NOT encourage you to use trace IDs in labels, as those are inherently high cardinality and go against the best practices for Loki. We have removed native ways of setting this value. |
|
cc @mar4uk |
|
Thanks @jpkrohling i will try This configuration |
|
thanks @jpkrohling it works now : I've got another problem/question : in Loki, i can choose only one filter :
|
|
You'd have to add a hint with the attributes to be promoted to labels: After that, add it to the pipeline: For more details on this, refer to the readme part of the Loki exporter. I'm closing this, but feel free if you think the original problem hasn't been addressed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Component(s)
exporter/loki, receiver/fluentforward
What happened?
Description
I deploy on a Kubernetes cluster FluentBit, OpenTelemetry Collector and Loki. It seems i can't send logs to Loki
Steps to Reproduce
Expected Result
Actual Result
With debug output :
Configuration :
Collector version
0.64.0
Environment information
Environment
OS: (e.g., "Ubuntu 20.04") Kubernetes 1.22
Compiler(if manually compiled): (e.g., "go 14.2")
OpenTelemetry Collector configuration
Log output
2022-11-18T06:41:16.744Z info service/telemetry.go:110 Setting up own telemetry... 2022-11-18T06:41:16.748Z info service/telemetry.go:140 Serving Prometheus metrics {"address": "0.0.0.0:8888", "level": "detailed"} 2022-11-18T06:41:16.749Z info components/components.go:30 In development component. May change in the future. {"kind": "exporter", "data_type": "logs", "name": "logging", "stability": "in development"} 2022-11-18T06:41:16.751Z warn loggingexporter@v0.64.0/factory.go:110 'loglevel' option is deprecated in favor of 'verbosity'. Set 'verbosity' to equivalent value to preserve behavior. {"kind": "exporter", "data_type": "logs ", "name": "logging", "loglevel": "info", "equivalent verbosity level": "normal"} 2022-11-18T06:41:16.755Z info lokiexporter@v0.64.0/legacy_exporter.go:59 using the legacy Loki exporter {"kind": "exporter", "data_type": "logs", "name": "loki"} 2022-11-18T06:41:16.761Z info memorylimiterprocessor@v0.64.0/memorylimiter.go:143 Using percentage memory limiter {"kind": "processor", "name": "memory_limiter", "pipeline": "logs", "total_memory_mib": 3072, "limit_percentage ": 90, "spike_limit_percentage": 30} 2022-11-18T06:41:16.761Z info memorylimiterprocessor@v0.64.0/memorylimiter.go:112 Memory limiter configured {"kind": "processor", "name": "memory_limiter", "pipeline": "logs", "limit_mib": 2764, "spike_limit_mib": 921, "check_interval": 5} 2022-11-18T06:41:16.763Z info service/service.go:89 Starting otelcol-contrib... {"Version": "0.64.0", "NumCPU": 4} 2022-11-18T06:41:16.763Z info extensions/extensions.go:41 Starting extensions... 2022-11-18T06:41:16.763Z info extensions/extensions.go:44 Extension is starting... {"kind": "extension", "name": "pprof"} 2022-11-18T06:41:16.763Z info pprofextension@v0.64.0/pprofextension.go:71 Starting net/http/pprof server {"kind": "extension", "name": "pprof", "config": {"ExtensionConfig":null,"TCPAddr":{"Endpoint":":1888"},"BlockProfileFr action":0,"MutexProfileFraction":0,"SaveToFile":""}} 2022-11-18T06:41:16.767Z info extensions/extensions.go:48 Extension started. {"kind": "extension", "name": "pprof"} 2022-11-18T06:41:16.767Z info extensions/extensions.go:44 Extension is starting... {"kind": "extension", "name": "zpages"} 2022-11-18T06:41:16.767Z info zpagesextension@v0.64.0/zpagesextension.go:64 Registered zPages span processor on tracer provider {"kind": "extension", "name": "zpages"} 2022-11-18T06:41:16.767Z info zpagesextension@v0.64.0/zpagesextension.go:74 Registered Host's zPages {"kind": "extension", "name": "zpages"} 2022-11-18T06:41:16.767Z info zpagesextension@v0.64.0/zpagesextension.go:86 Starting zPages extension {"kind": "extension", "name": "zpages", "config": {"ExtensionConfig":null,"TCPAddr":{"Endpoint":":55679"}}} 2022-11-18T06:41:16.767Z info extensions/extensions.go:48 Extension started. {"kind": "extension", "name": "zpages"} 2022-11-18T06:41:16.767Z info extensions/extensions.go:44 Extension is starting... {"kind": "extension", "name": "health_check"} 2022-11-18T06:41:16.767Z info healthcheckextension@v0.64.0/healthcheckextension.go:44 Starting health_check extension {"kind": "extension", "name": "health_check", "config": {"ExtensionConfig":null,"Endpoint":"0.0.0.0:13133","TLS Setting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"Path":"/","CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}} 2022-11-18T06:41:16.768Z warn internal/warning.go:51 Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks {"kind": "extension", "name": "health_check", " documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"} 2022-11-18T06:41:16.774Z info extensions/extensions.go:48 Extension started. {"kind": "extension", "name": "health_check"} 2022-11-18T06:41:16.774Z info extensions/extensions.go:44 Extension is starting... {"kind": "extension", "name": "memory_ballast"} 2022-11-18T06:41:16.863Z info ballastextension@v0.64.0/memory_ballast.go:55 Setting memory ballast {"kind": "extension", "name": "memory_ballast", "MiBs": 614} 2022-11-18T06:41:16.863Z info extensions/extensions.go:48 Extension started. {"kind": "extension", "name": "memory_ballast"} 2022-11-18T06:41:16.863Z info pipelines/pipelines.go:74 Starting exporters... 2022-11-18T06:41:16.863Z info pipelines/pipelines.go:78 Exporter is starting... {"kind": "exporter", "data_type": "logs", "name": "logging"} 2022-11-18T06:41:16.863Z info pipelines/pipelines.go:82 Exporter started. {"kind": "exporter", "data_type": "logs", "name": "logging"} 2022-11-18T06:41:16.863Z info pipelines/pipelines.go:78 Exporter is starting... {"kind": "exporter", "data_type": "logs", "name": "loki"} 2022-11-18T06:41:16.863Z info pipelines/pipelines.go:82 Exporter started. {"kind": "exporter", "data_type": "logs", "name": "loki"} 2022-11-18T06:41:16.863Z info pipelines/pipelines.go:86 Starting processors... 2022-11-18T06:41:16.863Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "resource", "pipeline": "logs"} 2022-11-18T06:41:16.864Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "resource", "pipeline": "logs"} 2022-11-18T06:41:16.864Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "memory_limiter", "pipeline": "logs"} 2022-11-18T06:41:16.864Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "memory_limiter", "pipeline": "logs"} 2022-11-18T06:41:16.864Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "batch", "pipeline": "logs"} 2022-11-18T06:41:16.864Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "batch", "pipeline": "logs"} 2022-11-18T06:41:16.864Z info pipelines/pipelines.go:98 Starting receivers... 2022-11-18T06:41:16.864Z info pipelines/pipelines.go:102 Receiver is starting... {"kind": "receiver", "name": "fluentforward", "pipeline": "logs"} 2022-11-18T06:41:16.864Z info pipelines/pipelines.go:106 Receiver started. {"kind": "receiver", "name": "fluentforward", "pipeline": "logs"} 2022-11-18T06:41:16.864Z info healthcheck/handler.go:129 Health Check state change {"kind": "extension", "name": "health_check", "status": "ready"} 2022-11-18T06:41:16.864Z info service/service.go:106 Everything is ready. Begin running and processing data. 2022-11-18T06:41:17.987Z info LogsExporter {"kind": "exporter", "data_type": "logs", "name": "logging", "#logs": 1000} 2022-11-18T06:41:17.990Z info LogsExporter {"kind": "exporter", "data_type": "logs", "name": "logging", "#logs": 829} 2022-11-18T06:41:17.992Z error exporterhelper/queued_retry.go:394 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "logs", "name": "loki", "error": "Permanent error: failed to transform logs into Loki log streams", "dropped_items": 829} go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send go.opentelemetry.io/collector@v0.64.0/exporter/exporterhelper/queued_retry.go:394 go.opentelemetry.io/collector/exporter/exporterhelper.(*logsExporterWithObservability).send go.opentelemetry.io/collector@v0.64.0/exporter/exporterhelper/logs.go:131 go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1 go.opentelemetry.io/collector@v0.64.0/exporter/exporterhelper/queued_retry.go:205 go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1 go.opentelemetry.io/collector@v0.64.0/exporter/exporterhelper/internal/bounded_memory_queue.go:61 2022-11-18T06:41:18.995Z info LogsExporter {"kind": "exporter", "data_type": "logs", "name": "logging", "#logs": 1000}Additional context
No response
The text was updated successfully, but these errors were encountered: