[exporter/loki]Collector crashing using latest image 'error decoding 'exporters': error reading configuration for "loki"'

[MysterioO]

Component(s)

exporter/loki

What happened?

Description

Using latest collector version we get our pods in CrashLoopBackOff, with error posted in log output. Once we rollback to otel/opentelemetry-collector-contrib:0.75.0, collector starts working as before. What changed has been introduced with latest image to be causing this ?

Collector version

latest

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

processors:
      batch:
        send_batch_size: 2000
        send_batch_max_size: 4000
      
      #Get k8s attributes 
      k8sattributes:
        auth_type: "serviceAccount"
        filter:
          node_from_env_var: KUBE_NODE_NAME
        pod_association:
          - sources:
              - from: resource_attribute
                name: k8s.pod.uid

      resource:
        attributes:
        - action: insert
          key: k8s.cluster.name
          value: $CLUSTER_NAME
        - action: insert
          key: cluster
          value: $CLUSTER_NAME
        - action: insert
          key: loki.tenant
          from_attribute: k8s.deployment.name
        - action: insert
          key: service.name
          from_attribute: k8s.deployment.name


    exporters:
      logging:
        verbosity: normal #detailed
        sampling_initial: 5
        sampling_thereafter: 200

      loki:
        endpoint: redacted
        tls:
          insecure_skip_verify: true
        labels:
          resource:
            k8s.container.name: k8s_container_name
            k8s.pod.name: k8s_pod_name
            k8s.node.name: k8s_node_name
            k8s.cluster.name: k8s_cluster_name
            k8s.namespace.name: k8s_namespace_name
            k8s.deployment.name: k8s_deployment_name

    service:
      pipelines:
        logs:
          receivers: 
            - filelog
          processors:  
            - batch
            - k8sattributes
            - resource
          exporters:
            - loki
            - logging

Log output

Error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:

* error decoding 'exporters': error reading configuration for "loki": 1 error(s) decoding:

* '' has invalid keys: labels
2023/05/03 07:47:22 collector server run finished with error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:

* error decoding 'exporters': error reading configuration for "loki": 1 error(s) decoding:

* '' has invalid keys: labels

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/loki: @gramidt @gouthamve @jpkrohling @kovrus @mar4uk

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[mar4uk]

Hey!
There was breaking change in the latest version (v.0.76.1):

exporter/lokiexporter: Remove deprecated Loki exporters parameters. (#15653, #15365) The support of labels.{attributes/resource}, labels.record, tenant, | and format configuration parameters are dropped in favor of attribute hints way of configuring the exporter.

From v0.76.1 labels for loki should be configured with attribute hints. See examples in the loki exporter README https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/lokiexporter#labels

In your case, one more action should be added to the resource processor:

- action: insert
   key: loki.resource.labels
   value: k8s.container.name, k8s.pod.name, k8s.node.name, k8s.cluster.name, k8s.namespace.name, k8s.deployment.name

and labels section should be removed from the loki exporter config

Could you please try this? Let me know if there are any problems with updating configuration

[jpkrohling]

Can this be closed?

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/loki: @gramidt @gouthamve @jpkrohling @mar4uk

See Adding Labels via Comments if you do not have permissions to add labels yourself.