[exporter/loadbalancing] v0.95.0 getting invalid configuration error: exporters::loadbalancing: requires a non-empty "endpoint"

[ryantanjunming]

Component(s)

exporter/loadbalancing

What happened?

Description

Trying to get tail-sampling working, no issues in v0.91.0 and v0.93

Getting the following only in v0.95.0,
Error: invalid configuration: exporters::loadbalancing: requires a non-empty "endpoint" 2024/02/27 04:43:48 collector server run finished with error: invalid configuration: exporters::loadbalancing: requires a non-empty "endpoint"

Steps to Reproduce

Deployed in a kind cluster with DaemonSet agent with the following loadbalancing exporter config

loadbalancing:
    protocol:
      otlp:
        tls:
          insecure: true
    resolver:
      dns:
        hostname: coralogix-opentelemetry-gateway
    routing_key: traceID

Expected Result

Actual Result

Collector version

v0.95.0

Environment information

Environment

OS: Kind Cluster, single agent

OpenTelemetry Collector configuration

exporters:
  coralogix:
    application_name: 'kind-tail-sampling-otel'
    application_name_attributes:
    - k8s.namespace.name
    - service.namespace
    domain: 'coralogixsg.com'
    private_key: ${CORALOGIX_PRIVATE_KEY}
    subsystem_name: 'kind-tail-sampling-integration'
    subsystem_name_attributes:
    - k8s.deployment.name
    - k8s.statefulset.name
    - k8s.daemonset.name
    - k8s.cronjob.name
    - service.name
    timeout: 30s
  debug: {}
  loadbalancing:
    protocol:
      otlp:
        tls:
          insecure: true
    resolver:
      dns:
        hostname: coralogix-opentelemetry-gateway
    routing_key: traceID
  logging: {}
extensions:
  file_storage:
    directory: /var/lib/otelcol
  health_check:
    endpoint: ${env:MY_POD_IP}:13133
  pprof:
    endpoint: localhost:1777
  zpages:
    endpoint: localhost:55679
processors:
  batch: {}
  k8sattributes:
    extract:
      metadata:
      - k8s.namespace.name
      - k8s.replicaset.name
      - k8s.statefulset.name
      - k8s.daemonset.name
      - k8s.cronjob.name
      - k8s.job.name
      - k8s.pod.name
      - k8s.node.name
    filter:
      node_from_env_var: KUBE_NODE_NAME
    passthrough: false
    pod_association:
    - sources:
      - from: resource_attribute
        name: k8s.pod.ip
    - sources:
      - from: resource_attribute
        name: k8s.pod.uid
    - sources:
      - from: connection
  memory_limiter:
    check_interval: 5s
    limit_percentage: 80
    spike_limit_percentage: 25
  resource/metadata:
    attributes:
    - action: upsert
      key: k8s.cluster.name
      value: 'kind-tail-sampling'
    - action: upsert
      key: cx.otel_integration.name
      value: coralogix-integration-helm
  resourcedetection/env:
    detectors:
    - system
    - env
    override: false
    system:
      resource_attributes:
        host.id:
          enabled: true
    timeout: 2s
  resourcedetection/region:
    detectors:
    - gcp
    - ec2
    override: true
    timeout: 2s
  transform/k8s_attributes:
    log_statements:
    - context: resource
      statements:
      - set(attributes["k8s.deployment.name"], attributes["k8s.replicaset.name"])
      - replace_pattern(attributes["k8s.deployment.name"], "^(.*)-[0-9a-zA-Z]+$",
        "$$1") where attributes["k8s.replicaset.name"] != nil
      - delete_key(attributes, "k8s.replicaset.name")
    metric_statements:
    - context: resource
      statements:
      - set(attributes["k8s.deployment.name"], attributes["k8s.replicaset.name"])
      - replace_pattern(attributes["k8s.deployment.name"], "^(.*)-[0-9a-zA-Z]+$",
        "$$1") where attributes["k8s.replicaset.name"] != nil
      - delete_key(attributes, "k8s.replicaset.name")
    trace_statements:
    - context: resource
      statements:
      - set(attributes["k8s.deployment.name"], attributes["k8s.replicaset.name"])
      - replace_pattern(attributes["k8s.deployment.name"], "^(.*)-[0-9a-zA-Z]+$",
        "$$1") where attributes["k8s.replicaset.name"] != nil
      - delete_key(attributes, "k8s.replicaset.name")
receivers:
  filelog:
    exclude: []
    include:
    - /var/log/pods/*/*/*.log
    include_file_name: false
    include_file_path: true
    operators:
    - id: get-format
      routes:
      - expr: body matches "^\\{"
        output: parser-docker
      - expr: body matches "^[^ Z]+ "
        output: parser-crio
      - expr: body matches "^[^ Z]+Z"
        output: parser-containerd
      type: router
    - id: parser-crio
      regex: ^(?P<time>[^ Z]+) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$
      timestamp:
        layout: 2006-01-02T15:04:05.999999999Z07:00
        layout_type: gotime
        parse_from: attributes.time
      type: regex_parser
    - combine_field: attributes.log
      combine_with: ""
      id: crio-recombine
      is_last_entry: attributes.logtag == 'F'
      max_log_size: 1048576
      output: extract_metadata_from_filepath
      source_identifier: attributes["log.file.path"]
      type: recombine
    - id: parser-containerd
      regex: ^(?P<time>[^ ^Z]+Z) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$
      timestamp:
        layout: '%Y-%m-%dT%H:%M:%S.%LZ'
        parse_from: attributes.time
      type: regex_parser
    - combine_field: attributes.log
      combine_with: ""
      id: containerd-recombine
      is_last_entry: attributes.logtag == 'F'
      max_log_size: 1048576
      output: extract_metadata_from_filepath
      source_identifier: attributes["log.file.path"]
      type: recombine
    - id: parser-docker
      timestamp:
        layout: '%Y-%m-%dT%H:%M:%S.%LZ'
        parse_from: attributes.time
      type: json_parser
    - combine_field: attributes.log
      combine_with: ""
      id: docker-recombine
      is_last_entry: attributes.log endsWith "\n"
      max_log_size: 1048576
      output: extract_metadata_from_filepath
      source_identifier: attributes["log.file.path"]
      type: recombine
    - id: extract_metadata_from_filepath
      parse_from: attributes["log.file.path"]
      regex: ^.*\/(?P<namespace>[^_]+)_(?P<pod_name>[^_]+)_(?P<uid>[a-f0-9\-]+)\/(?P<container_name>[^\._]+)\/(?P<restart_count>\d+)\.log$
      type: regex_parser
    - from: attributes.stream
      to: attributes["log.iostream"]
      type: move
    - from: attributes.container_name
      to: resource["k8s.container.name"]
      type: move
    - from: attributes.namespace
      to: resource["k8s.namespace.name"]
      type: move
    - from: attributes.pod_name
      to: resource["k8s.pod.name"]
      type: move
    - from: attributes.restart_count
      to: resource["k8s.container.restart_count"]
      type: move
    - from: attributes.uid
      to: resource["k8s.pod.uid"]
      type: move
    - from: attributes.log
      to: body
      type: move
    - drop_ratio: 1
      expr: (attributes["log.file.path"] matches "/var/log/pods/otel_coralogix-opentelemetry.*_.*/opentelemetry-agent/.*.log")
        and ((body contains "logRecord") or (body contains "ResourceLog"))
      type: filter
    retry_on_failure:
      enabled: true
    start_at: beginning
    storage: file_storage
  hostmetrics:
    collection_interval: '30s'
    root_path: /hostfs
    scrapers:
      cpu:
        metrics:
          system.cpu.utilization:
            enabled: true
      disk: null
      filesystem:
        exclude_fs_types:
          fs_types:
          - autofs
          - binfmt_misc
          - bpf
          - cgroup2
          - configfs
          - debugfs
          - devpts
          - devtmpfs
          - fusectl
          - hugetlbfs
          - iso9660
          - mqueue
          - nsfs
          - overlay
          - proc
          - procfs
          - pstore
          - rpc_pipefs
          - securityfs
          - selinuxfs
          - squashfs
          - sysfs
          - tracefs
          match_type: strict
        exclude_mount_points:
          match_type: regexp
          mount_points:
          - /dev/*
          - /proc/*
          - /sys/*
          - /run/k3s/containerd/*
          - /run/containerd/runc/*
          - /var/lib/docker/*
          - /var/lib/kubelet/*
          - /snap/*
      load: null
      memory:
        metrics:
          system.memory.utilization:
            enabled: true
      network: null
  jaeger:
    protocols:
      grpc:
        endpoint: ${MY_POD_IP}:14250
      thrift_binary:
        endpoint: ${MY_POD_IP}:6832
      thrift_compact:
        endpoint: ${MY_POD_IP}:6831
      thrift_http:
        endpoint: ${MY_POD_IP}:14268
  kubeletstats:
    auth_type: serviceAccount
    collection_interval: '30s'
    endpoint: ${env:K8S_NODE_IP}:10250
    insecure_skip_verify: true
  otlp:
    protocols:
      grpc:
        endpoint: ${MY_POD_IP}:4317
      http:
        endpoint: ${MY_POD_IP}:4318
  prometheus:
    config:
      scrape_configs:
      - job_name: opentelemetry-collector
        scrape_interval: 30s
        static_configs:
        - targets:
          - ${MY_POD_IP}:8888
  statsd:
    endpoint: ${MY_POD_IP}:8125
  zipkin:
    endpoint: ${MY_POD_IP}:9411
service:
  extensions:
  - zpages
  - pprof
  - health_check
  - file_storage
  pipelines:
    logs:
      exporters:
      - coralogix
      processors:
      - resource/metadata
      - k8sattributes
      - resourcedetection/env
      - resourcedetection/region
      - batch
      - transform/k8s_attributes
      receivers:
      - otlp
      - filelog
    metrics:
      exporters:
      - coralogix
      processors:
      - resource/metadata
      - k8sattributes
      - resourcedetection/env
      - resourcedetection/region
      - memory_limiter
      - batch
      - transform/k8s_attributes
      receivers:
      - otlp
      - prometheus
      - hostmetrics
      - statsd
      - kubeletstats
    traces:
      exporters:
      - loadbalancing
      processors:
      - resource/metadata
      - k8sattributes
      - resourcedetection/env
      - resourcedetection/region
      - memory_limiter
      - batch
      - transform/k8s_attributes
      receivers:
      - otlp
      - zipkin
      - jaeger
  telemetry:
    logs:
      encoding: json
      level: 'debug'
    metrics:
      address: ${MY_POD_IP}:8888
    resource:
    - service.instance.id: null
    - service.name: null

Log output

Error: invalid configuration: exporters::loadbalancing: requires a non-empty "endpoint"
2024/02/27 04:43:48 collector server run finished with error: invalid configuration: exporters::loadbalancing: requires a non-empty "endpoint"

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/loadbalancing: @jpkrohling

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[andrzej-stencel]

Duplicate of #31371, which has been closed even though the issue still exists in v0.95.0 and there's currently no newer version available.

To work around this, add endpoint: placeholder to your loadbalancing exporter configuration like this:

exporters:
  loadbalancing:
    protocol:
      otlp:
        endpoint: placeholder

[atmask]

For background this stemmed from the following issues and changes in otel-collector v0.95.0
open-telemetry/opentelemetry-collector#9523

This should be resolved in the otel-collector-contrib once the following has been released:
#31371
#31381

For now the endpoint: placeholder will be a required workaround.

[ryantanjunming]

Thanks for the info