Header Setter not being Set via orgID metadata

[JRosa316]

Component(s)

No response

What happened?

Description

Going off the comment here:
#24682 (comment)

We're trying to implement a downstream and upstream collector where the upstream collector is agnostic to the tenant in Loki. However when implementing these configs the orgID is not being set and we're getting authentication issues.

Steps to Reproduce

Create Upstream and Downstream collectors, set the HTTP headers using headersetter extension on the downstream collector, attempt to extract it in upstream and set it's own headers to the extracted value to authenticate against Loki

Expected Result

X-Scope-OrgID is set to the correct value set by the downstream collector and the log is ingested into Loki to specified tenant

Actual Result

Authentication error, No OrgID

Collector version

Helm version: 0.69.2

Environment information

Environment

Downstream: Kubernetes v1.24.15
Upstream: AWS Elastic Kuberentes Service: 1.27

OpenTelemetry Collector configuration

Downstream

opentelemetry-collector:
  mode: daemonset
  presets:
    logsCollection:
      enabled: true
      includeCollectorLogs: false
    kubernetesAttributes:
      enabled: true
    kubeletMetrics:
      enabled: true
    hostMetrics:
      enabled: true
  config:
    extensions:
      health_check: {}
      memory_ballast: {}
      headers_setter:
        headers:
          - action: upsert
            key: X-Scope-OrgID
            value: loki-poc-1
    exporters:
      debug:
        verbosity: normal
      otlphttp:
        endpoint: <poc_url>
        auth:
          authenticator: headers_setter
    service:
      extensions:
        - health_check
        - memory_ballast
        - headers_setter
      pipelines:
        logs:  
          exporters:
            - otlphttp



Upstream: 

mode: deployment
config:
  processors:
    attributes:
      actions:
      - action: upsert <-- this action of getting the header and setting it to an attribute works
        key: otel.context.id
        from_context: X-Scope-OrgID
  extensions:
    health_check: {}
    memory_ballast: {}
    headers_setter:
      headers:
        - action: upsert
          key: X-Scope-OrgID
          # value: loki-poc-1     <---this works 
          from_context: X-Scope-OrgID <--- this fails to insert
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
          include_metadata: true
        http:
          endpoint: 0.0.0.0:4318
          include_metadata: true
  exporters:
    debug:
      verbosity: normal
    loki:
      endpoint: "http://loki:xxxx@loki-gateway.loki.svc.cluster.local/loki/api/v1/push"
      auth:
        authenticator: headers_setter
  service:
    extensions:
    - health_check
    - memory_ballast
    - headers_setter
    pipelines:
      logs:
        exporters:
        - debug
        - loki
        processors:
        - attributes

Log output

2023-11-09T17:16:36.231Z	error	exporterhelper/retry_sender.go:145	Exporting failed. The error is not retryable. Dropping data.	{"kind": "exporter", "data_type": "logs", "name": "loki", "error": "Permanent error: HTTP 401 \"Unauthorized\": no org id", "dropped_items": 100}

Additional context

No response

[github-actions]

Pinging code owners for extension/headerssetter: @jpkrohling. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

Pinging code owners for exporter/loki: @gramidt @gouthamve @jpkrohling @mar4uk. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[mar4uk]

The collector version 0.69.2 is pretty old. Could you please try to use the newer version? I see from the changelog that headersetter started to support actions in 0.71.2 version

[JRosa316]

Good Morning,

You can mark this as closed, we figured out that the order of operations mattered a bit and this worked for us:

extensions:
    health_check: {}
    memory_ballast: {}
    headers_setter:
      headers:
        - action: upsert
          key: X-Scope-OrgID
          from_context: X-Scope-OrgID
  processors:
    attributes:
      actions:
      - action: upsert
        key: loki.tenant.id
        from_context: X-Scope-OrgID

This allowed us to inject the tenant id into the document and the header.

Thanks!