[processor/k8sattributes] Processor does not add resource attributes from labels when the collector is restarted

[go-follow]

Component(s)

processor/k8sattributes

What happened?

Description

Restarting the collector causes k8sattributes processor to not add resource attributes from pod labels for the first batch of logs. Subsequent logs already have all resource attributes. Adding a filter by node partially helps to fix the situation, but for large Kubernetes clusters the problem still reproduces:

filter:
 node_from_env_var: NODE_NAME

To check, I tried adding time.Sleep(5 * time.Second) in the Start(_ context.Context, host component.Host) method. In this case, the problem does not reproduce. But I think that time.Sleep() is not the most reliable and correct solution to this problem.

My configuration for k8sattributes processor:

processors: 
  k8sattributes:
    filter:
      node_from_env_var: NODE_NAME
    extract:
      metadata:
        - k8s.node.name
      labels:
        - from: pod
          key: app
          tag_name: app
        - from: pod
          key: product-code
          tag_name: product.code
        - from: pod
          key: resource-id
          tag_name: resource.id
        - from: pod
          key: resource-name
          tag_name: resource.name
    passthrough: false
    pod_association:
      - sources:
          - from: resource_attribute
            name: k8s.pod.uid

Steps to Reproduce

• Any restart of the collector. For example, an update
• The first batch of logs does not contain resource attributes
• Also, at startup there is a non-zero metric otelcol_otelsvc_k8s_ip_lookup_miss
• Subsequent logs already have all resource attributes

Expected Result

• All logs have resource attributes derived from pod labels
• Metric otelcol_otelsvc_k8s_ip_lookup_miss missing

Actual Result

• The first logs after restart have no resource attributes
• There is a metric otelcol_otelsvc_k8s_ip_lookup_miss which has values

Collector version

v0.112.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:
  filelog:
    exclude:
      - /var/log/pods/*otel-logs-*/*/*.log
    include:
      - /var/log/pods/*/*/*.log
    include_file_name: false
    include_file_path: true
    start_at: end
    resource:
      k8s.cluster.name: kube-develop-oaba
    storage: file_storage/offset
    operators:
      - id: get-format
        routes:
          - expr: body matches "^\\{"
            output: parser-docker
          - expr: body matches "^[^ Z]+ "
            output: parser-crio
          - expr: body matches "^[^ Z]+Z"
            output: parser-containerd
        type: router
      - id: parser-crio
        type: regex_parser
        regex: ^(?P<time>[^ Z]+) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$
        timestamp:
          layout: 2006-01-02T15:04:05.999999999Z07:00
          layout_type: gotime
          parse_from: attributes.time
      - combine_field: attributes.log
        combine_with: ""
        type: recombine
        id: crio-recombine
        is_last_entry: attributes.logtag == 'F'
        max_log_size: 50000
        output: extract_metadata_from_filepath
        source_identifier: attributes["log.file.path"]
      - id: parser-containerd
        type: regex_parser
        regex: ^(?P<time>[^ ^Z]+Z) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$
        timestamp:
          layout: '%Y-%m-%dT%H:%M:%S.%LZ'
          parse_from: attributes.time
      - combine_field: attributes.log
        type: recombine
        combine_with: ""
        id: containerd-recombine
        is_last_entry: attributes.logtag == 'F'
        max_log_size: 50000
        output: extract_metadata_from_filepath
        source_identifier: attributes["log.file.path"]
      - id: parser-docker
        type: json_parser
        output: extract_metadata_from_filepath
        timestamp:
          layout: '%Y-%m-%dT%H:%M:%S.%LZ'
          parse_from: attributes.time
      - id: extract_metadata_from_filepath
        parse_from: attributes["log.file.path"]
        regex: ^.*\/(?P<namespace>[^_]+)_(?P<pod_name>[^_]+)_(?P<uid>[a-f0-9\-]+)\/(?P<container_name>[^\._]+)\/(?P<restart_count>\d+)\.log$
        type: regex_parser
      - from: attributes.stream
        to: attributes["log.iostream"]
        type: move
      - from: attributes.container_name
        to: resource["k8s.container.name"]
        type: move
      - from: attributes.namespace
        to: resource["k8s.namespace.name"]
        type: move
      - from: attributes.pod_name
        to: resource["k8s.pod.name"]
        type: move
      - from: attributes.restart_count
        to: resource["k8s.container.restart_count"]
        type: move
      - from: attributes.uid
        to: resource["k8s.pod.uid"]
        type: move
      - from: attributes.log
        to: body
        type: move
processors: 
  k8sattributes:
    filter:
      node_from_env_var: NODE_NAME
    extract:
      metadata:
        - k8s.node.name
      labels:
        - from: pod
          key: app
          tag_name: app
        - from: pod
          key: product-code
          tag_name: product.code
        - from: pod
          key: resource-id
          tag_name: resource.id
        - from: pod
          key: resource-name
          tag_name: resource.name
    passthrough: false
    pod_association:
      - sources:
          - from: resource_attribute
            name: k8s.pod.uid
exporters:
  debug:
    verbosity: detailed

extensions:
  file_storage/offset:
    directory: /var/lib/otelcol/file_storage/offset
    timeout: 1s

service:
  telemetry:
    logs:
      encoding: json
      level: info
  extensions:
    - file_storage/offset
  pipelines:
    logs:
      exporters:
      - debug
      processors:
      - k8sattributes
      receivers:
      - filelog

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• processor/k8sattributes: @dmitryax @fatsheep9146 @TylerHelmuth

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[TylerHelmuth]

I believe this is a duplicate of #32556. Please ping me if you disagree.

[go-follow]

You're right. It looks like it really is a double. I'm looking forward to it being merged #32622