[Tracing] received corrupt message of type InvalidContentType - When Collector is not in mesh, or has ports marked inbound skip

[jseiser]

What is the issue?

Linkerd breaks traces when running against the OTLP collector, confusing the collector into thinking the traces come from the collector itself, not from the originating pod.

Example: grafana/alloy#1336 (comment)

As a work around, we wanted to just remove the collector from the mesh, that breaks linkerd-proxy being able to send traces. We then attempted to leave the collector in the mesh, but tell it to skip the relevant ports inbound, that also breaks linkerd-proxy being able to send traffic.

How can it be reproduced?

1. AWS EKS Cluster
2. Linkerd
3. Grafana Alloy
4. Configure Linkerd-jaeger to send traces to Grafana Alloy

Logs, error output, etc

This happens with Alloy removed from the mesh, or with alloy set to skip the ports in bound.

{"timestamp":"2024-12-04T16:30:47.984933Z","level":"WARN","fields":{"message":"Failed to connect","error":"endpoint 10.1.7.129:4317: received corrupt message of type InvalidContentType"},"target":"linkerd_reconnect","threadId":"ThreadId(1)"}
{"timestamp":"2024-12-04T16:30:48.202478Z","level":"WARN","fields":{"message":"Failed to connect","error":"endpoint 10.1.16.177:4317: received corrupt message of type InvalidContentType"},"target":"linkerd_reconnect","threadId":"ThreadId(1)"}
{"timestamp":"2024-12-04T16:30:48.229660Z","level":"WARN","fields":{"message":"Failed to connect","error":"endpoint 10.1.16.177:4318: received corrupt message of type InvalidContentType"},"target":"linkerd_reconnect","threadId":"ThreadId(1)"}

output of linkerd check -o short

kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ control plane pods are ready
√ cluster networks contains all pods
√ cluster networks contains all services

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ proxy-init container runs as root user if docker container runtime is used

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor

linkerd-webhooks-and-apisvc-tls
-------------------------------
√ proxy-injector webhook has valid cert
√ proxy-injector cert is valid for at least 60 days
√ sp-validator webhook has valid cert
√ sp-validator cert is valid for at least 60 days
√ policy-validator webhook has valid cert
√ policy-validator cert is valid for at least 60 days

linkerd-version
---------------
√ can determine the latest version
‼ cli is up-to-date
    is running version 24.11.4 but the latest edge version is 24.11.8
    see https://linkerd.io/2/checks/#l5d-version-cli for hints

control-plane-version
---------------------
√ can retrieve the control plane version
‼ control plane is up-to-date
    is running version 24.11.4 but the latest edge version is 24.11.8
    see https://linkerd.io/2/checks/#l5d-version-control for hints
√ control plane and cli versions match

linkerd-control-plane-proxy
---------------------------
√ control plane proxies are healthy
‼ control plane proxies are up-to-date
    some proxies are not running the current version:
	* linkerd-destination-c9b6b96c-lwwv4 (edge-24.11.4)
	* linkerd-destination-c9b6b96c-m44vg (edge-24.11.4)
	* linkerd-destination-c9b6b96c-xm56z (edge-24.11.4)
	* linkerd-identity-7d9c687659-mftdm (edge-24.11.4)
	* linkerd-identity-7d9c687659-qcq5l (edge-24.11.4)
	* linkerd-identity-7d9c687659-sj5zp (edge-24.11.4)
	* linkerd-proxy-injector-7b5f5c7d66-c2q6z (edge-24.11.4)
	* linkerd-proxy-injector-7b5f5c7d66-cltg2 (edge-24.11.4)
	* linkerd-proxy-injector-7b5f5c7d66-nn9t7 (edge-24.11.4)
    see https://linkerd.io/2/checks/#l5d-cp-proxy-version for hints
√ control plane proxies and cli versions match

linkerd-ha-checks
-----------------
√ multiple replicas of control plane pods

linkerd-extension-checks
------------------------
√ namespace configuration for extensions

linkerd-jaeger
--------------
√ linkerd-jaeger extension Namespace exists
√ jaeger extension pods are injected
√ jaeger injector pods are running
√ jaeger extension proxies are healthy
‼ jaeger extension proxies are up-to-date
    some proxies are not running the current version:
	* jaeger-injector-bdf688f96-ddcxf (edge-24.11.4)
	* jaeger-injector-bdf688f96-xf8hn (edge-24.11.4)
    see https://linkerd.io/2/checks/#l5d-jaeger-proxy-cp-version for hints
√ jaeger extension proxies and cli versions match

linkerd-viz
-----------
√ linkerd-viz Namespace exists
√ can initialize the client
√ linkerd-viz ClusterRoles exist
√ linkerd-viz ClusterRoleBindings exist
√ tap API server has valid cert
√ tap API server cert is valid for at least 60 days
√ tap API service is running
√ linkerd-viz pods are injected
√ viz extension pods are running
√ viz extension proxies are healthy
‼ viz extension proxies are up-to-date
    some proxies are not running the current version:
	* metrics-api-cfbfbfcbc-jr4fn (edge-24.11.4)
	* metrics-api-cfbfbfcbc-qbzgz (edge-24.11.4)
	* prometheus-5464dc854b-wl6w6 (edge-24.11.4)
	* tap-858b7b86d4-gl9mn (edge-24.11.4)
	* tap-858b7b86d4-vb8wz (edge-24.11.4)
	* tap-injector-d49bf4cfb-djfcv (edge-24.11.4)
	* tap-injector-d49bf4cfb-fv2zp (edge-24.11.4)
	* web-5dd7bf96db-dj55j (edge-24.11.4)
	* web-5dd7bf96db-qcg8m (edge-24.11.4)
    see https://linkerd.io/2/checks/#l5d-viz-proxy-cp-version for hints
√ viz extension proxies and cli versions match
√ prometheus is installed and configured correctly
√ viz extension self-check

Status check results are √

Environment

Kubernetes version - 1.30
Cluster Environment - EKS
Host OS - Bottle Rocket

Possible solution

No response

Additional context

Im not sure why port 4318, ever shows up in the logs. Its configured for 4317

LINKERD2_PROXY_TRACE_COLLECTOR_SVC_ADDR:                   alloy-cluster.grafana-alloy.svc.cluster.local:4317
LINKERD2_PROXY_TRACE_PROTOCOL:                             opentelemetry
LINKERD2_PROXY_TRACE_SERVICE_NAME:                         linkerd-proxy
LINKERD2_PROXY_TRACE_COLLECTOR_SVC_NAME:                   alloy.grafana-alloy.serviceaccount.identity.linkerd.cluster.local

The pod it's failing to connect to, is an alloy pod.

❯ kubectl get pods -A -o wide | rg 10.1.16.177                                                
grafana-alloy               alloy-68fbb65465-wc6v5                                      3/3     Running     0             20h    10.1.16.177   i-03b207599fb2636db.us-gov-west-1.compute.internal   <none>           <none>

The linkerd proxy, on the alloy pod logs this

{"timestamp":"2024-12-04T17:01:18.536899Z","level":"WARN","fields":{"message":"Failed to connect","error":"Connection refused (os error 111)"},"target":"linkerd_reconnect","threadId":"ThreadId(1)"}                                         
{"timestamp":"2024-12-04T17:01:18.553171Z","level":"WARN","fields":{"message":"Failed to connect","error":"Connection refused (os error 111)"},"target":"linkerd_reconnect","threadId":"ThreadId(1)"}                                         
{"timestamp":"2024-12-04T17:01:19.037516Z","level":"WARN","fields":{"message":"Failed to connect","error":"Connection refused (os error 111)"},"target":"linkerd_reconnect","threadId":"ThreadId(1)"}                                         
{"timestamp":"2024-12-04T17:01:19.054832Z","level":"WARN","fields":{"message":"Failed to connect","error":"Connection refused (os error 111)"},"target":"linkerd_reconnect","threadId":"ThreadId(1)"} 

There are no actual errors on the actual alloy pod itself.

Everything that is not Linkerd-proxy, is still able to send traces without a problem, including pods that are fully in the mesh themselves.

Would you like to work on fixing this bug?

None

[jseiser]

Is there any additional information which would make this easier to troubleshoot?

[wc-s]

We ran into the same error message. On a hunch, I thought it might be because the collector was not meshed. After meshing our collector, this error went away.

[jseiser]

When we mesh the collector, linkerd traces show as being generated BY the collector that processes the trace, so it creates a brand new issue.

…

On Thu, Jan 9, 2025, 7:42 AM Weichung Shaw ***@***.***> wrote: We ran into the same error message. On a hunch, I thought it might be because the collector was not meshed. After meshing our collector, this error went away. — Reply to this email directly, view it on GitHub <#13427 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABFBNZZAIWLDC5RVU5N6H3T2JZVEBAVCNFSM6AAAAABTAUB3GCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKOBQGA3DEMRWGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[wc-s]

Huh, what do you mean it's shown as being emitted by the collector?

For what it's worth, we now have it working correctly in our cluster, properly correlated with all our other spans.

[kflynn]

@wc-s I would love to hear how exactly you've set things up to have everything working – I'm not an Alloy expert and would like to learn more. 🙂

[jseiser]

I mean all linkerd traces contain metadata of the grafana alloy pod which received the trace, which is why we attempted to unmesh the collector

…

On Thu, Jan 9, 2025, 12:07 PM Weichung Shaw ***@***.***> wrote: Huh, what do you mean it's shown as being emitted by the collector? For what it's worth, we now have it working correctly in our cluster, properly correlated with all our other spans. — Reply to this email directly, view it on GitHub <#13427 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABFBNZ3FAREEPDP6DLJMIO32J2UEPAVCNFSM6AAAAABTAUB3GCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKOBQHAZTMMRSGI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[jseiser]

I'm boarding a flight, but there are open issues on grafana alloy, linkerd and the hotel collector all referencing similar issues. Trace ports are marked opaque but it doesn't change the result

…

On Thu, Jan 9, 2025, 12:08 PM Flynn ***@***.***> wrote: @wc-s <https://github.com/wc-s> I would love to hear how exactly you've set things up to have everything working – I'm not an Alloy expert and would like to learn more. 🙂 — Reply to this email directly, view it on GitHub <#13427 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABFBNZ64HNQS4Y3XD63DJO32J2UKNAVCNFSM6AAAAABTAUB3GCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKOBQHAZTSMJUGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[wc-s]

OK, I gotta apologize, I didn't read the original post carefully enough and didn't realize you're using Grafana Alloy. We're using the upstream opentelemetry-collector directly. What's more, we don't use the k8sattributes processor and instead populate the k8s metadata some other way, so our experience is largely irrelevant to you 😅

However, examining your Alloy config, have you tried applying linkerd people's recommended collector config?

It's here: https://github.com/linkerd/linkerd2/blob/main/jaeger/charts/linkerd-jaeger/values.yaml#L120

Grafana Alloy I think is just a thin wrapper around opentelemetry-collector, so the config format is pretty much the same.

I suspect that the linkerd people also realized that here we cannot rely on Pod IP to associate, so there they used the host.name attribute (which is populated by linkerd-proxy and not retrieved by collector). This attribute is basically the pod name.

Firstly they rename the host.name attribute to k8s.pod.name here: https://github.com/linkerd/linkerd2/blob/main/jaeger/charts/linkerd-jaeger/values.yaml#L112

Then they tell collector to use that attribute to find the right pod: https://github.com/linkerd/linkerd2/blob/main/jaeger/charts/linkerd-jaeger/values.yaml#L124

And they run the resources processor before the k8sattributes processor: https://github.com/linkerd/linkerd2/blob/main/jaeger/charts/linkerd-jaeger/values.yaml#L184

Hopefully that makes the Collector find the correct Pod.

If the above doesn't work though, for what it's worth, you can probably get like half of the data you want without even using the k8sattribute processor.

linkerd-proxy populates the following attributes:

linkerd.io/proxy-deployment
linkerd.io/workload-ns
host.name

Which gets you the workload name, the workload namespace, and pod name. The first two though, are overriden by your current k8sattributes config:

        extract {
          annotation {
            from      = "pod"
            key_regex = "(.*)/(.*)"
            tag_name  = "$1.$2"
          }
          label {
            from      = "pod"
            key_regex = "(.*)/(.*)"
            tag_name  = "$1.$2"
          }

And by removing that, you'd be able to see those two. But yea this still doesn't get you the node.name, pod.uid, pod.start_time, and the other labels and annotations.

[jseiser]

@wc-s

Grafana Alloy I think is just a thin wrapper around opentelemetry-collector, so the config format is pretty much the same.

It is, its just the config syntax is different as you see.

I know we had alot of this in place already, but Ill go back and make sure the above is in there.

Thanks,

[jseiser]

I will have this all re-deployed tomorrow morning. I have the new config loaded, just need to re-add linkerd to send traces. The config, which should pretty closely mirror the default

logging {
  level  = "info"
  format = "json"
}

otelcol.receiver.otlp "default" {
  debug_metrics {
    disable_high_cardinality_metrics = true
  }
  grpc {
    endpoint = "0.0.0.0:4317"
    include_metadata = true
  }

  http {
    endpoint = "0.0.0.0:4318"
    include_metadata = true
  }
  output {
    traces = [otelcol.processor.attributes.default.input]
  }
}

otelcol.processor.attributes "default" {
  action {
      key = "k8s.pod.name"
      from_attribute = "host.name"
      action = "insert"
  }
  action {
      key = "k8s.namespace.name"
      from_attribute = "linkerd.io/workload-ns"
      action = "insert"
  }
  output {
    traces = [otelcol.processor.memory_limiter.default.input]
  }
}

otelcol.processor.memory_limiter "default" {
  check_interval = "5s"

  limit = "512MiB"

  output {
      traces  = [otelcol.processor.k8sattributes.default.input]
  }
}

otelcol.processor.k8sattributes "default" {
  extract {
    metadata = [
      "k8s.pod.name",
      "k8s.pod.uid",
      "k8s.deployment.name",
      "k8s.node.name",
      "k8s.namespace.name",
      "k8s.pod.start_time",
      "k8s.replicaset.name",
      "k8s.replicaset.uid",
      "k8s.daemonset.name",
      "k8s.daemonset.uid",
      "k8s.job.name",
      "k8s.job.uid",
      "k8s.cronjob.name",
      "k8s.statefulset.name",
      "k8s.statefulset.uid",
      "k8s.container.name",
      "container.image.name",
      "container.image.tag",
      "container.id",
    ]
    label {
      from = "pod"
      key_regex = "kubernetes.io/(.*)"
      tag_name  = "$1"
    }
  }

  pod_association {
    source {
      from = "resource_attribute"
      name = "k8s.pod.name"
    }
    source {
      from = "resource_attribute"
      name = "k8s.namespace.name"
    }
  }

  output {
    traces = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch "default" {
  send_batch_size = 16384
  send_batch_max_size = 0
  timeout = "2s"

  output {
      traces  = [otelcol.exporter.otlp.to_tempo.input]
  }
}

otelcol.exporter.otlp "to_tempo" {
  client {
    endpoint = "tempo-distributed-distributor.tempo.svc.cluster.local:4317"
    tls {
        insecure             = true
        insecure_skip_verify = true
    }
  }
}

[jseiser]

Above still doesnt work, since the attributes do not appear to rewrite like you would expect.

Linkerd still shows unassociated, and the traces still container

host.name	
linkerd.io/control-plane-ns	
linkerd.io/proxy-deployment	
linkerd.io/workload-ns	

but does not container

k8s.pod.name
k8s.namespace.name

[jseiser]

I think the attributes transform fails, because linkerd is setting resource attributes, and its wanting to modify span attributes.

[jseiser]

I can confirm you can get the changes in place using the transform processor, but the end result is the same. Linkerd traces are not associated with their corresponding pods

      otelcol.processor.transform "default" {
        error_mode = "ignore"

        trace_statements {
          context = "resource"
          statements = [
            `replace_all_patterns(attributes, "key", "linkerd.io/workload-ns", "k8s.namespace.name")`,
            `replace_all_patterns(attributes, "key", "host.name", "k8s.pod.name")`,
          ]
        }

        output {
          traces  = [otelcol.processor.k8sattributes.default.input]
        }
      }