Otel unable to recieve data with Istio Service Mesh from instrumentated services

[blakeromano]

I am trying to get the Open Telemetry Collector to recieve traces for a Node Express application, I added debug logging on and I see the following error. This is on a kubernetes cluster with an Istio service mesh. I have attempted to use the 4317 and 4318 ports but I continually see the same errors and no data going to the otel collector.

{"stack":"Error: 14 UNAVAILABLE: No connection established\n    at Object.callErrorFromStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call.js:31:26)\n    at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client.js:189:52)\n    at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:365:141)\n    at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:328:181)\n    at /usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:187:78\n    at processTicksAndRejections (internal/process/task_queues.js:77:11)","message":"14 UNAVAILABLE: No connection established","code":"14","details":"No connection established","metadata":"[object Object]","name":"Error"}

and this in the istio proxy log

{"user_agent":null,"upstream_local_address":"10.200.153.198:41062","bytes_received":418,"upstream_transport_failure_reason":null,"route_name":null,"path":null,"istio_policy_status":null,"authority":null,"response_flags":"-","downstream_remote_address":"10.200.153.198:34254","request_id":null,"x_forwarded_for":null,"method":null,"upstream_host":"10.200.214.67:4318","downstream_local_address":"100.68.177.139:4318","requested_server_name":null,"start_time":"2022-07-01T19:24:15.055Z","protocol":null,"trace_id":null,"duration":1,"upstream_service_time":null,"response_code":0,"upstream_cluster":"outbound|4318||opentelemetry-collector.opentelemetry.svc.cluster.local","bytes_sent":103}

I have tried 4317 and 4318 ports and also added a destination rule to try and help but nothing seems to work 😭

[evan-bradley]

cc @frzifus

[frzifus]

hi @blakeromano, ive a few questions. how did you deploy your Collector? Seems to be a daemonset or deployment, right? What protocol are you using to transmit data? I assume grpc? When trying 4318 did you also switch to otel/http?

Iam not an isitio expert, but the response_code=0 makes we wonder if something is dropped.

[dmitryax]

Looks like the trace exporter cannot establish connection to the collector. make sure that collector ports are properly exposed and collector is running fine with otlp receiver enabled in the traces pipeline

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.