rpc encoding error

[schmurfy]

I am getting weird log entries on one of our services, I think the message is printed by uptrace-go and although this is not an error in the library itself I am trying to find it to find what exactly is the problematic content xD

2021/10/21 10:43:46 rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8

Do you have any idea where this might originate from ?
I am sure the message is not from our service itself since we use zap to output json formatted log, no log.Printxx is used anywhere.

[schmurfy]

I found another message:

2021/10/21 11:04:34 traces exporter is disconnected from the server otlp.uptrace.dev:4317: rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8

No idea where it is logged from either, can't find "traces exporter is disconnected" anywhere in the codebase for uptrace-go

[vmihailenco]

The error comes from grpc-go, because there is some invalid UTF-8 in the payload. The error is generated by protobuf. AFAIK the data is not even sent to Uptrace.

It looks like this issue can be triggered by using invalid UTF-8 in strings or converting []byte to string via attribute.String("foo", string(bytes)). I guess some instrumentations may do that, because OpenTelemetry does not directly support []byte and using string instead is a natural choice.

I am not sure what do, but you could try to silence those logs with SetErrorHandler

import "go.opentelemetry.io/otel"

otel.SetErrorHandler(otel.ErrorHandlerFunc(func(e error) {}))

[vmihailenco]

PS I guess I will create an issue in opentelemetry-go. And thanks for the report :)

[schmurfy]

I found the source of my error, as I said I knew the error was not caused by uptrace I just had a lot of trouble to find where was the origin xD

In that cas I was doing exactly what you mention:

rawData := []byte{...}

span.AddAttributes(
  attribute.String("data", string(rawData))
)

[vmihailenco]

👍 Some options:

• use strings.ToValidUTF8(s, "") if that is really a string with some invalid UTF-8 chars
• hex encode the data
• use json.Marshal which will hex-encode it for you

[schmurfy]

I have removed logging/tracing this field altogether, we use a code generator to generate those that's why it needed some time to track where this particular error was.
I have another one but I think I know the culprit ^^