Event sending failed: "\xC4" from ASCII-8BIT to UTF-8

[gencer]

Describe the bug

When HTTP Header contains characters (UTF-8) like one of ş,ğ,ü,ö,ç (Especially ğ), Sentry unable to report exception to Server due to conversion error.

To Reproduce

1. Set a header for example: my_header: Tekirdağ and send request to your endpoint that produces an error or captures a message.
2. It will fail because Tekirdağ word contains non-ascii character.

Event sending failed: "\xC4" from ASCII-8BIT to UTF-8
Unreported Event: Encoding::UndefinedConversionError: "\xC4" from ASCII-8BIT to UTF-8
Event capturing failed: "\xC4" from ASCII-8BIT to UTF-8

The root cause is here:

sentry-ruby/sentry-ruby/lib/sentry/transport.rb

Line 102 in 740ed8c

#{JSON.generate(event_hash)}

# Output

{"X-Forwarded-For"=>"***",
 "Host"=>"g.dev",
 "X-Real-Ip"=>"***",
 "X-Forwarded-Proto"=>"https",
 "Connection"=>"close",
 "User-Agent"=>"insomnia/2021.4.1",
 "my_header"=>"Tekirda\xC4\x9F",      # <<-- Here :)
 "Accept"=>"*/*",
 ...

I've solved this by using a middleware that does 'Tekirdağ'.force_encoding('utf-8') and replace in @app.call(env) that worked but i cannot do this for every header or project...

Expected behavior

Error / Message should be reported.

Actual behavior

Sentry is not be able to send report to the server...

Environment

• Ruby Version: 3.0.1
• SDK Version: 4.6.5
• Integration Versions (if any):
• Rails 6.1.4.1 with Puma 5.4.0

Sentry Config

Sentry.init do |config|
  # Very basic setup given by Sentry.io
end

[st0012]

In my reproduction, force encoding doesn't completely fix your case. It'll still cause partial character in source, but hit end error during the serialization.

So eventually I decide to drop the value if it's still not a valid encoding after a force conversion, which will look like this:

Wdyt?

Update

Actually, I think scrub may be a better solution here.

[gencer]

@st0012 Sounds great! Thank you for your prompt response and PR! 🎉

[st0012]

In my reproduction, force encoding doesn't completely fix your case. It'll still cause partial character in source, but hit end error during the serialization.

So eventually I decide to drop the value if it's still not a valid encoding after a force conversion, which will look like this:

Wdyt?

Update

Actually, I think scrub may be a better solution here.

[gencer]

@st0012 Sounds great! Thank you for your prompt response and PR! 🎉