sentry-ruby does not send profiling data because payload is oversized

[arusa]

Issue Description

I enabled profiling, but most of the time don't get any profiling information on interesting requests, because sentry-ruby throws away the Envelope with the message:

sentry: Envelope item [profile] is still oversized after size reduction: {event_id: 34, platform: 6, version: 3, profile: 261786, environment: 12, release: 4, timestamp: 27, device: 25, os: 72, runtime: 90, transaction: 153}

The serialize function tries to remove breadcrumbs and stacktrace to reduce the size below MAX_SERIALIZED_PAYLOAD_SIZE, but it seems to me, that MAX_SERIALIZED_PAYLOAD_SIZE (200kb) is too small for profiling data.

sentry-ruby/sentry-ruby/lib/sentry/envelope.rb

Line 25 in 7764384

def serialize

Reproduction Steps

Try to find profile data for requests with too much profile information. In my case I would need profiling in formation in Sentry for Proposal#index of the open source application https://github.com/decidim/decidim

Look in rails application logs for the message "Envelope item [profile] is still oversized after size reduction", probably it happens in a lot of rails applications.

Expected Behavior

I would expect sentry-rails to send profile information for all requests (according to config.profiles_sample_rate) to sentry, and to not throw away valid and important profiling information.

Actual Behavior

sentry-ruby throws away profiling information and I can't find any reason in Sentry why still no profiling data is arriving. I can only find information in the application logs, but also no error or warning.

Ruby Version

3.0.6

SDK Version

5.10.0

Integration and Its Version

Rails 6.1.7.4

Sentry Config

Sentry.init do |config|
config.breadcrumbs_logger = [:active_support_logger, :http_logger]
config.traces_sample_rate = 1.0
config.profiles_sample_rate = 1.0
end

[antonpirker]

Hey @arusa !

Thanks for bringing this up. @sl0thentr0py is on a conference right now, so he can not reply.
But I guess it would be save to increase the max payload size, because the ingestion part of Sentry has a higher limit.
@sl0thentr0py will address this when he is back from the conference.

[sl0thentr0py]

thx for reporting @arusa, will try to fix this or make the reporting better next week.

[arusa]

Hey, any updates on this topic? Does it make sense to allow a higher payload size than 200kb?

sentry-ruby/sentry-ruby/lib/sentry/envelope.rb

Line 8 in 97da98c

MAX_SERIALIZED_PAYLOAD_SIZE = 1024 * 200

[sl0thentr0py]

#2108 should help with this so closing for now

[twolfvb]

Although this was closed, my solution to a huge payload where the POST data had some binary encoded files was to butcher the sentry scope and remove the files.

    rescue_from 'my_custom_error' do |exception|
      Sentry.with_scope do |scope|
        scope.set_extra(:params, '')
        ...
        scope.set_rack_env(scope.rack_env
                                 .except('RAW_POST_DATA')
                                 .except('action_dispatch.request.request_parameters')
                                 .except('action_dispatch.request.parameters')
                                 .merge(<relevant parameters with files scrubbed>))
        Sentry.capture_exception(exception)
      end
      respond_unexpected_error(detail: exception.detail)
    end

[antonpirker]

Hey @arusa !

Thanks for bringing this up. @sl0thentr0py is on a conference right now, so he can not reply.
But I guess it would be save to increase the max payload size, because the ingestion part of Sentry has a higher limit.
@sl0thentr0py will address this when he is back from the conference.

[sl0thentr0py]

thx for reporting @arusa, will try to fix this or make the reporting better next week.

[arusa]

Hey, any updates on this topic? Does it make sense to allow a higher payload size than 200kb?

sentry-ruby/sentry-ruby/lib/sentry/envelope.rb

Line 8 in 97da98c

MAX_SERIALIZED_PAYLOAD_SIZE = 1024 * 200

[sl0thentr0py]

#2108 should help with this so closing for now

[twolfvb]

Although this was closed, my solution to a huge payload where the POST data had some binary encoded files was to butcher the sentry scope and remove the files.

    rescue_from 'my_custom_error' do |exception|
      Sentry.with_scope do |scope|
        scope.set_extra(:params, '')
        ...
        scope.set_rack_env(scope.rack_env
                                 .except('RAW_POST_DATA')
                                 .except('action_dispatch.request.request_parameters')
                                 .except('action_dispatch.request.parameters')
                                 .merge(<relevant parameters with files scrubbed>))
        Sentry.capture_exception(exception)
      end
      respond_unexpected_error(detail: exception.detail)
    end

[sl0thentr0py]

@twolfvb that works but you could also probably do that more cleanly in before_send which is a hook we expose for stuff like this.
https://docs.sentry.io/platforms/ruby/configuration/filtering/#using-