Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Verifies that client is available before sending breadcrumbs #2394

Merged

Conversation

lomefin
Copy link
Contributor

@lomefin lomefin commented Sep 2, 2024

Description

During build, we have encountered situations where Sentry wants to send
a breadcrumb but it fails to accomplish the task because of
NoMethodError: undefined method configuration' for nil:NilClass`.
After one of this error is triggered, Sentry fails to work at all,
passing from 0 to several hundred errors in our pipeline, and then back
to zero once we start rebuilding.

Our pipeline tests using parallelism and there is a ton of concurrency,
so we believe that has something to do with it.

The backtrace points out to Hub#add_breadcrumb and inside of it to
configuration. It is possible that while the new "local" hub is
available, the client is not and fails to work.

We have observed that several other method have checks
in place to verify that the client is really available but not on
add_breadcrumb so we added a verification before starting the rest
of the process.

During build, we have encountered situations where Sentry wants to send
a `breadcrumb` but it fails to accomplish the task because of
`NoMethodError: undefined method `configuration' for nil:NilClass`.
After one of this error is triggered, Sentry fails to work at all,
passing from 0 to several hundred errors in our pipeline, and then back
to zero once we start rebuilding.

Our pipeline tests using parallelism and there is a ton of concurrency,
so we believe that has something to do with it.

The backtrace points out to `Hub#add_breadcrumb` and inside of it to
`configuration`. It is possible that while the new "local" hub is
available, the client is not and fails to work.

We have observed that several other method have checks
in place to verify that the client is really available but not on
`add_breadcrumb` so we added a verification before starting the rest
of the process.
@lomefin lomefin force-pushed the adds-check-before-adding-breadcrumbs branch from df8e4e3 to 038fee8 Compare September 2, 2024 14:57
@st0012
Copy link
Collaborator

st0012 commented Sep 15, 2024

It is possible that while the new "local" hub is
available, the client is not and fails to work.

Can you elaborate more on this? I think this indicates a bigger problem either in your application's Sentry setup OR in the SDK itself.

I'd like to check a few things like:

  • Does the app itself create and manage Hubs? For example, by calling Hub#new_from_top or Hub#clone
  • Does any part of it call Hub#pop_scope?
    • @sl0thentr0py @solnic Although it may not be the cause of this issue, I feel like pop_scope is a very dangerous API to expose as it pops client as well. I think when there's only one layer left, pop_scope should retain the layer but just replace the scope?

@sl0thentr0py
Copy link
Member

sl0thentr0py commented Sep 16, 2024

@st0012 this is just part of the design problem of the older hub/scopes, we will eventually consolidate into a single OTEL-like context so there will be no more pop_scope.

Copy link

codecov bot commented Sep 16, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.67%. Comparing base (5980531) to head (038fee8).
Report is 2 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2394      +/-   ##
==========================================
- Coverage   98.68%   98.67%   -0.01%     
==========================================
  Files         210      210              
  Lines       13923    13923              
==========================================
- Hits        13740    13739       -1     
- Misses        183      184       +1     
Components Coverage Δ
sentry-ruby 99.05% <100.00%> (-0.01%) ⬇️
sentry-rails 97.34% <ø> (ø)
sentry-sidekiq 97.07% <ø> (ø)
sentry-resque 96.79% <ø> (ø)
sentry-delayed_job 98.92% <ø> (ø)
sentry-opentelemetry 100.00% <ø> (ø)
Files with missing lines Coverage Δ
sentry-ruby/lib/sentry/hub.rb 99.40% <100.00%> (+<0.01%) ⬆️

... and 3 files with indirect coverage changes

@sl0thentr0py sl0thentr0py merged commit 0c56e1c into getsentry:master Sep 16, 2024
139 of 140 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants