Open
Description
openedon May 15, 2023
Current behaviour
ddtrace occasionally causes applications to fail on boot with error:
can't add a new key into hash during iteration
/usr/local/bundle/gems/ddtrace-0.54.2/lib/ddtrace/contrib/extensions.rb:131:in `instrument'
/usr/local/bundle/gems/ddtrace-0.54.2/lib/ddtrace/contrib/rails/framework.rb:116:in `activate_action_pack!'
/usr/local/bundle/gems/ddtrace-0.54.2/lib/ddtrace/contrib/rails/framework.rb:51:in `block in setup'
/usr/local/bundle/gems/ddtrace-0.54.2/lib/ddtrace/configuration.rb:48:in `configure'
/usr/local/bundle/gems/ddtrace-0.54.2/lib/ddtrace/contrib/extensions.rb:40:in `configure'
/usr/local/bundle/gems/ddtrace-0.54.2/lib/ddtrace/contrib/rails/framework.rb:39:in `setup'
/usr/local/bundle/gems/ddtrace-0.54.2/lib/ddtrace/contrib/rails/patcher.rb:114:in `setup_tracer'
/usr/local/bundle/gems/ddtrace-0.54.2/lib/ddtrace/contrib/rails/patcher.rb:108:in `block in after_intialize'
/usr/local/bundle/gems/ddtrace-0.54.2/lib/ddtrace/utils/only_once.rb:26:in `block in run'
/usr/local/bundle/gems/ddtrace-0.54.2/lib/ddtrace/utils/only_once.rb:21:in `synchronize'
/usr/local/bundle/gems/ddtrace-0.54.2/lib/ddtrace/utils/only_once.rb:21:in `run'
/usr/local/bundle/gems/ddtrace-0.54.2/lib/ddtrace/contrib/rails/patcher.rb:105:in `after_intialize'
/usr/local/bundle/gems/ddtrace-0.54.2/lib/ddtrace/contrib/rails/patcher.rb:100:in `block in patch_after_intialize'
/usr/local/bundle/gems/activesupport-6.1.6/lib/active_support/lazy_load_hooks.rb:73:in `instance_eval'
/usr/local/bundle/gems/activesupport-6.1.6/lib/active_support/lazy_load_hooks.rb:73:in `block in execute_hook'
/usr/local/bundle/gems/activesupport-6.1.6/lib/active_support/lazy_load_hooks.rb:61:in `with_execution_control'
/usr/local/bundle/gems/activesupport-6.1.6/lib/active_support/lazy_load_hooks.rb:66:in `execute_hook'
/usr/local/bundle/gems/activesupport-6.1.6/lib/active_support/lazy_load_hooks.rb:52:in `block in run_load_hooks'
/usr/local/bundle/gems/activesupport-6.1.6/lib/active_support/lazy_load_hooks.rb:51:in `each'
/usr/local/bundle/gems/activesupport-6.1.6/lib/active_support/lazy_load_hooks.rb:51:in `run_load_hooks'
/usr/local/bundle/gems/railties-6.1.6/lib/rails/application/finisher.rb:140:in `block in <module:Finisher>'
/usr/local/bundle/gems/railties-6.1.6/lib/rails/initializable.rb:32:in `instance_exec'
/usr/local/bundle/gems/railties-6.1.6/lib/rails/initializable.rb:32:in `run'
/usr/local/bundle/gems/railties-6.1.6/lib/rails/initializable.rb:61:in `block in run_initializers'
This appears to be due to a concurrency issue with the instrumented_integrations hash. Multiple threads are accessing the hash without thread safe operations. Datadog::Diagnostics::EnvironmentLogger.log! iterates through instrumented_integrations in a separate worker thread. At the same time, Datadog::Contrib::Extensions#instrument is adding keys to the same instrumented_integrations hash object. Current workaround is to disable ddtrace startup logs.
Expected behaviour
ddtrace should not cause application to crash
Steps to reproduce
Difficult to reproduce since it's timing related but it may be possible by artificially triggering a trace write during startup.
How does ddtrace help you?
Environment
- ddtrace version: 0.54.2
- Configuration block (
Datadog.configure ...):
config/initializers/datadog.rb
Datadog.configure do |c|
c.tracer(hostname: ENV["DD_AGENT_HOST"] || "localhost",
port: ENV["DD_TRACE_AGENT_PORT"] || 8126,
debug: ENV["DD_TRACE_DEBUG"] == "true")
c.tracer.sampler = Datadog::PrioritySampler.new(
post_sampler: Datadog::Sampling::RuleSampler.new(
[
Datadog::Sampling::SimpleRule.new(service: c.service, sample_rate: 1.0000),
]
)
)
c.tracer.partial_flush.enabled = true
c.use(:aws, service_name: c.service)
c.use(:http, split_by_domain: true)
c.use(:httprb, split_by_domain: true)
c.use(:grpc, service_name: c.service)
c.use(:kafka, service_name: c.service)
c.use(:rack, service_name: c.service, web_service_name: "#{c.service}-nginx", request_queuing: true)
c.use(:redis, service_name: c.service)
c.use(:rails, service_name: c.service)
c.use(:rest_client, service_name: c.service)
c.use(:sidekiq, service_name: c.service, client_service_name: c.service)
c.use(:sneakers, service_name: c.service)
endspec/spec_helper.rb
Datadog.configure do |c|
c.tracer.enabled = true
c.ci_mode.enabled = true
c.use :rspec, operation_name: "ruby.rspec.integration"
end- Ruby version: 3.0.5
- Operating system: debian:bullseye-slim
- Relevant library versions: Rails 6.1.6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
