Closed
Description
openedon Jan 12, 2021
Hi guys, please help me to understand the issue we are having.
Lately we upgraded to rails 6 and ruby 2.7 and currently we are trying to resolve all deprecation messages and move to Zeitwerk.
The problem: On every deploy to Heroku some of our dynos crash with:
2021-01-12T09:10:15.201313+00:00 app[rpush.1]: bundler: failed to load command: rpush (/app/vendor/bundle/ruby/2.7.0/bin/rpush)
2021-01-12T09:10:15.208499+00:00 app[rpush.1]: ArgumentError: wrong number of arguments (given 2, expected 0)
2021-01-12T09:10:15.208502+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/ddtrace-0.43.0/lib/ddtrace/metrics.rb:47:in `initialize'
2021-01-12T09:10:15.208502+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/ddtrace-0.43.0/lib/ddtrace/metrics.rb:47:in `new'
2021-01-12T09:10:15.208503+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/ddtrace-0.43.0/lib/ddtrace/metrics.rb:47:in `default_statsd_client'
2021-01-12T09:10:15.208503+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/ddtrace-0.43.0/lib/ddtrace/metrics.rb:16:in `block in initialize'
2021-01-12T09:10:15.208504+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/ddtrace-0.43.0/lib/ddtrace/metrics.rb:16:in `fetch'
2021-01-12T09:10:15.208504+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/ddtrace-0.43.0/lib/ddtrace/metrics.rb:16:in `initialize'
2021-01-12T09:10:15.208505+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/ddtrace-0.43.0/lib/ddtrace/runtime/metrics.rb:15:in `initialize'
2021-01-12T09:10:15.208505+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/ddtrace-0.43.0/lib/ddtrace/configuration/components.rb:33:in `new'
2021-01-12T09:10:15.208507+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/ddtrace-0.43.0/lib/ddtrace/configuration/components.rb:33:in `build_runtime_metrics'
2021-01-12T09:10:15.208511+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/ddtrace-0.43.0/lib/ddtrace/configuration/components.rb:40:in `build_runtime_metrics_worker'
2021-01-12T09:10:15.208511+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/ddtrace-0.43.0/lib/ddtrace/configuration/components.rb:109:in `initialize'
2021-01-12T09:10:15.208512+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/ddtrace-0.43.0/lib/ddtrace/configuration.rb:90:in `new'
2021-01-12T09:10:15.208512+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/ddtrace-0.43.0/lib/ddtrace/configuration.rb:90:in `build_components'
2021-01-12T09:10:15.208513+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/ddtrace-0.43.0/lib/ddtrace/configuration.rb:27:in `configure'
2021-01-12T09:10:15.208513+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/ddtrace-0.43.0/lib/ddtrace/contrib/extensions.rb:26:in `configure'
2021-01-12T09:10:15.208514+00:00 app[rpush.1]: /app/config/initializers/datadog_tracer.rb:5:in `<main>'
2021-01-12T09:10:15.208514+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:59:in `load'
2021-01-12T09:10:15.208515+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:59:in `load'
2021-01-12T09:10:15.208515+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.4/lib/active_support/dependencies.rb:318:in `block in load'
2021-01-12T09:10:15.208516+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.4/lib/active_support/dependencies.rb:291:in `load_dependency'
2021-01-12T09:10:15.208524+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.4/lib/active_support/dependencies.rb:318:in `load'
2021-01-12T09:10:15.208525+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/railties-6.0.3.4/lib/rails/engine.rb:666:in `block in load_config_initializer'
2021-01-12T09:10:15.208525+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.4/lib/active_support/notifications.rb:182:in `instrument'
2021-01-12T09:10:15.208525+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/railties-6.0.3.4/lib/rails/engine.rb:665:in `load_config_initializer'
2021-01-12T09:10:15.208526+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/railties-6.0.3.4/lib/rails/engine.rb:625:in `block (2 levels) in <class:Engine>'
2021-01-12T09:10:15.208526+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/railties-6.0.3.4/lib/rails/engine.rb:624:in `each'
2021-01-12T09:10:15.208527+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/railties-6.0.3.4/lib/rails/engine.rb:624:in `block in <class:Engine>'
2021-01-12T09:10:15.208527+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/railties-6.0.3.4/lib/rails/initializable.rb:32:in `instance_exec'
2021-01-12T09:10:15.208528+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/railties-6.0.3.4/lib/rails/initializable.rb:32:in `run'
2021-01-12T09:10:15.208528+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/railties-6.0.3.4/lib/rails/initializable.rb:61:in `block in run_initializers'
2021-01-12T09:10:15.208529+00:00 app[rpush.1]: /app/vendor/ruby-2.7.2/lib/ruby/2.7.0/tsort.rb:228:in `block in tsort_each'
2021-01-12T09:10:15.208529+00:00 app[rpush.1]: /app/vendor/ruby-2.7.2/lib/ruby/2.7.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
2021-01-12T09:10:15.208530+00:00 app[rpush.1]: /app/vendor/ruby-2.7.2/lib/ruby/2.7.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
2021-01-12T09:10:15.208530+00:00 app[rpush.1]: /app/vendor/ruby-2.7.2/lib/ruby/2.7.0/tsort.rb:431:in `each_strongly_connected_component_from'
2021-01-12T09:10:15.208530+00:00 app[rpush.1]: /app/vendor/ruby-2.7.2/lib/ruby/2.7.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
2021-01-12T09:10:15.208531+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/railties-6.0.3.4/lib/rails/initializable.rb:50:in `each'
2021-01-12T09:10:15.208531+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/railties-6.0.3.4/lib/rails/initializable.rb:50:in `tsort_each_child'
2021-01-12T09:10:15.208532+00:00 app[rpush.1]: /app/vendor/ruby-2.7.2/lib/ruby/2.7.0/tsort.rb:415:in `call'
2021-01-12T09:10:15.208532+00:00 app[rpush.1]: /app/vendor/ruby-2.7.2/lib/ruby/2.7.0/tsort.rb:415:in `each_strongly_connected_component_from'
2021-01-12T09:10:15.208540+00:00 app[rpush.1]: /app/vendor/ruby-2.7.2/lib/ruby/2.7.0/tsort.rb:349:in `block in each_strongly_connected_component'
2021-01-12T09:10:15.208540+00:00 app[rpush.1]: /app/vendor/ruby-2.7.2/lib/ruby/2.7.0/tsort.rb:347:in `each'
2021-01-12T09:10:15.208541+00:00 app[rpush.1]: /app/vendor/ruby-2.7.2/lib/ruby/2.7.0/tsort.rb:347:in `call'
2021-01-12T09:10:15.208541+00:00 app[rpush.1]: /app/vendor/ruby-2.7.2/lib/ruby/2.7.0/tsort.rb:347:in `each_strongly_connected_component'
2021-01-12T09:10:15.208542+00:00 app[rpush.1]: /app/vendor/ruby-2.7.2/lib/ruby/2.7.0/tsort.rb:226:in `tsort_each'
2021-01-12T09:10:15.208542+00:00 app[rpush.1]: /app/vendor/ruby-2.7.2/lib/ruby/2.7.0/tsort.rb:205:in `tsort_each'
2021-01-12T09:10:15.208543+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/railties-6.0.3.4/lib/rails/initializable.rb:60:in `run_initializers'
2021-01-12T09:10:15.208543+00:00 app[rpush.1]: /app/vendor/bundle/ruby/2.7.0/gems/railties-6.0.3.4/lib/rails/application.rb:363:in `initialize!'
2021-01-12T09:10:15.208544+00:00 app[rpush.1]: config/environment.rb:3:in `<top (required)>'
It could be Sidekiq and Sneakers but not a web dyno.
It seems like here:
dd-trace-rb/lib/ddtrace/metrics.rb
Line 16 in a770176
the problem is with initialising
default_statsd_client
After adding Statsd client with default values to the initialiser the dynos stopped crashing
c.tracer.enabled = true
c.runtime_metrics.enabled = true
c.runtime_metrics.statsd = Datadog::Statsd.new('127.0.0.1', 8125)But... We also upgraded the Airbrake gem and it added a mechanism that polls for remote configurations every x seconds. Once I disabled it, the crashes also stopped (without the mentioned fix).
Can you please elaborate on that error?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment