Skip to content

ArgumentError thrown from auto-instrumentation code when running OpenTelemetry in parallel #3575

New issue
New issue
Open
Open
ArgumentError thrown from auto-instrumentation code when running OpenTelemetry in parallel#3575

Description

@anandhasudhan

anandhasudhan
openedon Apr 5, 2024

Current behaviour
When running Datadog and OpenTelemetry in parallel, the auto-instrumentation code from datadog throws an ArgumentError since it ends up calling the opentelemetry library code. It appears that datadog instrumentation is clashing with the opentelemetry instrumentation. I briefly looked at the ddtrace code and it has this method annotate_span_with_response! that takes 3 arguments and opentelemetry code has the same method annotate_span_with_response! that takes 2 arguments.

The ArgumentError we received was for this particular method where opentelemetry expected 2 arguments but received 3 (from the datadog instrumentation). Although, they have their own separate name spaces, I'm not sure why datadog called the opentelemetry library code.

Expected behaviour
Datadog does not call the opentelemetry library code and continues instrumenting my application code without errors when opentelemetry is enabled.

Steps to reproduce
Run datadog and opentelemetry in parallel. The auto-instrumentation code from datadog ends up calling the opentelemetry library code resulting in ArgumentError.

Below is the complete stack trace for the ArgumentError when running datadog and opentelemetry in parallel.

/usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-instrumentation-net_http-0.21.1/lib/opentelemetry/instrumentation/net/http/patches/instrumentation.rb:77:in `annotate_span_with_response!': wrong number of arguments (given 3, expected 2) (ArgumentError)
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-1.10.1/lib/datadog/tracing/contrib/http/instrumentation.rb:59:in `block in request'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-1.10.1/lib/datadog/tracing/trace_operation.rb:192:in `block in measure'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-1.10.1/lib/datadog/tracing/span_operation.rb:150:in `measure'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-1.10.1/lib/datadog/tracing/trace_operation.rb:192:in `measure'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-1.10.1/lib/datadog/tracing/tracer.rb:380:in `start_span'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-1.10.1/lib/datadog/tracing/tracer.rb:160:in `block in trace'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-1.10.1/lib/datadog/tracing/context.rb:43:in `activate!'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-1.10.1/lib/datadog/tracing/tracer.rb:159:in `trace'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-1.10.1/lib/datadog/tracing.rb:16:in `trace'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-1.10.1/lib/datadog/tracing/contrib/http/instrumentation.rb:40:in `request'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-instrumentation-net_http-0.21.1/lib/opentelemetry/instrumentation/net/http/patches/instrumentation.rb:38:in `block in request'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-api-1.1.0/lib/opentelemetry/trace.rb:72:in `block in with_span'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-api-1.1.0/lib/opentelemetry/context.rb:87:in `with_value'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-api-1.1.0/lib/opentelemetry/trace.rb:72:in `with_span'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-api-1.1.0/lib/opentelemetry/trace/tracer.rb:29:in `in_span'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-instrumentation-net_http-0.21.1/lib/opentelemetry/instrumentation/net/http/patches/instrumentation.rb:31:in `request'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/webmock-3.6.2/lib/webmock/http_lib_adapters/net_http.rb:97:in `block in request'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/webmock-3.6.2/lib/webmock/http_lib_adapters/net_http.rb:110:in `block in request'
	from /usr/local/lib/ruby/2.6.0/net/http.rb:920:in `start'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/webmock-3.6.2/lib/webmock/http_lib_adapters/net_http.rb:109:in `request'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/sentry-ruby-5.5.0/lib/sentry/net/http.rb:29:in `request'
	from /usr/local/lib/ruby/2.6.0/net/http.rb:1228:in `get'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/faraday-0.15.4/lib/faraday/adapter/net_http.rb:85:in `perform_request'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/faraday-0.15.4/lib/faraday/adapter/net_http.rb:43:in `block in call'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/faraday-0.15.4/lib/faraday/adapter/net_http.rb:92:in `with_net_http_connection'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/faraday-0.15.4/lib/faraday/adapter/net_http.rb:38:in `call'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/faraday-0.15.4/lib/faraday/request/url_encoded.rb:15:in `call'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/faraday-0.15.4/lib/faraday/rack_builder.rb:143:in `build_response'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/faraday-0.15.4/lib/faraday/connection.rb:387:in `run_request'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/faraday-0.15.4/lib/faraday/connection.rb:138:in `get'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/faraday-0.15.4/lib/faraday.rb:103:in `method_missing'
	from /usr/src/app/lib/avant/webpack/webpack.rb:55:in `load_manifest'
	from /usr/src/app/lib/avant/webpack/webpack.rb:44:in `initialize'
	from /usr/src/app/lib/avant/webpack/webpack.rb:12:in `new'
	from /usr/src/app/lib/avant/webpack/webpack.rb:12:in `manifest'
	from /usr/src/app/config/initializers/webpack.rb:9:in `block in <top (required)>'
	from /usr/src/app/config/initializers/webpack.rb:9:in `each'
	from /usr/src/app/config/initializers/webpack.rb:9:in `<top (required)>'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/activesupport-4.2.11.3/lib/active_support/dependencies.rb:268:in `load'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/activesupport-4.2.11.3/lib/active_support/dependencies.rb:268:in `block in load'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/activesupport-4.2.11.3/lib/active_support/dependencies.rb:240:in `load_dependency'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/activesupport-4.2.11.3/lib/active_support/dependencies.rb:268:in `load'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/railties-4.2.11.3/lib/rails/engine.rb:652:in `block in load_config_initializer'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/activesupport-4.2.11.3/lib/active_support/notifications.rb:166:in `instrument'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/railties-4.2.11.3/lib/rails/engine.rb:651:in `load_config_initializer'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/railties-4.2.11.3/lib/rails/engine.rb:616:in `block (2 levels) in <class:Engine>'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/railties-4.2.11.3/lib/rails/engine.rb:615:in `each'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/railties-4.2.11.3/lib/rails/engine.rb:615:in `block in <class:Engine>'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/railties-4.2.11.3/lib/rails/initializable.rb:30:in `instance_exec'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/railties-4.2.11.3/lib/rails/initializable.rb:30:in `run'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/railties-4.2.11.3/lib/rails/initializable.rb:55:in `block in run_initializers'
	from /usr/local/lib/ruby/2.6.0/tsort.rb:228:in `block in tsort_each'
	from /usr/local/lib/ruby/2.6.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
	from /usr/local/lib/ruby/2.6.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
	from /usr/local/lib/ruby/2.6.0/tsort.rb:431:in `each_strongly_connected_component_from'
	from /usr/local/lib/ruby/2.6.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/railties-4.2.11.3/lib/rails/initializable.rb:44:in `each'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/railties-4.2.11.3/lib/rails/initializable.rb:44:in `tsort_each_child'
	from /usr/local/lib/ruby/2.6.0/tsort.rb:415:in `call'
	from /usr/local/lib/ruby/2.6.0/tsort.rb:415:in `each_strongly_connected_component_from'
	from /usr/local/lib/ruby/2.6.0/tsort.rb:349:in `block in each_strongly_connected_component'
	from /usr/local/lib/ruby/2.6.0/tsort.rb:347:in `each'
	from /usr/local/lib/ruby/2.6.0/tsort.rb:347:in `call'
	from /usr/local/lib/ruby/2.6.0/tsort.rb:347:in `each_strongly_connected_component'
	from /usr/local/lib/ruby/2.6.0/tsort.rb:226:in `tsort_each'
	from /usr/local/lib/ruby/2.6.0/tsort.rb:205:in `tsort_each'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/railties-4.2.11.3/lib/rails/initializable.rb:54:in `run_initializers'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/railties-4.2.11.3/lib/rails/application.rb:352:in `initialize!'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/railties-4.2.11.3/lib/rails/railtie.rb:194:in `public_send'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/railties-4.2.11.3/lib/rails/railtie.rb:194:in `method_missing'
	from /usr/src/app/config/environment.rb:5:in `<top (required)>'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/activesupport-4.2.11.3/lib/active_support/dependencies.rb:274:in `require'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/activesupport-4.2.11.3/lib/active_support/dependencies.rb:274:in `block in require'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/activesupport-4.2.11.3/lib/active_support/dependencies.rb:240:in `load_dependency'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/activesupport-4.2.11.3/lib/active_support/dependencies.rb:274:in `require'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/sidekiq-4.2.10/lib/sidekiq/cli.rb:244:in `boot_system'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/sidekiq-4.2.10/lib/sidekiq/cli.rb:50:in `run'
	from /usr/src/app/vendor/gems/sidekiq-ent-1.5.4/lib/sidekiq-ent/swarm.rb:179:in `block in forkit'
	from /usr/src/app/vendor/gems/sidekiq-ent-1.5.4/lib/sidekiq-ent/swarm.rb:175:in `fork'
	from /usr/src/app/vendor/gems/sidekiq-ent-1.5.4/lib/sidekiq-ent/swarm.rb:175:in `forkit'
	from /usr/src/app/vendor/gems/sidekiq-ent-1.5.4/lib/sidekiq-ent/swarm.rb:171:in `spawn'
	from /usr/src/app/vendor/gems/sidekiq-ent-1.5.4/lib/sidekiq-ent/swarm.rb:82:in `monitor'
	from /usr/src/app/vendor/gems/sidekiq-ent-1.5.4/lib/sidekiq-ent/swarm.rb:42:in `start_and_monitor'
	from /usr/src/app/vendor/gems/sidekiq-ent-1.5.4/bin/sidekiqswarm:51:in `<top (required)>'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/bin/sidekiqswarm:23:in `load'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/bin/sidekiqswarm:23:in `<top (required)>'
	from /usr/local/lib/ruby/2.6.0/bundler/cli/exec.rb:74:in `load'
	from /usr/local/lib/ruby/2.6.0/bundler/cli/exec.rb:74:in `kernel_load'
	from /usr/local/lib/ruby/2.6.0/bundler/cli/exec.rb:28:in `run'
	from /usr/local/lib/ruby/2.6.0/bundler/cli.rb:463:in `exec'
	from /usr/local/lib/ruby/2.6.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
	from /usr/local/lib/ruby/2.6.0/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
	from /usr/local/lib/ruby/2.6.0/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
	from /usr/local/lib/ruby/2.6.0/bundler/cli.rb:27:in `dispatch'
	from /usr/local/lib/ruby/2.6.0/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
	from /usr/local/lib/ruby/2.6.0/bundler/cli.rb:18:in `start'
	from /usr/local/lib/ruby/gems/2.6.0/gems/bundler-1.17.2/exe/bundle:30:in `block in <top (required)>'
	from /usr/local/lib/ruby/2.6.0/bundler/friendly_errors.rb:124:in `with_friendly_errors'
	from /usr/local/lib/ruby/gems/2.6.0/gems/bundler-1.17.2/exe/bundle:22:in `<top (required)>'
	from /usr/local/bin/bundle:23:in `load'
	from /usr/local/bin/bundle:23:in `<main>'

How does ddtrace help you?

Environment

  • ddtrace version: 1.10.1
  • Configuration block (Datadog.configure ...):
  Datadog.configure do |c|
    c.env     = ENV['DD_SERVICE_ENV']
    c.version = ENV['DEVOPS_GIT_SHA']

    c.tracing.enabled = true
    c.agent.host = ENV['DD_AGENT_HOST']
    c.agent.port = Integer(ENV['DD_PORT']) if ENV['DD_PORT'].present?

    location = Rails.logger&.logdev&.dev || STDOUT
    c.logger.instance = Datadog::Core::Logger.new(location)
    c.logger.level = ENV.fetch('DD_LOG_LEVEL', 'warn')
    c.diagnostics.startup_logs.enabled = ENV.fetch('DD_STARTUP_LOGS_ENABLED', "false") == "true"

    c.tracing.instrument :rails
    c.tracing.instrument :redis
    c.tracing.instrument :http
    c.tracing.instrument :rest_client
    c.tracing.instrument :sidekiq,
                         service_name:        Datadog::Tracing::Contrib::Sidekiq::Ext::SERVICE_NAME,
                         client_service_name: Datadog::Tracing::Contrib::Sidekiq::Ext::CLIENT_SERVICE_NAME
    c.tracing.instrument :aws
    c.tracing.instrument :grape, service_name: Datadog::Tracing::Contrib::Grape::Ext::TAG_COMPONENT
    c.tracing.instrument :dalli
    c.tracing.instrument :concurrent_ruby
  end
  • Ruby version: 2.6.10
  • Operating system: Linux
  • **Relevant library versions:
ddtrace | 1.10.1
opentelemetry-api | 1.1.0
opentelemetry-common | 0.19.7
opentelemetry-instrumentation-all | 0.33.0
opentelemetry-instrumentation-sidekiq | 0.22.1
opentelemetry-instrumentation-net_http | 0.21.1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

Labels

bugInvolves a bugcommunityWas opened by a community membertracing

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions