Skip to content

Unusual behavior using manual instrumentation of distributed traces using OpenTracer in v1.x #2191

Closed

Description

Hi,

We’re encountering an issue when trying to doing the major version upgrade of our codebase from ddtrace v0.54.2 to v1.2.0

We’re using OpenTracing and doing manual instrumentation between different types of services: apis —> jobs. On the jobs side, it doesn’t seem like we’re able to successfully resume a trace like we were in the 0.54.x version.

Current behaviour
When attempting to upgrade to v1.2.0, we are missing many spans from our downstream job execution (activesupport, memcached, etc.) and two of our custom traces are out of order
image

Expected behaviour
On the v0.54.2 version of ddtrace, we see the expected trace hierarchy and spans from many integrations
image

Steps to reproduce

I’ve written a small spec that fails that might be related to the problem - it seems like there’s an issue using the extracted trace context from OpenTracing.extract(…). https://github.com/okhwaja/ddtrace-bug is the repo and the test code is here: https://github.com/okhwaja/ddtrace-bug/blob/main/spec/v1_repro_spec.rb. The setup instructions are in the README

One odd thing I noticed when looking in code, the RackPropagator always returns a nil value for datadog_context since it seems a migration was made to use trace_digests instead. Then in Datadog::OpenTracer::Tracer.start_active_span, we use that always nil value. Note that if child_of were not passed in, the code would generate a span with a not-null datadog_context and use that instead.

Happy to provide more information if you need it! Appreciate any guidance or tips you’re able to provide - thanks!

How does ddtrace help you?

Environment

  • ddtrace version: v1.2.0
  • Configuration block (Datadog.configure ...): I can provide a full version if needed - summarized version below
# Important: OpenTracing.global_tracer must be assigned before the Datadog.configure block.
datadog_opentracer = Datadog::OpenTracer::Tracer.new(
  enabled: true, ...
)
OpenTracing.global_tracer = datadog_opentracer

Datadog.configure do |c|
  c.tracing.instrument :rack, request_queuing: true,
    service_name: "#{@service_name}-rack",
    web_service_name: "#{@service_name}-proxy"
  c.tracing.instrument :rails, request_queuing: true

  c.tracing.instrument :active_support, cache_service: "#{@service_name}-dbcache"

  c.tracing.instrument :active_record,
    analytics_enabled: false,
    service_name: "#{@service_name}-activerecord"
  c.tracing.instrument :dalli,
    analytics_enabled: false
  c.tracing.instrument :elasticsearch,
    analytics_enabled: false
  c.tracing.instrument :faraday,
    analytics_enabled: false
  c.tracing.instrument :http,
    analytics_enabled: false,
    split_by_domain: _to_bool(ENV["DATADOG_APM_HTTP_SPLIT_BY_DOMAIN"])
  c.tracing.instrument :redis,
    analytics_enabled: false
  c.tracing.instrument :sidekiq,
    analytics_enabled: false,
    service_name: "#{@service_name}"
  c.tracing.instrument :aws
end
  • Ruby version: v2.7.4
  • Operating system:
  • Relevant library versions: see repro repo
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

Labels

bugInvolves a bugcommunityWas opened by a community member

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions