Skip to content

Traces lost on upgrading to Ruby 3.1.0 & dd-trace 1.1.0 #2101

Closed

Description

Current behaviour
We have been working on upgrading from Ruby 2.6.5 to Ruby 3.1.0.

With such upgrade, we had to upgrade also dd-trace gem version from 0.54.1 to 1.1.0

Once we have done it successfully, seems we have lost APM traces data for our service.

image

The traces lost are the incoming received HTTP Requests to our service (rack-request).

Notice in the picture the release time of Ruby 3.1.0 + ddtrace 1.1.0.

It is weird because it seems, we lost some API traces but some others appear (some /v3/first_resource appear, but /v3/another_resource don't appear).
Same for lots of ActiveAdmin traces lost.

Checkout also the number of errors has been lost
image

so in general, we suspect we are loosing data for HTTP.

Meanwhile, it seems in terms of logs, we have not lost any data (this controller APM has disappeared for instance)
image

By contrast, for sidekiq for instance, all is working fine.
image

So, potentially, it might be related with our rack or rails configuration, but we don't see what could be the issue 🤔 .

Expected behaviour
ALL the HTTP requests are traced correctly in the APM.

Before upgrade configuration

Ruby
2.6.5
dd-trace
0.54.1

require './components/platform/base/graphql/schema.rb'
require File.expand_path('../../lib/datadog_tracer', __dir__)

require 'datadog/statsd'
require 'ddtrace'

Datadog.configure do |c|
  service = 'web'
  hostname = DatadogTracer.resolve_datadog_hostname
  c.tracer(
    env: Rails.env,
    priority_sampling: true,
    hostname: hostname,
    tags: DatadogTracer.tags,
    enabled: ENV['DATADOG_TRACER_ENABLED'].present?)

  c.runtime_metrics_enabled = true
  c.runtime_metrics(statsd: Datadog::Statsd.new(hostname, DatadogTracer.statsd_port))

  c.use(:rack)
  c.use(:rails, service_name: service, distributed_tracing: true)
  c.use(:sidekiq, analytics_enabled: true, service_name: "#{service}-sidekiq", client_service_name: "#{service}-sidekiq")
  c.use(:rake, analytics_enabled: true, service_name: "#{service}-rake", quantize: { args: { show: :all } })
  c.use(:aws, service_name: "#{service}-aws")
  c.use :redis, service_name: "#{service}-redis-cache"
  c.use :redis, describes: { host: ENV['REDIS_HOST'] }, service_name: "#{service}-redis-sidekiq"
  c.use(:elasticsearch, service_name: "#{service}-elasticsearch")
  c.use(:shoryuken, service_name: "#{service}-shoryuken")

  # web backend-graphql-api
  c.use :graphql, schemas: [::Platform::Base::GraphQL::Schema], analytics_enabled: true, service_name: ENV['BACKEND_GRAPHQL_API'].present? ? 'backend-graphql-api' : 'web'
end

Current configuration (issue)

Ruby
3.1.0
dd-trace
1.1.0

require './components/platform/base/graphql/schema.rb'
require File.expand_path('../../lib/datadog_tracer', __dir__)

require 'datadog/statsd'
require 'ddtrace'

Datadog.configure do |c|
  service = 'web'
  hostname = DatadogTracer.resolve_datadog_hostname
  c.env = Rails.env
  c.agent.host = hostname
  c.tags = DatadogTracer.tags

  # runtime
  c.runtime_metrics.enabled = true
  c.runtime_metrics.statsd = Datadog::Statsd.new(hostname, DatadogTracer.statsd_port)

  # tracing
  c.tracing.enabled = ENV['DATADOG_TRACER_ENABLED'].present?
  c.tracing.priority_sampling = true
  c.tracing.report_hostname = true
  c.tracing.log_injection = true
  c.tracing.partial_flush.enabled = true

  # instrumenting
  c.tracing.instrument(:action_mailer, service_name: "#{service}-action-mailer")
  c.tracing.instrument(:active_job, service_name: "#{service}-active-job")
  c.tracing.instrument(:active_record, service_name: "#{service}-active-record")
  c.tracing.instrument(:active_support, service_name: "#{service}-active-support")
  c.tracing.instrument(:aws, service_name: "#{service}-aws")
  c.tracing.instrument(:elasticsearch, service_name: "#{service}-elasticsearch")
  c.tracing.instrument(:mysql2, service_name: "#{service}-mysql2")
  c.tracing.instrument(:rack)
  c.tracing.instrument(:rails, service_name: service, distributed_tracing: true)
  c.tracing.instrument(:rake, analytics_enabled: true, service_name: "#{service}-rake", quantize: { args: { show: :all } })
  c.tracing.instrument(:redis, service_name: "#{service}-redis-cache")
  c.tracing.instrument(:redis, describes: { host: ENV['REDIS_HOST'] }, service_name: "#{service}-redis-sidekiq")
  c.tracing.instrument(:shoryuken, service_name: "#{service}-shoryuken")
  c.tracing.instrument(:sidekiq, analytics_enabled: true, service_name: "#{service}-sidekiq", client_service_name: "#{service}-sidekiq")

  # web backend-graphql-api
  c.tracing.instrument(:graphql, schemas: [::Platform::Base::GraphQL::Schema], analytics_enabled: true, service_name: ENV['BACKEND_GRAPHQL_API'].present? ? 'backend-graphql-api' : 'web')
end

Current issue Environment

  • ddtrace version:
    1.1.0
  • Ruby version:
    3.1.0
  • Operating system:
  • Relevant library versions:
    Rails 6.1.6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

Labels

bugInvolves a bugcommunityWas opened by a community member

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions