Skip to content

Using Sentry breadcrumbs and tags alongside OpenTelemetry isn't working correctly #7538

Closed

Description

Is there an existing issue for this?

How do you use Sentry?

Sentry Saas (sentry.io)

Which SDK are you using? If you use the CDN bundles, please specify the exact bundle (e.g. bundle.tracing.min.js) in your SDK setup.

@sentry/node

SDK Version

7.43.0

Framework Version

No response

Link to Sentry event

https://paragraph.sentry.io/performance/node:08b586a3e77243548a96ac02db296d0f/?project=6606495&query=&statsPeriod=5m&transaction=POST+%2Fmetrics%2Fpost-track

SDK Setup

Sentry.init({
  dsn: "MY_DSN",

  enabled: isProduction,

  instrumenter: "otel",

  release: process.env.COMMIT,
  environment: process.env.FLY_APP_NAME || "development",


  tracesSampler: (samplingContext) => {
    // Drop transactions for routes we don't care about
    for (const route of SENTRY_IGNORED_ROUTES) {
      if (samplingContext?.request?.url?.endsWith(route)) return 0
    }

    // Drop transactions for non-production environments. (We still care about errors
    // on non-production environments, but we don't need to see all the transactions.)
    if (process.env.FLY_APP_NAME !== "paragraph-master") return 0

    return TRACE_SAMPLE_PERCENTAGE
  },

  profilesSampleRate: 1.0, // Profiling sample rate is relative to tracesSampleRate

  beforeSend: (event) => {
    if (!event.request?.data) return event

    const size = JSON.stringify(event.request?.data || "").length

    // Arbitrarily cut off massive payloads. Sentry silently drops events that are too large,
    // so trim them.
    // See https://github.com/getsentry/sentry-javascript/issues/3036#issuecomment-1066491190
    if (size > 500000) {
      event.request = event.request?.data?.slice(0, 500000)
    }

    return event
  },

  integrations: [new ProfilingIntegration()],
})

// Only load OpenTelemetry in production,
// so we aren't spamming Sentry with test data.
if (isProduction) {
  const sdk = new opentelemetry.NodeSDK({
    /*
    sampler: new opentelemetry.tracing.TraceIdRatioBasedSampler(
      TRACE_SAMPLE_PERCENTAGE
    ),
    */
    // Existing config
    traceExporter: new OTLPTraceExporter(),
    instrumentations: [
      new HttpInstrumentation({
        ignoreIncomingRequestHook(req) {
          for (const route of SENTRY_IGNORED_ROUTES) {
            if (req.url.endsWith(route)) {
              return true
            }
          }
          return false
        },
      }),

      new ExpressInstrumentation(),
      new GrpcInstrumentation({
        ignoreGrpcMethods: [
          "google.logging.v2.LoggingServiceV2/WriteLogEntries",
          new RegExp(".*google.logging.*"),
          new RegExp(".*WriteLogEntries*"),
          (name) => {
            return name.startsWith("google.logging")
          },
        ],
      }),
    ],

    // Sentry config
    spanProcessor: new SentrySpanProcessor(),
  })

  otelApi.propagation.setGlobalPropagator(new SentryPropagator())

  sdk.start()
}

Steps to Reproduce

I've been happily using Sentry for some time. using @sentry/node with Express.

Today we switched from using Sentry tracing to OpenTelemetry-based tracing, using @sentry/opentelemetry-node. Transactions are getting logged fine, but it appears that breadcrumbs and tags are not correctly added to the transaction.

Viewing any Sentry event (eg the one linked above) has tags from what seems like other requests. Additionally, the breadcrumbs also don't correspond to this request.

There's a chance I'm doing something incorrect in Sentry.init() above. For example, I don't need to use any Sentry tracing functionality now that I have OpenTelemtry tracing working, right? The documentation here is a bit light so it's unclear to me.

Additionally, I'm getting an exception on startup (see logs below).

Expected Result

I'd expect to be able to use Sentry.addBreadcrumb and Sentry.addTag throughout the codebase, and it would add the breadcrumb and tag to the current transaction/span created by Opentelemetry.

Actual Result

I get this non-fatal exception on startup:

2023-03-20T19:21:52Z app[53d5f709] iad [info]Error: @opentelemetry/api: Attempted duplicate registration of API: propagation
2023-03-20T19:21:52Z app[53d5f709] iad [info]    at registerGlobal (/workspace/node_modules/@opentelemetry/api/build/src/internal/global-utils.js:32:21)
2023-03-20T19:21:52Z app[53d5f709] iad [info]    at PropagationAPI.setGlobalPropagator (/workspace/node_modules/@opentelemetry/api/build/src/api/propagation.js:52:50)
2023-03-20T19:21:52Z app[53d5f709] iad [info]    at NodeTracerProvider.register (/workspace/node_modules/@opentelemetry/sdk-trace-base/build/src/BasicTracerProvider.js:100:31)
2023-03-20T19:21:52Z app[53d5f709] iad [info]    at NodeTracerProvider.register (/workspace/node_modules/@opentelemetry/sdk-trace-node/build/src/NodeTracerProvider.js:43:15)
2023-03-20T19:21:52Z app[53d5f709] iad [info]    at NodeSDK.start (/workspace/node_modules/@opentelemetry/sdk-node/build/src/sdk.js:158:24)
2023-03-20T19:21:52Z app[53d5f709] iad [info]    at Object.<anonymous> (/workspace/packages/api/dist/trace.cjs:127:7)
2023-03-20T19:21:52Z app[53d5f709] iad [info]    at Module._compile (node:internal/modules/cjs/loader:1159:14)
2023-03-20T19:21:52Z app[53d5f709] iad [info]    at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
2023-03-20T19:21:52Z app[53d5f709] iad [info]    at Module.load (node:internal/modules/cjs/loader:1037:32)
2023-03-20T19:21:52Z app[53d5f709] iad [info]    at Module._load (node:internal/modules/cjs/loader:878:12)

If I replace the NodeSDK init with the following, the error goes away, but I don't know if it's actually using SentryPropagator now:

  const sdk = new opentelemetry.NodeSDK({
    textMapPropagator: new SentryPropagator(),
... 
})
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    • Status

      Waiting for: Product Owner

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions