`http.route` missing from spans after enabling Sentry with OTEL

[jirimoravcik]

Is there an existing issue for this?

• I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
• I have reviewed the documentation https://docs.sentry.io/
• I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?

Sentry Saas (sentry.io)

Which SDK are you using?

@sentry/node

SDK Version

8.43.0

Framework Version

Node js 20.12.2

Link to Sentry event

No response

Reproduction Example/SDK Setup

First, we initialize sentry using:

Sentry.init({
        dsn: process.env.SENTRY_DSN,
        environment: process.env.SENTRY_ENVIRONMENT || 'local',
        release: process.env.BUILD_VERSION || 'local_version',
        skipOpenTelemetrySetup: true, // We have our custom OpenTelemetry setup
        registerEsmLoaderHooks: false,
        beforeBreadcrumb() { return null; }, // Disable breadcrumbs
        beforeSend(event, hint) {
            // some stuff that doesn't affect this in any way
        },
    });

Next, we initialize OpenTelemetry SDK using:

import type { InstrumentationConfig } from '@opentelemetry/instrumentation';
import { AwsInstrumentation, type AwsSdkInstrumentationConfig } from '@opentelemetry/instrumentation-aws-sdk';
import { DnsInstrumentation, type DnsInstrumentationConfig } from '@opentelemetry/instrumentation-dns';
import { ExpressInstrumentation, type ExpressInstrumentationConfig } from '@opentelemetry/instrumentation-express';
import { FsInstrumentation, type FsInstrumentationConfig } from '@opentelemetry/instrumentation-fs';
import { HttpInstrumentation, type HttpInstrumentationConfig } from '@opentelemetry/instrumentation-http';
import { IORedisInstrumentation, type IORedisInstrumentationConfig } from '@opentelemetry/instrumentation-ioredis';
import { MongoDBInstrumentation, type MongoDBInstrumentationConfig } from '@opentelemetry/instrumentation-mongodb';
import { NestInstrumentation } from '@opentelemetry/instrumentation-nestjs-core';
import { NetInstrumentation } from '@opentelemetry/instrumentation-net';
import { envDetectorSync } from '@opentelemetry/resources';
import { NodeSDK } from '@opentelemetry/sdk-node';

const sdk = new NodeSDK({
    instrumentations: [
        new AwsInstrumentation({
            //  This hides the underlying HTTP spans that aren't useful since we already have instrumentation for AWS SDK
            suppressInternalInstrumentation: true,
            ...config?.awsInstrumentationConfig,
        }),
        new DnsInstrumentation(config?.dnsInstrumentationConfig),
        new ExpressInstrumentation(config?.expressInstrumentationConfig),
        new FsInstrumentation(config?.fsInstrumentationConfig),
        new HttpInstrumentation({
            ignoreIncomingRequestHook(req) {
                // Do not include health check, icon, and metrics endpoints in traces.
                const isHealthCheckIconOrMetricsUrl = ['/health-check', '/favicon.ico', '/metrics'].includes(req.url || '');
                return isHealthCheckIconOrMetricsUrl;
            },
            ...config?.httpInstrumentationConfig,
        }),
        new IORedisInstrumentation(config?.ioRedisInstrumentationConfig),
        new MongoDBInstrumentation(config?.mongoDBInstrumentationConfig),
        new NestInstrumentation(config?.nestInstrumentationConfig),
        new NetInstrumentation(config?.netInstrumentationConfig),
    ],
    resourceDetectors: [envDetectorSync],
});

sdk.start();

Steps to Reproduce

If you use the setup above, you'll have spans that look like following (coming from debug OpenTelemetry logs):

Span {
    attributes: {
      'http.url': 'http://localhost:3333/v2/users/me?token=secret',
      'http.host': 'localhost:3333',
      'net.host.name': 'localhost',
      'http.method': 'GET',
      'http.scheme': 'http',
      'http.target': '/v2/users/me?token=secret',
      'http.user_agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36',
      'http.flavor': '1.1',
      'net.transport': 'ip_tcp',
      'sentry.origin': 'auto.http.otel.http',
      'net.host.ip': '::1',
      'net.host.port': 3333,
      'net.peer.ip': '::1',
      'net.peer.port': 59381,
      'http.status_code': 200,
      'http.status_text': 'OK'
    },
}

If I turn Sentry instrumentation off, there'll also be http.route, which is coming from OpenTelemetry express instrumentation.

If I instrument OpenTelemetry before Sentry, it breaks the ignoreIncomingRequestHook defined above (i.e. it doesn't get called at all, probably overwritten by something from Sentry)

Expected Result

The expected span would be something like the following (which I can get when I turn Sentry off):

Span {
    attributes: {
      'http.url': 'http://localhost:3333/v2/users/me?token=secret',
      'http.host': 'localhost:3333',
      'net.host.name': 'localhost',
      'http.method': 'GET',
      'http.scheme': 'http',
      'http.target': '/v2/users/me?token=secret',
      'http.user_agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36',
      'http.flavor': '1.1',
      'net.transport': 'ip_tcp',
      'net.host.ip': '::1',
      'net.host.port': 3333,
      'net.peer.ip': '::1',
      'net.peer.port': 61320,
      'http.status_code': 200,
      'http.status_text': 'OK',
      'http.route': '/v2/users/:username'
    },
}

you can notice the http.route at the end, that's what's missing.

Actual Result

Span {
    attributes: {
      'http.url': 'http://localhost:3333/v2/users/me?token=secret',
      'http.host': 'localhost:3333',
      'net.host.name': 'localhost',
      'http.method': 'GET',
      'http.scheme': 'http',
      'http.target': '/v2/users/me?token=secret',
      'http.user_agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36',
      'http.flavor': '1.1',
      'net.transport': 'ip_tcp',
      'sentry.origin': 'auto.http.otel.http',
      'net.host.ip': '::1',
      'net.host.port': 3333,
      'net.peer.ip': '::1',
      'net.peer.port': 59381,
      'http.status_code': 200,
      'http.status_text': 'OK'
    },
}

[lforst]

Hi, we need to narrow this down a bit (unless you happen to be able to provide repro). Is the route attr that is missing related to express or nestjs? I am wondering if we are stomping the options of one of the instrumentations you provided.

[jirimoravcik]

Hello, it's an express API. IMO you should find the actual code doing it in https://github.com/open-telemetry/opentelemetry-js-contrib/blob/main/plugins/node/opentelemetry-instrumentation-express/src/instrumentation.ts#L175-L183

Let me know if it helps. Thank you

[lforst]

Thanks. Looking at our code, I cannot find any places relevant to this stack where we are overriding http.route 🤔 Would you mind providing a minimal repro example we could pull to reproduce this and debug more effectively?

[jirimoravcik]

I'll try to prepare a repro example for you.