8.0.0-beta.6 massive performance degradation/connections dropped

[gajus]

Is there an existing issue for this?

• I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
• I have reviewed the documentation https://docs.sentry.io/
• I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?

Sentry Saas (sentry.io)

Which SDK are you using?

@sentry/node

SDK Version

8.0.0-beta.6

Framework Version

No response

Link to Sentry event

No response

SDK Setup

N/A

Steps to Reproduce

We upgraded from 8.0.0-alpha.9 to 8.0.0-beta.6.

Expected Result

Everything to work.

Actual Result

Experienced major performance degradation.

There are no errors per se. However, service performance has dropped to a halt.

The most noticeable symptom was occasional errors coming from pg module complaining about timeouts, e.g.

Error: timeout expired
    at Timeout._onTimeout (/srv/node_modules/.pnpm/pg@8.11.5/node_modules/pg/lib/client.js:106:28)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)

Worth noticing that we are also heavily relying on Zod. I noticed that it was mentioned in the upgrade changelog.

[mydea]

Hey, thank you for reporting this!

So zodIntegration has been added but is not enabled by default, so that's probably not related to the issue you are describing.

We did bump a bunch of underlying instrumentation, including @opentelemetry/instrumentation-pg 🤔 But I don't really see anything in there that would explain why this would become slower. Just to clarify, you only bumped @sentry/node but no other packages?

One thing you could try is to force it to use v0.38.0 of @opentelemetry/instrumentation-pg, which is what we used before. If you are using yarn, you can do this like this in your package.json

{
  "resolutions": {
    "@sentry/node/@opentelemetry/instrumentation-pg": "^0.38.0"
  }
}

I think this should work to try if that has an impact on performance 🤔 But it could of course also be another instrumentation, hard to tell. I guess you don't have a public reproduction for this that we could take a look at?

[gajus]

Same issue with rc2 just in case. Our app goes to a complete halt after trying to deploy it. Average response time increases from 150ms to over 1500ms.

What's weird is that the CPU usage remains about the same. I would suspect that some instrumentation logic is introducing big overhead, but that does not appear to be the case.

I have not had much luck replicating this in isolation, i.e. we are only seeing this when deployed to production, which would indicate that this issue surfaces under high concurrency.

Will continue to troubleshoot and report back if we can locate the root cause.

[mydea]

Thanks for the added info!

Could you share your Sentry.init() code, and also maybe your app setup - e.g. what framework (express, fastify, ...) are you using, what middleware, other relevant config, ...?

[gajus]

Could you share your Sentry.init() code, and also maybe your app setup - e.g. what framework (express, fastify, ...) are you using, what middleware, other relevant config, ...?

import { RequestError } from '@contra/http-client';
import { init, rewriteFramesIntegration } from '@sentry/node';
import { nodeProfilingIntegration } from '@sentry/profiling-node';
import { type Integration } from '@sentry/types';

export { findActiveSpan } from './findActiveSpan.js';
export { getTraceAndBaggage } from './getTraceAndBaggage.js';
export { startSpan } from '@sentry/core';
export {
  type Breadcrumb,
  captureException,
  captureMessage,
  close,
  setupFastifyErrorHandler,
  setUser,
  withScope,
} from '@sentry/node';
export { type Extras } from '@sentry/types';

const {
  NODE_ENV,
  NODE_PROFILING,
  // eslint-disable-next-line n/no-process-env
} = process.env;

type ErrorDetail = {
  message: string;
  path: string[];
};

// Type Guard for ErrorDetail
const isErrorDetail = (error: unknown): error is ErrorDetail => {
  return (
    typeof error === 'object' &&
    error !== null &&
    'message' in error &&
    'path' in error &&
    typeof error.message === 'string' &&
    Array.isArray(error.path)
  );
};

// Extracts error message from JSON error string
const extractErrorMessage = (errorString: string): string | null => {
  try {
    const outerArray = JSON.parse(errorString);
    const parsed = outerArray[0];

    if (Array.isArray(parsed) && parsed.every(isErrorDetail)) {
      return parsed.map((error) => error.message).join(', ');
    } else {
      return parsed.message;
    }
  } catch {
    return null;
  }
};

// Extracts path from JSON error string and removes index of edges
const extractErrorPath = (errorString: string): string | null => {
  try {
    const outerArray = JSON.parse(errorString);
    const parsed = outerArray[0];

    if (Array.isArray(parsed) && parsed.every(isErrorDetail)) {
      return parsed
        .map((error) => error.path.join('.').replaceAll(/\d+/gu, ''))
        .filter(Boolean)
        .join(', ');
    } else {
      return parsed.path.join('.').replaceAll(/\d+/gu, '');
    }
  } catch {
    return null;
  }
};

// Extracts error details
const extractErrorDetails = (error: unknown): Record<string, string | null> => {
  let errorMessage: string | null = null;
  let errorPath: string | null = null;

  errorMessage = extractErrorMessage(typeof error === 'string' ? error : '');
  errorPath = extractErrorPath(typeof error === 'string' ? error : '');

  return { errorMessage, errorPath };
};

const formatFingerprint = (fingerprint: string[]): string[] => {
  return fingerprint.map((item) => {
    const { errorMessage, errorPath } = extractErrorDetails(item);
    if (errorMessage && errorPath) {
      return `{message: "${errorMessage}", path: "${errorPath}"}`;
    }

    return item;
  });
};

export type SentryTracing = {
  shutdown: () => Promise<void>;
};

export const createSentry = ({
  integrations,
  root,
}: {
  integrations?: (integrations: Integration[]) => Integration[];
  root: string;
}) => {
  init({
    beforeSend(event, hint) {
      let fingerprint =
        typeof hint.originalException === 'string'
          ? [hint.originalException]
          : hint.originalException instanceof Error
            ? [hint.originalException?.message]
            : undefined;

      if (!event.contexts) {
        event.contexts = {};
      }

      if (hint.originalException instanceof RequestError) {
        event.contexts.httpRequest = {
          method: hint.originalException.request?.options.method,
          // eslint-disable-next-line @typescript-eslint/no-explicit-any
          source: (hint.originalException as any)?.sourceTrace,
          url: hint.originalException.request?.requestUrl,
        };
        event.contexts.httpResponse = {
          body: hint.originalException?.response?.body ?? null,
          statusCode: hint.originalException?.response?.statusCode,
        };
        fingerprint = [
          hint.originalException.message,
          hint.originalException.request?.options.method ?? '',
          // @ts-expect-error TODO why origin does not exist?
          String(hint.originalException.request?.options.url?.origin),
        ];
      }

      if (fingerprint) {
        if (event.contexts.graphql) {
          fingerprint.push(event.contexts.graphql.path as string);
        } else if (event.contexts.temporal) {
          fingerprint.push(event.contexts.temporal.workflowType as string);
        }
      }

      if (fingerprint) {
        event.fingerprint = formatFingerprint(fingerprint);
      } else {
        event.fingerprint = fingerprint;
      }

      return event;
    },
    integrations: (defaultIntegrations) => {
      const customIntegrations = [
        ...defaultIntegrations,
        rewriteFramesIntegration({
          root,
        }),
      ];

      if (NODE_PROFILING === 'true') {
        customIntegrations.push(nodeProfilingIntegration());
      }

      if (integrations) {
        return integrations(customIntegrations);
      }

      return customIntegrations;
    },
    normalizeDepth: 5,
    spotlight: NODE_ENV === 'development',
    tracesSampleRate: 1,
  });

  return {
    shutdown: async () => {
      // This was made redundant by Sentry abstracting away opentelemetry
    },
  };
};