feat(node-experimental): Keep breadcrumbs on transaction

[mydea]

This implements an experimental way to handle breadcrumbs for the POTEL world.

This patches stuff so that we get the breadcrumbs from the root transaction, instead of from the scope. This is how it works:

• Patches Scope.addBreadcrumb() to fetch the current transaction, and if there is one, add breadcrumbs there
• Patches Scope.getBreadcrumbs() to fetch breadcrumbs from current transaction (if there is one)
• Patches hub stuff to ensure we always create a patched OtelScope
• Patches transaction in-place via Proxy to allow to store breadcrumbs on them.
• Patches _prepareEvent on the experimental node Client to ensure captureContext works

How does that work with breadcrumbs in practice

This means that for each isolated request (=that creates its own transaction), we'll keep the breadcrumbs in there. So anything that happens inside of this branch will get all breadcrumbs from this transaction.

This works well 98% of the time, in my testing. For most scenarios, this resulted in the most correct (subjectively speaking) breadcrumbs for both transactions as well as errors.

Note that one scenario where this may not work as expected is with parallel withScope() calls, e.g.:

 app.get("/multiple-scopes", (request, reply) => {
    const id = Math.floor(Math.random() * 1000000);
    console.log(`parallel request ${id}`);

    Sentry.withScope((scope) => {
      const inner = `scope1-${id}`;
      console.log(inner);
      Sentry.captureException(new Error(`parallel error ${inner}`));
    });
    
    Sentry.withScope((scope) => {
      const inner = `scope2-${id}`;
      console.log(inner);
      Sentry.captureException(new Error(`parallel error ${inner}`));
    });
  });

In this scenario, parallel error 2 will also show console.log(scope1-...).
For me, this is an acceptable case for now.

On the other hand, here are some cases where it works as expected:

app.get("/parallel-error", (request, reply) => {
    const id = Math.floor(Math.random() * 1000000);
    console.log(`parallel request ${id}`);

    Sentry.withScope((scope) => {
      scope.setTag("parallel-outer", `${id}`);
      scope.setUser({id: `user-${id}`, email: 'test@example.com' });
      console.log('outer scope', scope._tags)

      setTimeout(() => {
        makeRequest(`http://example.com/${id}`).then(() => {
          Sentry.withScope((innerScope) => {
            innerScope.setTag("parallel-inner", `${id}`);
            innerScope.setTag(`parallel-inner-${id}`, `${id}`);

            console.log("inner scope", innerScope._tags);

            Sentry.captureException(new Error(`parallel error ${id}`));
            reply.send({ hello: "world", status: 'OK', id });
          })
        })
      }, 5000);
    });
  });

app.get("/error", async (request, reply) => {
    console.log('before request');

    await makeRequest(`http://example.com/`);

    console.log('after request');

    Sentry.captureException(new Error(`test error`));

    console.log('after capture');
    reply.send({status: "OK"})
  });

Overall, I think it works much better than the status quo of the POTEL-backed version.

TODO: What if no transactions?

With the current implementation, this means that if you do not enable tracing, you'll get "incorrect" breadcrumbs again (=the old behavior).

We actually always initialize OTEL (even if tracing is not enabled), but we do not setup any default performance integrations, thus do not generate any transactions by default etc. Probably OK for now (=it will still kind of work, but not great), something to think about later.

[github-actions]

size-limit report 📦

Path	Size
@sentry/browser (incl. Tracing, Replay) - Webpack (gzipped)	75.29 KB (-0.1% 🔽)
@sentry/browser (incl. Tracing) - Webpack (gzipped)	31.29 KB (+0.06% 🔺)
@sentry/browser - Webpack (gzipped)	21.89 KB (+0.07% 🔺)
@sentry/browser (incl. Tracing, Replay) - ES6 CDN Bundle (gzipped)	70 KB (-0.14% 🔽)
@sentry/browser (incl. Tracing) - ES6 CDN Bundle (gzipped)	28.4 KB (-0.07% 🔽)
@sentry/browser - ES6 CDN Bundle (gzipped)	20.47 KB (-0.06% 🔽)
@sentry/browser (incl. Tracing, Replay) - ES6 CDN Bundle (minified & uncompressed)	221.17 KB (-0.17% 🔽)
@sentry/browser (incl. Tracing) - ES6 CDN Bundle (minified & uncompressed)	85.82 KB (+0.01% 🔺)
@sentry/browser - ES6 CDN Bundle (minified & uncompressed)	60.67 KB (+0.02% 🔺)
@sentry/browser (incl. Tracing) - ES5 CDN Bundle (gzipped)	31.28 KB (-0.06% 🔽)
@sentry/react (incl. Tracing, Replay) - Webpack (gzipped)	75.32 KB (-0.1% 🔽)
@sentry/react - Webpack (gzipped)	21.92 KB (+0.07% 🔺)
@sentry/nextjs Client (incl. Tracing, Replay) - Webpack (gzipped)	93.18 KB (-0.08% 🔽)
@sentry/nextjs Client - Webpack (gzipped)	50.85 KB (+0.04% 🔺)

[AbhiPrasad]

Let's experiment with this, I'd be interested to see what the data looks like.

[AbhiPrasad]

l: we can proxy patch startTransaction (hub extension) so that we don't have to rely on getActiveTransaction being used all the time.

Can skip this for now just to get it merged.

[mydea]

Good idea, wrote something up here: #9010