Sending only those Transactions which are related to transmitted Events (Exceptions)

[softworkz]

Context Data for Error Reporting
Inclusion of Transactions/Spans, Logs and Metrics

The Problem

Sentry supports quite a range of data which is useful and sometimes crucial for understanding bugs, problems and errors, namely: Traces, Logs and Metrics.

But when you want to use error reporting only, that data is not included in error reports.

Please correct me where wrong, but as far as I understand and according to my testing so far, the following seems to be true:

• Traces
  In order to have error reports enriched by trace information (transations/spans), it is required
  • to send ALL trace data to Sentry
  • to set the trace sampling rate to 1.0
  • if you set the trace sampling rate to anything lower than 1.0, you will no longer get trace information included in all error reports
    (because when using trace sampling, the decision about sending a trace or not is made at the beginning of a new transaction where it isn't known yet, whether an error will occur or not)
• Logs
  • log items are associated with a trace, but sent separately
  • they are sent before a transaction completes
  • also, they are sent, before an error occurs, so when you filter out log items and an error occurs, the log items preceding the error are already gone
• Metrics
  Same story like for logs

Assessment

One can get the impression that this might be driven by intent - for maximizing the amount of data that Sentry users are required to transmit in order to get useful data for analyzing errors.
Anyway - whether intentional or not: - this is forcing Sentry customers to send huge amounts of data, even though there's just a fraction of that data which is actually of interest - that's quite unfortunate, to say the least.

Target Use Cases

It seems that large parts of Sentry are tailored for situations where customers are running a range of backend services, serving their users which are accessing those services by a certain client/frontend software.

Unlike others, we are not offering a service, but a software, and our server software can produce a huge amount of transactions as a single instance. We're using Sentry in beta versions only, but that's still above 1k installations. Sending full logs, traces and metrics to Sentry would be totally insane. Not just because of the amounts of data, but also because it's data that we wouldn't ever look at or have use for it in any way.
It would be a stupidity, burning time, money and energy - just for nothing.

Full traces and logs data can be useful at times - but only when investigating specific issues on a specific installation. In those cases, it can still be enabled individually, but that can't be the default setup for the masses.

What's needed?

For Session Replays, the Sentry implementation is recording the required data continuously - but only when an error is going to be reported, the related session replay data from shortly before until shortly after the error occurrence is eventually transmitted to Sentry.

The exact same thing should be possible with the Sentry .NET SDK (and possibly others):

Implement a special reporting mode, where the SDK operates as follows:

• It record transactions/spans, metrics and log data
  • It caches the data
  • But it doesn't sent it
• Only when an error event is reported, the related transactions/spans, metrics and log data gets transmitted as well

[linear-code]

NET-630

[softworkz]

Workaround

As a temporary workaround, here's a solution that I've come up with. It appears to be working, even though I'm not sure whether it's correct in every detail...

public class SentryProvider
{
    private const int LogBufferSize = 100;

    private const string SentryDsn = @"....";
    private const string LogTriggerToken = "buffer_trigger";

    private readonly bool sendAllData;
    private readonly ConcurrentDictionary<SentryId, DateTimeOffset> tracesToSend = new();
    private readonly ConcurrentDictionary<SpanId, DateTimeOffset> spansToSend = new();
    private readonly ConcurrentQueue<SentryLog> logBuffer = new();
    private readonly ConcurrentQueue<SentryLog> logSendQueue = new();


    public SentryProvider()
    {
        this.sendAllData = false; // For demo
        this.SetupSentry();
    }

    private void SetupSentry()
    {
        SentrySdk.Init(options =>
        {
            options.Dsn = SentryDsn;

            options.AutoSessionTracking = true;
            options.IsGlobalModeEnabled = false;
            options.DeduplicateMode = DeduplicateMode.All;
            options.StackTraceMode = StackTraceMode.Enhanced;

            options.TracesSampleRate = 1.0;

            options.EnableLogs = true;

            options.SetBeforeSend(ev =>
            {
                if (!this.sendAllData)
                {
                    this.tracesToSend.TryAdd(ev.Contexts.Trace.TraceId, DateTimeOffset.Now);
                    this.tracesToSend.TryAdd(ev.EventId, DateTimeOffset.Now);
                    this.spansToSend.TryAdd(ev.Contexts.Trace.SpanId, DateTimeOffset.Now);

                    if (ev.Contexts.Trace.ParentSpanId.HasValue)
                    {
                        this.spansToSend.TryAdd(ev.Contexts.Trace.ParentSpanId.Value, DateTimeOffset.Now);
                    }

                    Task.Run(this.SendApplicableLogMessages);
                }

                return ev;
            });

            options.SetBeforeSendTransaction(transaction =>
            {
                if (!this.sendAllData
                    && !(this.tracesToSend.ContainsKey(transaction.TraceId)
                    || this.tracesToSend.ContainsKey(transaction.EventId)
                    || this.spansToSend.ContainsKey(transaction.SpanId)
                    || transaction.Spans.Any(e => this.spansToSend.ContainsKey(e.SpanId))))
                {
                    return null;
                }

                return transaction;
            });

            options.SetBeforeSendLog(log =>
            {
                if (log.Message == LogTriggerToken)
                {
                    if (this.logSendQueue.TryDequeue(out var queuedLog))
                    {
                        return queuedLog;
                    }

                    return null;
                }

                if (!this.sendAllData
                    && !(this.tracesToSend.ContainsKey(log.TraceId)
                         || (log.SpanId.HasValue && this.spansToSend.ContainsKey(log.SpanId.Value))))
                {
                    this.BufferLogMessage(log);
                    return null;
                }

                return log;
            });
        });
    }

    private void BufferLogMessage(SentryLog logMessage)
    {
        lock (this.logBuffer)
        {
            while (this.logBuffer.Count >= LogBufferSize)
            {
                this.logBuffer.TryDequeue(out _);
            }

            this.logBuffer.Enqueue(logMessage);
        }
    }

    private void SendApplicableLogMessages()
    {
        lock (this.logBuffer)
        {
            var applicable = this.logBuffer
                .Where(log => this.tracesToSend.ContainsKey(log.TraceId)
                              || (log.SpanId.HasValue && this.spansToSend.ContainsKey(log.SpanId.Value)))
                .ToList();

            Task.Run(() => this.SendQueuedLogMessaged(applicable));

            var remainder = this.logBuffer.Except(applicable);

            this.logBuffer.Clear();

            remainder.ForEach(this.logBuffer.Enqueue);
        }
    }

    private void SendQueuedLogMessaged(IList<SentryLog> logMessages)
    {
        logMessages.ForEach(this.logSendQueue.Enqueue);

        // One extra to be safe
        for (int i = 0; i <= logMessages.Count; i++)
        {
            SentrySdk.Logger.LogTrace(LogTriggerToken);
        }
    }
}

[jamescrosswell]

For context, we were creating full transactions regardless of whether the transaction would be sampled or not... but particularly for complex transactions with loads of spans in them, this sometimes lead to memory bloat and performance issues. We figured, if we made the sampling decision early, we could instead instantiate a light weight UnsampledTransaction that was way easier on the memory load:

• UnsampledTransactions to reduce memory pressure #4212

If you're OK with that compromise, you could potentially set your sample rate to 1.0 so that everything gets sampled, but then in the BeforeSendTransaction callback decide whether you actually want to send the transaction or not (e.g. by checking whether the transaction has an exception associated with it or not).

[softworkz]

If you're OK with that compromise, you could potentially set your sample rate to 1.0 so that everything gets sampled, but then in the BeforeSendTransaction callback decide whether you actually want to send the transaction or not (e.g. by checking whether the transaction has an exception associated with it or not).

Well, that's what the code is doing that I posted above. I'm not sure, though, whether that's the right way for "checking whether the transaction has an exception associated with it or not"...?

[softworkz]

that was way easier on the memory load:

• UnsampledTransactions to reduce memory pressure #4212

If you're OK with that compromise

A more useful way would be to get an indication of the current amount of spans in a transaction so that we can stop adding once it has reached a certain limit (or under specific conditions).