Linking Sentry errors to your logging tools

[Dhrumil-Sentry]

Error details, Replays, and Breadcrumbs in Sentry Issues are beneficial for debugging and fixing app problems. However, we know there are cases when you’d want to review your logs as well to get a more exhaustive picture of what transpired when the error occurred.

We’re brainstorming an idea that lets you navigate directly from an error event in Sentry to the corresponding logs in your logging tool.

How might this work?

Sharing Context between Logs and Sentry

When an error occurs, the Sentry SDK captures it along with a unique event_id. This event_id can be programmatically accessed and then included in your logging statements. For example, in a Python application:

import sentry_sdk

sentry_sdk.init("your_dsn_here")

try:
    # Your code that might fail
    1 / 0
except Exception as e:
    event_id = sentry_sdk.capture_exception(e)
    # Include this event_id in your log
    **your_logging_function**(f"Error occurred with event_id: {event_id}")

Linking to Logs from Sentry

On the Sentry side, we’d ask you to add configuration details of your logging integrations:

• Base URL, Query format, and other details to enable creating a query link to the logs

With these details, Sentry could construct a URL for each error event you view in our app

query=event_id:"${eventContext.event_id}"&from=${eventContext.timestamp - 600}&to=${eventContext.timestamp}

This would allow you to jump to your logs as you view various errors in Sentry.

We need your feedback 🙏

• What logging tools do you use?
• What does your current Sentry/Logging workflow look like?
  • Do you find yourself looking at logs after seeing an error in Sentry? How do you search for the relevant logs?
• Do you find this idea to be useful for your debugging workflow?

We’d love your input on the questions above and please let us know if there’s anything we’re missing or should do differently.

[mwarkentin]

I would recommend taking a look through notion to see how teams currently handle log queries in our own systems. :)

Here's a quite extensive set of queries from 3S for various systems that they care about.

Here's how one of those translates to a URL:

https://console.cloud.google.com/logs/query;query=--%20Service%20etc%0Aresource.type%20%3D%20k8s_container%0Aresource.labels.namespace_name%20%3D%20default%0Alabels.%22k8s-pod%2Fservice%22%20%3D%20%22snuba-admin%22%0A%0A--%20Audit%20log%20module%0AjsonPayload.module%20%3D%20%22snuba.admin.audit_log.base%22%0A%0A--%20Only%20show%20successful%20queries%0AjsonPayload.status%20%3D%20%22succeeded%22;cursorTimestamp=2024-01-04T15:15:41.329332Z;duration=PT1H?project=internal-sentry

[AbhiPrasad]

I would aim to recommend attaching the trace/span ID to link logs to errors/spans/crons. With the tracing without performance initiative, every event generated from an sdk should have a trace and span ID attached to them.

This way we have a couple of benefits:

• We align more with OpenTelemetry, see https://opentelemetry.io/docs/specs/otel/logs/#log-correlation.
• We can do frontend/backend tracing much easier (see backend logs related to frontend error) - user's don't have to propagate an event ID across services themselves.

[Dhrumil-Sentry]

This is a great suggestion! 🙌 Not having to send an event ID makes this a lot easier to set up

[tonyo]

+1 on making it easier to jump from a Sentry issue to logging services (the "Linking to Logs from Sentry" section); it should reduce manual work required to search for the related logs file, and hence bring down the mean-time-to-repair.

Following up on the idea, a few notes/ideas:

• Different logging tools use different timestamp formats in URLs, so we should support rendering of different formats (ISO 8601, epoch, etc.)
• Basic arithmetic should definitely be supported (like in the initial example -- to build the from-to time range)
• Giving access to more event fields/attributes would be cool: e.g. tags and additional event contexts could be used to build a more specific query, so that less filtering will be needed in the logging service.

Generally speaking, if we add support for user-defined links, it'll allow to link Sentry issues not only to logging services, but pretty much any other.

I drafted a more generic proposal some time ago here (internal for now).

[adamgsg]

+1 on this feature.

We would use this to link from Sentry to Kibana to view ElasticSearch logs. Using the Sentry event_id is not useful, however being able to construct a link using timestamps, tags, and context would be perfect. Being able to build the timestamp in arbitrary formats will be crucial.

It would also be useful to display multiple links. For example, links to multiple logging systems, or different views within the same logging system.

[Dhrumil-Sentry]

Hi @adamgsg - Thanks a lot for this feedback!

A few questions:

however being able to construct a link using timestamps, tags, and context would be perfect

Would this link query change for each issue or do you think setting this up once for your Sentry project will suffice for your issues?

Being able to build the timestamp in arbitrary formats will be crucial.

Could you please provide some examples of formats you'd like to see supported 🙏

[adamgsg]

Would this link query change for each issue or do you think setting this up once for your Sentry project will suffice for your issues

The link would change for each event (not just issue).

The idea is, upon seeing an error event in Sentry, click the "view logs" link for that event, to be taken to our logging system, and see logs related to that event. In our case, we'd likely have one link that views all logs related to that event across all systems. Another link that showed the user's activity immediately before/after the error event (so we could see what might have led up to the error, and attempt to reproduce it ourselves).

Could you please provide some examples of formats you'd like to see supported

epoch (seconds), epoch (milliseconds), ISO 8601.

[ajesler-hatch]

This sounds like a feature I'd use fairly frequently when triaging sentry reports if it was available.

What logging tools do you use?

We use Datadog.

Do you find yourself looking at logs after seeing an error in Sentry? How do you search for the relevant logs?
What does your current Sentry/Logging workflow look like?

Yes.
Depending on how the error is triggered (frontend vs backend, scheduled tasks/event vs web request) we have different correlating identifiers available. Eg user context, request id, trace & span, job id, etc.
Worst case, there are none and you have to search the logs around the time of the error.
Best case, we have a correlation id that is also in the logs, eg a request id and we can accurately search based on that.
More common than I'd like case, all we have is a user id, so we'd search logs tagged with the user id around the error time. This is often the case for front end Javascript events, and usually there isn't much useful in the logs, the Sentry breadcrumbs are generally more helpful.

Do you find this idea to be useful for your debugging workflow?

Yes.

Would this link query change for each issue or do you think setting this up once for your Sentry project will suffice for your issues?
Could you please provide some examples of formats you'd like to see supported 🙏

adamgsg's suggestions match what I'd like. Being able to pull in tags like environment, request_id, trace_id, etc would mean better scoping of log searches.
I don't think we record the Sentry event_id anywhere in our logs, so using attributes and tags from the event would be more useful generally.

https://app.datadoghq.com/logs?query=8fa16fea-0356-447d-ae6f-c4865ed207fa%20%40env%3Aproduction&cols=host%2Cservice&index=%2A&stream_sort=desc&viz=stream&from_ts=1707364113667&to_ts=1707968913667 is an example of a query I'd find it useful for Sentry to generate.
This tells Datadog I want to search for 8fa16fea-0356-447d-ae6f-c4865ed207fa in the production environment logs, between the epochs 1707364113667 and 1707968913667.
8fa16fea-0356-447d-ae6f-c4865ed207fa is the value of the request_id tag in this case.

[Dhrumil-Sentry]

@ajesler-hatch Thanks a lot for this detailed feedback! 🙌

[ykamo001]

Really cool feature idea!

A question/concern I had regarding the URL link experience: I wonder if Sentry would get pinged a lot if the link/url is "broken," but the client just doesn't have the data retention policy for the logs.

If the user clicks on the url, it takes them to, for example, gcp, but their log retention policy is only 24 hours, would the user think something is broken on Sentry side? Would each log vendor have a different experience? Maybe GCP just loads the default view, but Datadog it 404s? Would it be a misleading experience for the user?

[edhgoose]

We would love this feature.

We have a uniquely generated event ID for all our requests, and we generate a link to our self hosted version of Kibana which contains that value. We then use Sentry to add a tag with that Kibana link.

Unfortunately, we're limited by only being able to add 200 characters, and the link is significantly longer. e.g. here's a link which is for a search of uniqueId:uniqueIdValue (where uniqueId is the field name, and uniqueIdValue is the value we're looking for):

https://our-logging-cluster/_plugin/kibana/app/kibana#/discover?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-24h,mode:quick,to:now))&_a=(columns:!(_source),index:bd5d0440-fc4a-11e9-9360-01330cb660d7,interval:auto,query:(language:lucene,query:'uniqueId:uniqueIdValue'),sort:!('@timestamp',desc))

It gets even more obnoxious if you add specific dates in as well.

The time component becomes: time:(from:'2024-03-05T00:00:00.000Z',mode:absolute,to:'2024-03-06T23:59:59.999Z').

I hope this helps - we'd be delighted to test and help.

[Dhrumil-Sentry]

@edhgoose thanks a lot for this feedback, wrt uniqueId:uniqueIdValue would you send these as tags to Sentry? I am trying to understand what event properties would you need to generate the link to your logs

[edhgoose]

Hey - yes, we do send those values into Sentry currently.

What I'd love is the ability to build a URL from those values and dates.

In fact, we have quite a few values we'd like to be able to click on and open our logs for.

E.g. we might have an IP address - it'd be awesome to click on that and open our log search for that IP.