[APM] Display related trace logs in the trace sample content

[formgeist]

Blocked: An embeddable log component from Logs team is required.

Summary

Based on the design solution in elastic/apm#179

We're currently linking out to the Logs app to display the related logs for a specific trace.id, but it would be a better experience to show the logs in the context of the trace sample immediately available from within the APM app.

Design proposal

Figma prototype link

Possible API

<LogStream 
 timestamp="1590690626648" 
 filter={encodeURIComponent('trace.id:"0570667f4e27e2cac0d6c5b311c65918"')} 
/>

Prerequisites for starting implementation

There's a number of things that can be seen as dependencies for this feature to be implemented and this work should be planned ahead when we prioritize this feature.

• A way to display the log events in a similar style as the Logs stream in the Logs app. Currently the log stream is not embedabble.
• The embedded view would also need to support actions like "View log line in context" and "View log details" which would link to the Logs app with the relevant views.
• Ways of adding our own data columns to the logs viewer e.g. the service legend (see screenshots).

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[tbragin]

Big +1 from me.

I see this seems blocked by what looks like Logs Stream app being embeddable.

@elastic/logs-metrics-ui do we have an issue we're tracking around converting Logs Stream app to an embeddable component?

cc @roncohen @alvarolobato @mukeshelastic @cyrille-leclerc

[weltenwort]

do we have an issue we're tracking around converting Logs Stream app to an embeddable component?

Not yet - the approach probably highly depends on how exactly the embedding UI would have to control and inject the source configuration.

[sorenlouv]

@weltenwort makes sense. What do you think about a simple first version that takes the same parameters as the link from apm to logs (timestamp and filter)

<LogStream 
 timestamp="1590690626648" 
 filter={encodeURIComponent('trace.id:"0570667f4e27e2cac0d6c5b311c65918"')} 
/>

In the first version I think it's fine if it's just a static number of log lines. If users want to see more lines, they can click "Open in Logs" which will take them to the logs app.

[sorenlouv]

An alternative approach could be to specify the time range:

<LogStream 
 startTime="1590690626648" 
 endTime="1590690636648" 
 filter={encodeURIComponent('trace.id:"0570667f4e27e2cac0d6c5b311c65918"')} 
/>

[graphaelli]

@formgeist and I came up with two other contexts (besides per-trace) to consider:

• per-transaction logs - similar to marks but with the full abilities of a log statement
• per-span logs - common in OT. but without specialized APIs and event types.

In both cases, the data should actually be small so a more minimal UI might be appropriate, handy if an embeddable logging ui component is a ways off. @formgeist is exploring the possibilities in parallel with this issue.

[sorenlouv]

per-span logs

Do we annotate log lines with transaction.id/span.id - I thought we only did this for trace.id ?

[weltenwort]

The complicated part of embedding the logs is the source configuration, i.e. which log indices to look at and which columns to show. I see two options for how to implement it:

1. We refactor the log stream (and underlying APIs) to work with a dynamically passed source configuration. This configuration would have to be passed to the embedded log stream component and via the link URL.
2. APM statically injects a (hidden) source configuration via the server-side plugin contract and references that when embedding and linking to the Logs UI. (This is what stack monitoring already does when linking.)

The former is more powerful but also requires more effort and handling of edge-cases. The latter is more restricted but requires fewer changes on the logs stream side.

[felixbarny]

per-span logs

Do we annotate log lines with transaction.id/span.id - I thought we only did this for trace.id ?

In the Java agent, we add transaction.id and trace.id. If possible, I'd like to avoid span-scoped logs (adding span.id to logs) as the performance overhead for that would be significantly higher. Given that the transaction.id and the timestamp should tell you most of the time in which span the log happened, the cost/benefit ratio seems like it's not worth it to me.

[sorenlouv]

@weltenwort The first approach definitely sounds more appealing to me since it's stateless and therefore typically less error-prone but if the other approach is significantly easier on your part we might have to go that route.

Either way, how can APM know which log indices to link to?

[sorenlouv]

Thanks @felixbarny . Not having span scoped logs sounds okay to me.

[weltenwort]

Either way, how can APM know which log indices to link to?

And how can the Logs UI know? One way out might be to rely on the new indexing strategy, from which we could derive a convention about the data type and namespace. Does the APM config have the concept of a namespace?