APM/Log correlation: add span ID and trace ID

[felixbarny]

In order to correlate APM data with logs, we need to specify stable identifiers for the span ID and trace ID.

Note that for distributed tracing there are some changes in the data model. Most notably, we have the concept of a trace. The trace ID will be propagated to each service which participates in a "distributed transaction" aka trace. The format is a hex encoded 128 bit byte array string, for example cd03e525c1c801cc666099d5c2108e4e

Some suggestions for the field name:

• trace.id
• trace_id
• event.trace_id

We still have a separation of transaction and span, although, with distributed tracing, the semantics of those are a bit blurred. Both have an id and a traceId A transaction is the first event in a particular service, also known as the entry point (or sometimes entry span). A span is a unit of work, like a DB request, within a transaction.

Open question: do we want different field names for the transaction and span ids?

I'd suggest just using the already defined id field both for transaction ids and span ids. This makes it future-proof if we decide we want to merge the transaction and span concepts.

The format is a hex encoded 64 bit byte array string, for example dbebf61570c4dd6e.

However, id seems to be quite a generic name. It also does not seem to be namespaced like apm.id right? Couldn't that lead to ambiguities?

relates to #96
/cc @roncohen @ruflin

[ruflin]

++ of having tracing specific fields in ECS. The id you linked from the apm use case has become event.id: https://github.com/elastic/ecs#-event-fields It's definitively possible to use this for your use case above.

At the same time I think it would not hurt being more specific on what the id actually is to make it more straight forward for users to query. What are the queries you expect users to run for it?

I like trace.id.

[felixbarny]

I guess one of the biggest use cases would be to correlate application logs with traces.

Users should be able to answer questions like, which lines have been logged across all services which took part in a particular distributed trace and which lines were logged within a particular span.

Also, the other way should be possible. For a particular log line (for example an exception log), what is the corresponding transaction. This lets users find out which HTTP request parameters were transmitted, which might have been the cause for the exception.

For spans and transactions IDs, which field names would you consider idiomatic for ECS?

• span.id
• transaction.id
• id
• event.id

/cc @simitt

[ruflin]

I would exclude id and event.id as I would expect them to uniquely describe a single event. span.id and transaction.id seem from my perspective to fit well into ECS. I would assume there is more information around a span and transaction that potentially can make it into ECS?

One thing I'm asking myself is if there should be a prefix before span and transaction to have them under one prefix. It would be something like apm.transaction.id. At the same time I think transaction.id could also be valuable in other cases so we should not tie it to APM only.