Lack of clarity on how the trace fields are supposed to be used

[nikclayton-dfinity]

Description of the issue:

The description of the tracing fields at https://www.elastic.co/guide/en/ecs/current/ecs-tracing.html is unclear on how they are supposed to be used.

Any additional context or examples:

The ordering of the fields on the page is alphabetical, but this presents them out of order with how they are supposed to be used.

I think the hierarchy is:

A trace contains one or more transactions, a transaction contains zero or more spans (cite for "zero or more" is https://www.elastic.co/guide/en/apm/get-started/current/transactions.html).

If this is correct then the documentation for a span, which starts "Unique identifier of the span within the scope of its trace" should probably be changed to "Unique identifier of the span within the scope of its transaction".

Linking to https://www.elastic.co/guide/en/apm/get-started/current/distributed-tracing.html from the descriptions would also be helpful.

As would an explicit description of the hierarchy in the introductory information in the page.

Uniqueness constraint is unclear

Does the "unique" in that sentence mean that span IDs should aim to be universally unique, or does it mean that a span ID only has to be unique within a single transaction, so:

  trace: {id: "123"},
  transaction: {id: "456"},
  span: {id: "authorization"},

and

  trace: {id: "123"},
  transaction: {id: "789"},
  span: {id: "authorization"},

would be OK (identical span IDs, but the transaction IDs are different) ?

How these work together in a tiered architecture is unclear

For example, suppose you have:

Client -> Load Balancer -> Authorization Server
              `----------> Application Server -> Database 1 -> Filesystem 1
                                    `----------> Database 2 -> Filesystem 2

1. Client makes request which terminates at the Load Balancer.

2. The Load Balancer will perform a lookup request to the authorization server, then forward the request to the application server (this is a slightly contrived example), so the load balancer has to:

a. Create a new trace id (since this is "a user request handled by multiple inter-connected services")

b. Contact the authorization server

• Create a transaction ID ("the highest level of work measured within a service, such as a request to a server")
• Create a span ID ("an operation within a transaction, such as a request to another service")

3. The Authorization server receives the request.

What does it need to receive in order to effectively log an ECS event?

In particular, how do we correlate the event log the Authorization server is going to generate with the span for this request that the Load Balancer generated?

Does the span id that the Load Balancer generated become the trace id that the authorization server uses?

https://www.elastic.co/guide/en/apm/get-started/current/transaction-spans.html suggests that there are supposed to transaction.id and parent.id attributes, but they're not present in the schema.

[I'm generating doing this in Rust, for which there isn't an APM agent yet, I want my app to emit logs that can be seamlessly ingested in to Elastic with tracing support]

[ebeahan]

Thanks @nikclayton-dfinity for the feedback. The detailed issue write-up is greatly appreciated.

@felixbarny @axw would either of you be able to help answer the APM-related questions concerning the tracing.* fields and their appropriate usage?

[ebeahan]

The ordering of the fields on the page is alphabetical, but this presents them out of order with how they are supposed to be used.

Yes the fields in each field set are sorted alphabetical, but agree we can improve some of the supporting documentation to make usage clearer.

[axw]

@nikclayton-dfinity apologies for the lack of clarity. As you have discovered, we have only added a very small subset of tracing fields to ECS. We added these primarily to document how correlation across traces and logs should work. Excuses over, I'll try to clarify :)

I think the hierarchy is:

A trace contains one or more transactions, a transaction contains zero or more spans (cite for "zero or more" is https://www.elastic.co/guide/en/apm/get-started/current/transactions.html).

You understand correctly.

If this is correct then the documentation for a span, which starts "Unique identifier of the span within the scope of its trace" should probably be changed to "Unique identifier of the span within the scope of its transaction".

Although spans are logically scoped to transactions, span IDs must be unique within a trace. To uniquely identify any transaction or span, you must consider both trace.id and (transaction.id or span.id). A transaction can have a span as a parent.

Does the span id that the Load Balancer generated become the trace id that the authorization server uses?

No, all events related to the same original client request will share the same trace ID. The span ID corresponding to the Load Balancer's outgoing request to the Authorization Server will be used as the parent ID for the transaction recorded by the Authorization Server.

I'll illustrate what the events would look like with your example:

Client -> Load Balancer -> Authorization Server
              `----------> Application Server -> Database 1 -> Filesystem 1
                                    `----------> Database 2 -> Filesystem 2

Following your description, it sounds like Client is not instrumented so I'll generally ignore it. I'll assume the following components are instrumented: Load Balancer, Authorization Server, Application Server.

1. Load Balancer receives a request, and starts a transaction (T1). Because the Client is uninstrumented, this is the root transaction, and a new trace is started.
2. Load Balancer makes a request to Authorization Server
   i. The outgoing request is recorded as a span (T1_S1), which is a child of T1. T1_S1's span and trace IDs are injected into the HTTP request headers sent to Authorization Server.
   ii. Authorization Server receives the request, and reports it as a transaction (T2), which is a child of T1_S1.
3. Load Balancer forwards the client's request to Application Server
   i. The request is recorded as a span (T1_S2). T1_S2's span and trace IDs are injected into the HTTP request headers sent to Application Server.
   ii. Application Server receives the request, and reports it as a transaction (T3), which is a child of T2_S2.
4. Application Server performs database queries to Database 1 and Database 2. These are recorded as additional spans (T3_S1, T3_S2) which are children of T3. Databases are typically not instrumented.

So we end up with the following:

• T1: {trace.id: "trace_uuid", transaction.id: "t1"}
  • T1_S1: {trace.id: "trace_uuid", transaction.id: "t1", span.id: "t1_s1", parent.id: "t1"}
    • T2: {trace.id: "trace_uuid", transaction.id: "t2", parent.id: "t1_s1"}
  • T1_S2: {trace.id: "trace_uuid", transaction.id: "t1", span.id: "t1_s2", parent.id: "t1"}
    • T3: {trace.id: "trace_uuid", transaction.id: "t3", parent.id: "t1_s2"}
      • T3_S1: {trace.id: "trace_uuid", transaction.id: "t3", span.id: "t3_s1", parent.id: "t3"}
      • T3_S2: {trace.id: "trace_uuid", transaction.id: "t3", span.id: "t3_s2", parent.id: "t3"}

So I'll try to summarise:

• All events that were caused by the original request should share the same trace ID
• An incoming request will be reported as a transaction.
  • If it is the first (client-facing) service then it will also start a new trace.
  • If the requestor is also instrumented, and there is a corresponding outgoing span, then the transaction's parent ID should be set to that span's ID.
• An outgoing request will be reported as a span.

[ebeahan]

Thanks @axw for jumping in and helping provide clarity!

elastic.co/guide/en/apm/get-started/current/transaction-spans.html suggests that there are supposed to transaction.id and parent.id attributes, but they're not present in the schema.

transaction.id is already defined under the tracing fieldset, but correct parent.id is not currently.

@axw - parent.id isn't currently one of the defined tracing fields. Is adding it into ECS something to consider?

[axw]

Initially at least the goal of the tracing section in ECS was to explain correlation across types of data (namely traces and logs). For correlating traces and logs you would be using trace.id, transaction.id, and span.id -- but parent.id is unlikely.

For a somewhat more comprehensive explanation of the fields, https://www.elastic.co/guide/en/apm/get-started/current/transaction-spans.html is the source of truth.

I'm not against adding more of the tracing fields to ECS, but I'd want to know what problem we're solving with that.

[webmat]

Thanks for the detailed answer, @axw!

I'd want to know what problem we're solving with that

A good question. Actually I'd like to understand how users can leverage these fields in custom data sources. Since @felixbarny contributed the first two fields here, perhaps he can chime in as well.

Were these fields added purely for documentation purposes of important APM fields? Or is it possible e.g. for someone to carry on with tagging events with these identifiers in custom sources for them to bubble up in APM to help complete a trace?

If it's possible for users to do so, is there APM docs we could link to from here, to help explain the process and the possibilities?

[axw]

Were these fields added purely for documentation purposes of important APM fields? Or is it possible e.g. for someone to carry on with tagging events with these identifiers in custom sources for them to bubble up in APM to help complete a trace?

The fields we have here were added to explain how to enable trace/log correlation. i.e. if you want to correlate logs with traces, then you should include trace.id and transaction.id or span.id. For purposes of log correlation I don't think parent.id is useful.

If it's possible for users to do so, is there APM docs we could link to from here, to help explain the process and the possibilities?

We document all fields in https://www.elastic.co/guide/en/apm/server/current/exported-fields.html, however the documentation isn't likely to be very helpful for implementing an agent. For non-agent custom data sources which produce Elasticsearch docs directly, we don't have a good reference guide; but these are also exceedingly rare.

For developing an agent for Elastic APM the fields aren't particularly relevant. What matters most is the protocol between the agent and APM Server. For developing agents we have https://github.com/elastic/apm/tree/master/specs/agents, which references https://www.elastic.co/guide/en/apm/server/current/intake-api.html for the protocol.

[ebeahan]

The fields we have here were added to explain how to enable trace/log correlation. i.e. if you want to correlate logs with traces, then you should include trace.id and transaction.id or span.id.

If there's collective agreement, I'd like to adjust the current description of the tracing.* fieldset to better capture this intent.

[webmat]

Thanks for the details @axw.

For non-agent custom data sources which produce Elasticsearch docs directly, we don't have a good reference guide; but these are also exceedingly rare.

This is very helpful context. If it's not a first class workflow, we can say so.

[axw]

If there's collective agreement, I'd like to adjust the current description of the tracing.* fieldset to better capture this intent.

@ebeahan sounds good. Do you have something in mind already? Maybe some kind of disclaimer at the top that this documentation is intended for log correlation, with pointers to the other docs for other use cases?

[webmat]

We actually have some flexibility to document this in ECS, if needed.

Since #988, a given field set can have a whole free form documentation page that accompanies it. So we can do more than just a quick warning. We could go into some details in free form asciidoc, and yes of course defer and link to APM docs as well (we don't want to repeat everything).

If you want to see an example, this draft PR #1066 is the first to add such a docs page, in this case to the "user" fields. You can see the "usage" subsection in the sidebar, and there's also a call out to it at the top of the normal "user page".

Normal user page (PR)

Usage docs for user (PR)