CloudEvent Tracing Compliance

[travis-minke-sap]

The Knative Channel Specification specifies the expected behavior of handling W3C Tracecontext headers. The current implementation does NOT meet these requirements in that the traceparent and tracestate headers are lost in transmission. A completely new traceparent value is instead sent to the subscriber.

Currently, the Knative In-Memory Channel reference implementation DOES adhere to the specification, but the eventing-contrib/kafka implementation does not. Similar to this implementation the latter has the intermediate step of persisting to/from a Kafka Topic. There is currently confusion and a debate around the best way to handle this persistence, how it relates to the potential use of ce-traceparent and ce-tracestate, and where in the Knative / CloudEvent stack to handle this. Further, it is unclear whether the CloudEvents DTE Extension is useful or not.

See the following Github Issues/PRs for the history...

Knative / CloudEvents
knative/eventing#2918 & cloudevents/spec#603 & cloudevents/spec#607
eventing/in-memory
knative/eventing#1757
eventing-contrib/kafka
knative/eventing-contrib#1139 & knative/eventing-contrib#1155
knative/eventing-contrib#563
Previous Knative Implementation Removal
knative/eventing#2873
Therefore, we are going to wait until this is resolved before attempting to implement a solution that complies with the expected specification.

There are also larger tracing issue to be dealt with within the Knative Eventing ecosystem such as how to integrate Istio B3 Header usage with the current OpenTelemetry Headers. See knative/eventing#2581.

W3C Tracecontext Notes
The traceparent field is composed of the following values...

- - -

...where the (usually 00) specifies the format of the following data fields, and the is the end-to-end ID of the entire trace through a distributed system. The is also referred to as the , and can/should change with each new HTTP Client Request being made. Therefore the expectation is that when the Channel receives an HTTP request with a traceparent header, that it is forwarded to the Subscriber with the same traceparent header, albeit with a new .

[eric-sap]

The tracing has been enabled for the channel and dispatcher (via #30) by calling tracing.SetupDynamicPublishing() inside an InitializeTracing() function called from their respective main() functions.

This currently results in traces that have a different TraceID for the channel and dispatcher tracing spans, which might be a bit confusing when looking through the results (zipkin, etc) for what is a single event. Fixing the Tracecontext headers may alleviate this.

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[travis-minke-sap]

/remove-lifecycle stale

[eric-sap]

PR #140 contains a fix that may have fixed this issue, but a little more research is necessary to say that for certain. In particular, PR #140 fixes the conformance tests for the distributed channel, which verify a tracing tree that was not correct without the traceparent header, but that may not be the only thing required to resolve this issue.

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[matzew]

@eric-sap @travis-minke-sap @aliok is this something that we should also do on the "Code unification" ?

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[travis-minke-sap]

/remove-lifecycle stale

This issue has come up again for us related to istio-mesh tracing throughout - we might pursue forwarding istio/zipkin headers in the near future...