Adding Correlations API and removing DistributedContext

[codeboten]

Updating DistributedContext to CorrelationContext as per otep-66. Updated the following:

• added correlations API details
• removing Entry as a separate class in favour of name/value pair to align more closely with the w3c spec
• library layout, file name and README
• overview

Signed-off-by: Alex Boten aboten@lightstep.com

[dyladan]

Worth noting that the Correlation Context spec has been getting more attention in recent meetings of the W3C Distributed Tracing working group. One of the things discussed has been changing the name of the header. Currently it is Correlation-Context, but correlationcontext is being floated as the new name. Partially, this is to help with operation in contexts which have tighter restrictions on header names such as some messaging systems.

It may be worth noting in this specification, that we're basing it on the current version, but that it might change.

[mwear]

Here's the relevant issue for @dyladan's comment: w3c/baggage#13

[codeboten]

This will close #93

[c24t]

Some questions about the default context, but it looks good otherwise!

[carlosalberto]

Hey @bogdandrutu if this looks correct to you, lets definitely approve/merge, as this is the last item prior to do a 0.3.0 release (observe a related ticket was created so we don't forget about the Entry and EntryKey issue ;) )

[reyang]

LGTM.

[bogdandrutu]

I am very confused why did we write CorrelationContext with the idea that there is no "CorrelationContext" object that is part of the context, instead we have just functions/methods that change the object in the Context.

We are not consistent with the trace API where we do expose a "Span" object, and there are discussion we had about that.

[codeboten]

@bogdandrutu I added a few more details around the CorrelationContext, which make it more explicit that the CorrelationContext is a set of name/value pairs stored in the Context. The functions defined are interacting with the Context explicitly and implicitly with the CorrelationContext. Let me know if this change 975d4c5 makes this any clearer.

[bogdandrutu]

It is not clear to me why we choose to not have all the functions interact explicitly with a "CorrelationContext" object and we want this path of interacting explicitly with Context. In my opinion this case is more natural for languages that explicitly pass Context around (like Go) but not that natural for Python where for example the ContextVar library does not work very well with this API.

[tedsuo]

@bogdandrutu for an OO language, it would be fine to make CorrelationContext an interface. And when the language has an implicit active context, the context parameter is optional.

interface Correlation {
 Clear([context]) => [context]
 Get([context], key) => value
 GetAll([context]) => dictionary
 Remove([context], key) => [context]
 Set([context], key, value) => [context]
}

Does that clarify things? Or am I missing your point?

[carlosalberto]

I'd like to see CorrelationContext mentioned here (as an interface) but as a truly optional thing. I remember that for Ruby it felt better to not have it, even though CorrelationContext is handled implicitly (not explicitly like in Go).

[tedsuo]

I agree @carlosalberto, the spec should have a way of being "language neutral." The concepts, features, and requirements should be clear, but languages should be allowed to implement features using whichever language construct feels the most natural (as long as it does not violate a requirement). I think we could improve the spec on this front, but that should be separate initiative from adding the features we need for the beta launch.

[bogdandrutu]

It seems to me that I am not making myself well understood, but this API has some serious problems when it is used with an implicit (thread-local) storage and not with an explicit (manually) propagated mechanism.

Here are some reasons why we need an explicit object called Correlation exposed:

• Current API makes impossible to move one item from a Context object to another. For example in case of a batch operation if there are 5 operations batched and I want to extract the Correlation from one object I need to extract all elements in a dictionary, then iterate over them and manually add 1 by 1.
• It makes a clear separation between the Context functionality and the Correlation functionality.
• Allow users to manually propagate individual items in the Context.
• Consistency with trace where we expose the Span object.
• Consistency with languages that requires to have an object defined like Python if you want to use ContextVar API.
• To avoid creating long Context chains (if implemented using a reference to the parent instead of copying all the elements), we can build instrumentation that creates a Context object once an adds all the items (like Span + CorrelationContext).

For an experiment you can look at https://github.com/grpc/grpc-java/blob/b7ccc0d14297f255557e67a4f0364480f4a80bcf/census/src/main/java/io/grpc/census/CensusStatsModule.java#L102 to see why a TagContext was needed in order to implement the tags propagation in gRPC.

[carlosalberto]

Merging this PR as we have enough approvals, to keep the pace and do our 0.3.0 release as expected (we are already behind schedule anyway) - we will follow up on #482 as I think it's very important to follow up and solve the CorrelationContext question/issue (and to properly clarify it), and it definitely we need this for 0.4.0

[carlosalberto]

Oh, definitely a race condition between my comment/merge and your latest comment ;)

Thanks for the explanation, and specially the code example, as it will definitely help us have an even better understanding of the requirement.

Observe that the current specification doesn't restrict the usage of CorrelationContext, seeing it more as an implementation detail (that's why in Java and other languages we will have it, because it makes totally sense there). AFAIK some implementations, like Ruby, work well without it though.

I'm wondering if this is worth, overall, to prevent us from doing the 0.3.0 release. Next release is expected to happen in 1 week, so I guess we need to ponder all our alternatives here.

@open-telemetry/specs-approvers opinions?

[bogdandrutu]

My main point is that the way how this is written now will make a lot of the sigs to implement this way then they will realize that it is not good for some cases.

[bogdandrutu]

I think the best thing to do is to create an issue and move the discussion there. Maybe mark this as experimental for the moment.