Make extension attributes siblings to our attributes

[duglin]

A top-level extensions bag doesn't scope the extensions to anything other than to the event itself, which means in:

{
    "cloudEventsVersion" : "0.1",
    "eventType" : "com.example.someevent",
    "eventTypeVersion" : "1.0",
    "source" : "/mycontext",
    "eventID" : "A234-1234-1234",
    "eventTime" : "2018-04-05T17:31:00Z",
    "contentType" : "text/xml",
    "data" : "<much wow=\"xml\"/>",
    "extensions" : {
        "foo" : "value"
    },
    "bar" : "value"
}

foo and bar would have the same scoping. This would serve no purpose other than to confuse people as to when to use one extension point verses the other. And what does it mean when one is used vs the other? What if both are used? What if the same name is used in both spots - which one wins? These types of questions, while solvable, add unnecessary complexity. I think its much cleaner if we just used the natural extensibility of whatever encoding/transport being used.

Signed-off-by: Doug Davis dug@us.ibm.com

[evankanderson]

FWIW, this aligns with RFC 6648 advice about not separating

Advice to protocol designers from section 4:

Designers of new application protocols that allow extensions using parameters:

1. SHOULD establish registries with potentially unlimited value-spaces, defining both permanent and provisional registries if appropriate.

2. SHOULD define simple, clear registration procedures.

3. SHOULD mandate registration of all non-private parameters, independent of the form of the parameter names.

4. SHOULD NOT prohibit parameters with an "X-" prefix or similar constructs from being registered.

5. MUST NOT stipulate that a parameter with an "X-" prefix or similar constructs needs to be understood as unstandardized.

6. MUST NOT stipulate that a parameter without an "X-" prefix or similar constructs needs to be understood as standardized.

This suggests that we should establish some sort of registry (with "simple, clear registration procedures") as well. Should that be part of this PR?

[inlined]

I'll need to think about this more. This has pros and cons for each of our representations.

For HTTP it seems that the prevailing wisdom has shifted (though I still think there are problems with header casing that we should continue to discuss in #177)

For protobuffers we'll still need a separate map of extensions due to format restrictions (only well named types will have a tag number; we'd need a map<string, google.protobuf.Any> for extensions. Any coding framework that has well-typed structs also will fields vs maps.

Unless I'm missing something, I think we're still going to have some noticeable differences between extensions and ratified fields, though this should definitely make the graduation problem simpler since there is no difference between many wire protocols.

[duglin]

During 6/15 F2F we agreed to:

• Doug: rework PR:
  Each serialization/binding spec will define exactly one way to represent extension attributes
  The main spec will say “you can add extension attributes” and not much more

[duglin]

Reworked and added a sentence about trying to avoid name conflicts.

[markpeek]

s/descripting/descriptive/ ?

[duglin]

fixed

[inlined]

LGTM with nits

[inlined]

I thought each transport binding would also indicate a default placement for extensions. Is that not the case or are you waiting for Clemens to add?

[duglin]

I was assuming Clemens would open a new PR to do that part of the work. He has an AI:

AI Clemens: Need an “escape hatch” in our binding specs to allow attributes to override the serialization rules defined.

[inlined]

It seems like we should still have some prose that says something along the lines of "additional attributes (extensions) are added as top-level keys to this document."

[duglin]

Hoping that'll be done by @clemensv's PR

[inlined]

Maybe s/possible/known|documented ?

[duglin]

I like "documented", will change

[inlined]

Nit: It's often best to avoid the word "simply" in documentation.

[duglin]

I was simply copying the old text LOL
But I removed it

[markfisher]

s/defined/define/

[duglin]

fixed - thanks

[brianneisler]

I'm a little concerned with extensions having the ability to define top level keys. This area seems like it should be reserved for keys that have been adopted by this specification and we should provide a path for promoting keys to the top level. If we allow for anyone to write a transport spec that defines how their values map to the event envelope we run the risk of quickly polluting the top level keys and leaving little room for this specification to grow without conflicting with various implementations in the ecosystem. Keeping extensions tucked away within an extensions property does not offer a solution for preventing extensions for colliding with one another but it does prevent them from colliding with the specification.

I do agree with the idea that the transport specification should be responsible for laying out how the transport maps values (like headers) to the extensions property.

[duglin]

@evankanderson I'd prefer to have the discussion about some kind of "extension registry" as part of a separate PR/issue because I suspect it'll require a lot of discussions - and might be controversial.

[duglin]

@brianneisler one of the reasons the use of "X-" prefixes on HTTP headers have been dropped is to reduce the pain involved with promotion of extensions to "spec-defined" attributes. Once the promotion happens we're in a tough spot because then people have to support both variants (X- and non-X-) and that's annoying as well as error prone, and for how long do people need to support both? So, for this reason having some mechanism in which we make extensions "different" from spec-approved metadata is now considered less than optimal.

This, of course, still leaves us with the other issue you mention... collisions. To me we can't avoid this no matter what we do. There are lots of ways this might happen:

• within an "extensions" bag
• top-level collisions because JSON itself allows for people to add top-level properties even if we did have an "extensions" bag
• http headers - with or w/o an "extensions" bag there's no guarantee there won't be collisions

In the end it feels a bit like a problem we can't solve via some kind of prefixing/bagging/etc... because we can't control (or ban) all others options - so, is trying going to cause more pain/confusion than not? I think so. Mainly because I think providing more than one way to do something is a recipe for interop issues and confusion - and that then leads me to the conclusions that we should just use whatever extension mechanism is automatically provided by the serialization/transport we're using. AND, tell people to be smart about how they name thing. If they're defining something for a particular app then prefix the name with something related to that app. If they honestly think they're defining something "for everyone" then they can choose a more generic name but they run the risk of conflicting with an official name later on, if they can't convince the standards owner to adopt it - its a gamble but it might force them to do the rigth think and bring their proposal to the spec owner rather than trying to "standardize" it through some other means.

[Vlaaaaaaad]

A project I am working on is currently exploring plans to have their Lambda functions work only on CloudEvents. As in they will have an initial entry point that wraps and packages an event nicely. Something along the lines of:

exports.myHandler = function(event, context)
{
    cloudEvent = createCloudEvent(event, context)

    result = actualFunction(cloudEvent)

    return result;
}

function actualFunction(CloudEvent ev)
{
    // ... logic here ...
}

A "packaged CloudEvent" created by the cloudEvent = createCloudEvent(event, context) would look something like:

{
    "cloudEventsVersion" : "0.1",
    "eventType" : "com.Something",
    "eventTypeVersion" : "1.0",
    "source" : "/mycontext",
    "eventID" : "A234-1234-1234",
    "eventTime" : "2018-04-05T17:31:00Z",
    "contentType" : "application/json",
    "data" : " ... actual playload here ... ",
    "application-properties": {},
    "extensions" : {
      "NAME-aws-context" : {
        // some can be missing or be empty, app logic will decide what to do
        "version" : "ef1202", // context.functionVersion
        "memoryLimitInMB" : "1024", // context.memoryLimitInMB
        "awsRequestId" : "UUID", // context.awsRequestId
        "credentials" : "", // context.identity AKA Cognito details
        "client" : {
            "version" : "1", // clientContext.client.app_version_name
            "platform" : "iOS", //clientContext.env.platform
            "platformVersion" : "11.3", // clientContext.env.platform_version
            "model" : "iPhone X", // clientContext.env.model
            "locale": "en-GB", // clientContext.env.locale
        }
      },
      "NAME-request" : {
          // extracted headers
          // request path
          // env variables and whatever else
      }
    }
}

Now, this is very early planning and the whole thing might be a bad idea. But the plan is incompatible with #260 and #225. Is the workflow described above something that is worth supporting?

---

Personally I really don't like the idea of allowing top-level attributes other than the ones defined by the spec.

The promoting fields from extensions to "spec-defined" is definitely something that needs to properly planned. In my opinion, making everything top-level is just asking for lots of conflicts.

[duglin]

Based on some conversations I had last week, I think there is some confusion around what "extensions" are (and re not). So I put this together: https://docs.google.com/document/d/1ujwLmiV16l3fT7sCn1KqXPNkKHUXq3ZwBVYYNX6ekkc/edit#
I'd like to briefly discuss this on the call this week.

[duglin]

@Vlaaaaaaad its not clear to me why #260 and #225 would be incompatible. Sure, the way createCloudEvent(event, context) is currently written doesn't align, but I'm pretty sure it could be written to do so. You didn't explain why createCloudEvent had to be written the way it is.

[Vlaaaaaaad]

@duglin : #260 makes everything a string which results in no more nested objects. The extensions.NAME-aws-context.client from above would no longer be allowed as client is an object. If #260 only affects fields( as in still allowing objects as long as all properties are stings) then yup, #260 is not affecting the example I gave.

Regarding the createCloudEvent(event, context), are you saying that it could take all the top-level properties and create a CloudEvent from that?

[duglin]

#260 could still work, its just encoding the object as a string and would need to be deserialized into JSON if that's what people want. What's interesting about your usecase is that its not clear to me that we should encourage people to treat the CloudEvent attributes as the same as "data". I envisioned people moving just the bare minimum into the CE attributes to do proper routing - but that all "real" data would go into "data". So, when I see "memoryLimitInMB" I really question why that would be at the CE level and not embedded within "data". Do we want to encourage people to duplicate all "data" into CE's properties? Or ask them to think a lot about which spot it should go? I guess I assumed almost all data would go into "data" and just routing stuff would be CE properties. Thinking about HTTP headers again... if you were to encode this CE as raw HTTP (with "data" as the HTTP Body) would you put all of that aws-context data as an http header? I'm guessing no, but I could be wrong - but if you did, I would point out that it would be a string. :-)

Regarding the createCloudEvent(event, context), are you saying that it could take all the top-level properties and create a CloudEvent from that?

Sure, why not? I can just as easily look for NAME-aws-context at the top-level as I can when its under extensions.

[matzew]

For instance, with the distributed tracing spec, fields like traceparent: xxxxxx and tracestate: yyyyyyyy would be on the top-level, but not directly mentioned out in the actual spec - only described in the Documented Extensions doc/spec, right ?

[duglin]

@matzew correct.

[Vlaaaaaaad]

@duglin: hm... I was looking at data as the place to only put the payload, not metadata too. All the other relevant metadata I was thinking of putting in extensions.

I would've say kept the payload to "this file was created" and any details about what transaction that is part of, who created that file( say Cognito details in AWS), device details( if it was a direct client action through say iOS AWS SDK), Lambda details, and so on would've been extensions.

Looks like extensions is not intended for that. I think I can easily put the stuff in data with some data.payload and data.metadata separation or whatever structure.

I'll keep exploring ideas. Probably using just 1 CloudEvent without a context object is not a good idea for what I was thinking. Has there been any discussion on how data should look like? Any examples? I imagine it's not something the WG wants to restrict, but some structure best practices would be nice.

[duglin]

@Vlaaaaaaad you certainly can use extensions for how you want to use it, by definition extensions can be anything and we can't stop people from doing so. However, and this may just be my perception of things, I always envisioned the CE properties to be more about routing than the payload/data of the event for a couple of reasons:
1 - if the real data of an event can be in both data and CE properties then how will people decide where to put the information? Will it be confusing to have data split between two locations?
2 - we know that some people want to map CE properties to HTTP headers. While people can put anything they want in HTTP headers, I suspect most people will not put things like "who created the file" in there, instead it would end up in the HTTP body. This leads me to think of the CE properties as extra info just used for things like routing - and in some cases would be duplicate info of what's in data.

I tend to think of data as "the real and complete event" and we're just wrappering it in some common envelope to get interop at the API/transport layer. But once the event has been delivered I expect the processing would have stripped off the envelope and focus on the processing of the data information. Putting some of the real data in CE properties feels a bit like we're getting into defining (yet another) common event format that all events should adhere to, and I didn't think we were doing that. We were going to let people define events however their app needs it to be defined (which means to me it needs to be complete w/o the CE wrapper) and we're just giving a thin layer on top to get API-level interop - similar to HTTP.

This entire discussion of the purpose of "CE properties" vs "data" is probably a good discussion to have, and good info to put into a primer regardless of whether we have an "extensions" bag or not.

[cathyhongzhang]

Need to clarify what is meant by "include additional extension attributes within the event"? Do you mean Event producer can include any random attributes (and give it a name the producer likes) within the top level event attributes? If so, how could the event consumers know the semantics of those attributes? How could the event consumers know how to decode the random attributes (are they strings or ints ?) and properly use them?

[duglin]

your question applies to all extensions no matter where they appear.

[cathyhongzhang]

If we put all new attributes at the top level of the CloudEvents or in one big extension bag, then no matter how descriptive the name is, different people could interpret it differently since the same name could mean different things in different context. Proper categorization/grouping of attributes seems the way to go.

[duglin]

Actually, I come the opposite conclusion. Trying to define lots of different bags feels like an exercise in futility to me. Even the one bag you've defined for "correlation" already is ambiguous (see #258 (comment)) and overlaps with CE's own properties. If we can't even define one bag that doesn't overlap then I don't see how we'll be able to create lots.

I would much prefer a simple model where we have a single location for properties and people clearly define what each property "means", not necessarily what its "used for" since how each might be used (and in how many ways it might be used) will vary in each case. Note that the same property might be used for more than one reason at the same time and I think asking people to duplicate it would be a bad UX.

[cathyhongzhang]

I feel defining lots of and thousands of random unrelated attributes in a single bag will be very confusing to the event consumer. The ambiguity you listed in #258 (comment)) applies to existing attribute eventTime and contentType etc. too.

[fsunaval]

A few of the serverless applications we are looking at require the event producer to provide information to the event gateway for easy identification. Having a separate bag called "correlationLabel" or "identificationLabel" as proposed in #258 suits our application well.
I concur that defining lots of bags defeats the purpose but that should not be the guiding principle for defining the important bags. Reliably identifying the event source using multiple key-value pairs is something that many serverless app designers will need. IMHO, this need justifies having a separate bag.

[duglin]

If the true purpose of the "correlationLabel" bag is to identify the source then the source property becomes less useful on its own and it should be moved into a sourceIdentification type of bag along with what Cathy wants. Splitting the identifying properties across two locations doesn't seem optimal.

However, this doesn't explain why these other source identifying bits must be in a bag when the semantics of those extra fields will clearly state they're for identifying the source. See "Defining, Using and Discovering Extensions" in https://docs.google.com/document/d/1ujwLmiV16l3fT7sCn1KqXPNkKHUXq3ZwBVYYNX6ekkc/edit#heading=h.9kda29ynebh5. An extension does not exist w/o a definition and its presence in a bag, or not, does not change that.

[fsunaval]

I'd say the true purpose of the bag (identificationLabel or correlationLabel) is to identify / differentiate the event within the source. In this sense, it is more of an event identifier rather than a source identifier.
In some cases (or most cases for that matter) the 'source' is probably enough. In some cases, the 'source' by itself is not enough to sufficiently identify the event.
Example: Consider a 24 * 7 home monitoring station. 'A'. It monitors events from three home security providers 'X', 'Y', and 'Z'. When a sensor sends an event to the monitoring station 'A', the 'source' field identifies the security provider ('X', 'Y' or 'Z') while the 'identificationLabel' could be used to provide information about the 'sensorId', 'subscriberId', etc.

[duglin]

Closing in favor of #277