Size Constraints #405

clemensv · 2019-03-15T16:38:35Z

This is an alternate proposal for #395 and #364 which states a simple 64 KByte "should" wire-size limit for publishers and for consumers along with guidance to refer to large data items instead of including them,

Signed-off-by: Clemens Vasters clemensv@microsoft.com

rperelma · 2019-03-15T19:34:31Z

I vastly prefer this approach to the others. 👍

JemDay

I think everybody will prefer the simplicity of this kind of statement but i think we ended up going down the other rabbit-holes because we thought more specificity was needed.

Whatever we do i think it'd help to have a compliance test, maybe leveraging our SDKs, that can be used to assert the 64K boundary condition.

JemDay · 2019-03-15T20:10:23Z

spec.md

+encoding for an event used by the application SHOULD be considered for
+compliance with this size constraint.
+
+A consumer or intermediaries MAY refuse to accept events that exceed an


s/intermediaries/intermediary/

JemDay · 2019-03-15T20:23:29Z

spec.md

+of events, because the referenced external data items allow for differentiated
+access control to sensitive details.
+
+The overall wire-size of a published event SHOULD NOT exceed 64 KByte. This


Can we re-word this more along the lines of _"... CloudEvent intermediaries MUST be capable of transporting and delivering CloudEvents with a wire-size of 64 KByte.." - we're not looking to limit the size and i believe SHOULD NOT is a bit of a strong statement.

cneijenhuis

A personal note first: I am sorry that the following dismissal sounds rather harsh. I have invested significant amount of time in trying to build a consensus in the workgroup, also based on feedback from Clemens. I think I have made it clear that I'm willing to compromise on virtually anything, except for a guarantee of interoperability for event producers. #395 is based on the feedback of the workgroup over several months and has significantly changed compared to my original PR.

I wish we could have resolved this in the previous workgroup meetings, and I am also more than willing to take back my PR if Clemens proposal would guarantee interoperabilty.

This proposal does not ensure interoperability between different encodings and transport formats: An event producer may not know which encodings and transports will be used for their events, and they may not implement those.

This proposal is simpler to implement for those that want to limit the wire-size to exactly 64KB. The only middleware that qualifies is Azure EventGrid. Every other tech stack we looked at (see #257) has higher limits (e.g. 256KB or 1MB).

Additionally, this proposal goes against a previous decision by the workgroup against a size constraint.

cneijenhuis · 2019-03-20T09:59:04Z

spec.md

@@ -288,6 +289,31 @@ encapsulated within the `data` attribute.
 * Constraints:
  * OPTIONAL

+# Size Constraints


As I have pointed out on last weeks call, the workgroup already discussed whether we should have a size constraint, or a size guarantee. On Jan 17, the workgroup decided that we want to go with a size guarantee, and not with a constraint.

If you don't tryst my comments in the PR, feel free to listen to the recording of the call.

One of the reasons is that 64 KB is the lowest common denominator due to Azure EventGrid. All other tech stacks we looked at supported higher wire sizes, and members of the workgroup still want to use larger events if they know their tech stack supports it without violating a SHOULD.

Perhaps Minimum Supported Size or CloudEvent Size?

Suggested change

# Size Constraints

# Size Guarantee

The suggestion above is still valid.

cneijenhuis · 2019-03-20T11:31:31Z

spec.md

+If an application configuration requires for events to be routed across different
+transports or for events to be re-encoded, the least efficient transport and
+encoding for an event used by the application SHOULD be considered for
+compliance with this size constraint.


Practically, if the least efficient transport is:

JSON, the event producer is exactly in the same scenario as in Encoding & Transport Independent Interoperability Guarantee (JSON-based Size Measurement) #364

something else, the event producer SHOULD know the size in that transport/encoding.

The first case was not acceptable to you, and you have requested that #364 is changed to not force event producers to serialize events to JSON if they are using a different encoding. I am wondering what suddenly changed your mind?

The second case is significantly worse for the event producer. JSON must be implemented, so the event producer can easily measure the size. For anything else, there is no guarantee the event producer has it implemented.

My main motivation for this whole effort is to enable event producers that will be part of a huge number of applications and tech stacks (examples: GitHub, PayPal, Twilio, ...) to be confident their events will be accepted by any event consumer, independently of what is used on the wire. This proposal does not achieve it.

duglin · 2019-03-22T01:43:49Z

Please see: https://lists.cncf.io/g/cncf-cloudevents/topic/vote_how_to_deal_with_sizes/30703703?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,30703703
for today's vote on this topic.

erikerikson

nits

erikerikson · 2019-03-22T16:45:21Z

spec.md

@@ -288,6 +289,31 @@ encapsulated within the `data` attribute.
 * Constraints:
  * OPTIONAL

+# Size Constraints


Perhaps Minimum Supported Size or CloudEvent Size?

erikerikson · 2019-03-22T16:54:32Z

spec.md

@@ -288,6 +289,31 @@ encapsulated within the `data` attribute.
 * Constraints:
  * OPTIONAL

+# Size Constraints
+


It might be nice to start with the requirements in a compact/clear declaration followed by the explainer text. Seek time is high as written.

Perhaps

The size of an event is its wire-size. Wire-size includes every bit that is transmitted on the wire: transport frame-metadata, event metadata, and event data. * Consumers MUST accept events up to 64KM in size. * Consumers MAY reject any event over 64KB in size.

Perhaps MUST should be SHOULD to allow for compliant low memory footprint consumers but my expectation is that such circumstances would be fairly narrow and would reduce the value of this section.

erikerikson · 2019-03-22T16:57:06Z

spec.md

+
+The overall wire-size of a published event SHOULD NOT exceed 64 KByte. This
+includes the frame-metadata of the chosen transport and all event metadata and
+event data, regardless of encoding. 


nit: s/the frame-metadata of the chosen transport and all event metadata and event data/the frame-metadata of the chosen transport, all event metadata, and event data/

duglin · 2019-03-22T19:46:14Z

rebase needed

cneijenhuis

As discussed

cneijenhuis · 2019-04-11T16:56:44Z

spec.md

+of events, because the referenced external data items allow for differentiated
+access control to sensitive details.
+
+The overall wire-size of a published event SHOULD NOT exceed 64 KByte. This


Suggested change

The overall wire-size of a published event SHOULD NOT exceed 64 KByte. This

CloudEvent intermediaries MUST be capable of transporting and delivering CloudEvents with a wire-size of 64 KByte.

(Jems suggestion)

Suggested change

The overall wire-size of a published event SHOULD NOT exceed 64 KByte. This

All CloudEvent consumers SHOULD accept events with an overall wire-size of 64 KByte.

Sounds similar to my PR.

cneijenhuis · 2019-04-11T16:57:12Z

spec.md

@@ -288,6 +289,31 @@ encapsulated within the `data` attribute.
 * Constraints:
  * OPTIONAL

+# Size Constraints


Suggested change

# Size Constraints

# Size Guarantee

duglin · 2019-05-02T14:39:30Z

@clemensv rebase needed, and I think you were going to make some editoral tweaks.

duglin · 2019-05-09T01:09:08Z

I think the rebase did some funky stuff -for example, the new version number is reverted and the new "required attributes" header section is gone.

duglin · 2019-05-10T02:12:09Z

ping @clemensv to check the rebase - see the previous comment

duglin · 2019-05-15T13:15:05Z

Ignoring the rebase needed, and just focusing on the text in the "Size Constraints" section - this LGTM and is aligned with what the group agreed to in a previous call.

cneijenhuis

@duglin Asked me to make my usual comments for the third time, because on last weeks call, multiple people found the constraint-enforcement (or kinda-enforcement via SHOULD) odd.

As discussed previously, these issues don't arise if we focus on a size guarantee from intermediaries.

cneijenhuis · 2019-05-17T15:52:40Z

spec.md

@@ -288,6 +289,31 @@ encapsulated within the `data` attribute.
 * Constraints:
  * OPTIONAL

+# Size Constraints


The suggestion above is still valid.

cneijenhuis · 2019-05-17T15:53:47Z

spec.md

+transport and encoding used by the application SHOULD be considered for
+compliance with these size constraints:
+
+- Publishers SHOULD NOT publish events exceeding a size of 64 KByte.


This line should be removed.

duglin · 2019-05-27T11:21:17Z

@clemensv rebase needed and there are some outstanding questions from @cneijenhuis

Signed-off-by: Clemens Vasters <clemensv@microsoft.com>

clemensv · 2019-06-04T13:12:32Z

I redid the commit because I messed up that branch. To address @cneijenhuis' comments, I now call the section "Size Limits" (I find "Guarantees" too strong) and I also removed the statement about the Publisher as suggested. What's left is that intermediaries MUST forward 64KB and that consumers SHOULD be able to deal with 64KB.

erikerikson

I appreciate the intent of introducing a distinction between intermediaries and consumers here. It seems that you are introducing a new dimension that could and perhaps should impact the whole specification. Doing so in the scope of this PR rather than as a separate and comprehensive PR doesn't seem like the right approach. Otherwise, I offer a few nits and point out language where it doesn't appear you've written what you intend.

erikerikson · 2019-06-05T17:02:00Z

spec.md

+transport and encoding used by the application SHOULD be considered for
+compliance with these size constraints:
+
+- Intermediaries MUST forward events of a size of at least 64 KByte.


Although I believe I understand your intent, this technically only creates a requirement for events of "at least 64 KByte" or in other words, events with a size that is 64KB or more. As such, as written it doesn't discuss events of the smaller size but worse, it specifies that all events 64KB or larger MUST be forwarded by intermediaries.

@erikerikson can you suggest some alternate wording?

My original recommendation was offered in a previous [comment] (#405 (comment)) but new framing has been developed which is good. I'll attempt to make a suggestion in the next few days.

.... of a size of 64 KBytes or less.

erikerikson · 2019-06-05T17:03:30Z

spec.md

+embedded devices, that are storage or memory-constrained and therefore
+would struggle with large singular events.
+
+The "size of an event" is it's wire-size, and includes every bit that is


NIT: I believe size is the term here. I think only that should be quoted.

erikerikson · 2019-06-05T17:12:45Z

spec.md

+- Intermediaries MUST forward events of a size of at least 64 KByte.
+- Consumers SHOULD accept events of a size of at least 64 KByte.
+
+In effect, these rules will allow events of up to 64 KByte to be published


Consider In effect, these rules will allow producers to publish events up to 64KB in size safely. Safely here means that it is generally reasonable to expect the event to be accepted and retransmitted by all intermediaries.

clemensv · 2019-06-06T07:07:09Z

@erikerikson

"Intermediary" and "consumer" are terms explicitly defined in the spec in the roles that I use here, so this isn't adding anything new.
"At least 64KByte" sets a lower boundary for what an intermediary MUST be able to handle, and that is the exact intent. An intermediary is conformant if it allows for exactly 64KByte, because that satisfies "at least".
Yes, you are correct about only "size" needing quotes and there's also an it's/its mixup. Will fix that.
I'll merge your proposal, while leaving the consumer statement.

Signed-off-by: Clemens Vasters <clemensv@microsoft.com>

duglin · 2019-06-06T17:53:20Z

Approved on the 6/6 call with the minor wording change in the latest commit ( b723e49 ).

JemDay reviewed Mar 15, 2019

View reviewed changes

cneijenhuis reviewed Mar 21, 2019

View reviewed changes

erikerikson approved these changes Mar 22, 2019

View reviewed changes

cneijenhuis reviewed Apr 11, 2019

View reviewed changes

clemensv force-pushed the event-size-quota branch from a1f7de4 to 24ab19b Compare April 12, 2019 20:00

clemensv force-pushed the event-size-quota branch from 24ab19b to 740be10 Compare May 8, 2019 13:44

cneijenhuis reviewed May 17, 2019

View reviewed changes

duglin mentioned this pull request May 30, 2019

Should we define errors for size limits exceeded? #433

Closed

duglin added needs work v0.3 labels May 30, 2019

added size limits

fc7b84e

Signed-off-by: Clemens Vasters <clemensv@microsoft.com>

clemensv force-pushed the event-size-quota branch from 740be10 to fc7b84e Compare June 4, 2019 13:09

duglin removed the needs work label Jun 4, 2019

erikerikson reviewed Jun 5, 2019

View reviewed changes

Clemens Vasters added 2 commits June 6, 2019 09:09

addressing @erikerikson feedback

af81b44

Signed-off-by: Clemens Vasters <clemensv@microsoft.com>

project call feedback

b723e49

Signed-off-by: Clemens Vasters <clemensv@microsoft.com>

duglin merged commit 9d54ab2 into cloudevents:master Jun 6, 2019

cneijenhuis mentioned this pull request Jun 7, 2019

Dealing with Message and Request Size Limits #257

Closed

ratankr39 mentioned this pull request Dec 20, 2022

Event-data spec when data is too large, so truncated/pruned #1136

Closed

	The overall wire-size of a published event SHOULD NOT exceed 64 KByte. This
	CloudEvent intermediaries MUST be capable of transporting and delivering CloudEvents with a wire-size of 64 KByte.

	The overall wire-size of a published event SHOULD NOT exceed 64 KByte. This
	All CloudEvent consumers SHOULD accept events with an overall wire-size of 64 KByte.

Size Constraints #405

Size Constraints #405

Conversation

clemensv commented Mar 15, 2019

rperelma commented Mar 15, 2019

JemDay left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cneijenhuis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

duglin commented Mar 22, 2019

erikerikson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

duglin commented Mar 22, 2019

cneijenhuis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

duglin commented May 2, 2019

duglin commented May 9, 2019

duglin commented May 10, 2019

duglin commented May 15, 2019

cneijenhuis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

duglin commented May 27, 2019

clemensv commented Jun 4, 2019

erikerikson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clemensv commented Jun 6, 2019

duglin commented Jun 6, 2019