Split collection limit out of cardinality limit

[MrAlias]

As discussed in the last specification SIG meeting (2024-01-09) the existing cardinality limit is being used to represent two different limits:

1. The maximum number of measurements made for distinct attribute sets within a collection cycle
2. The maximum number of time-series exported at the end of the collection cycle

This distinction is meaningful for a few reasons.

• Users may not want these to be the same value. One applies to the system telemetry is being produced on, and the other applies to the downstream telemetry transmission and storage systems.
• The produced telemetry will differ base on how this limit is applied in relation to any attribute filtering.
• A user trying to remediate the limit (1) being exceeded using an attribute filter may not be able to if the implementation is filtering at the end of the collection cycle.

Proposal

• Introduce the new "collection limit" to directly set the maximum number of measurements allowed for distinct attribute sets within a collection cycle (1)
• Include recommendations for implementations to document that users should resolve "collection limit" scenarios using the instrument attribute advisory parameter
• Refine the definition of "cardinality limit" to only be the maximum number of time-series exported at the end of the collection cycle (2)
• Include recommendations for implementations to document that users should resolve "cardinality limit" scenarios using the instrument attribute advisory parameter or with an attribute filter on a view.

cc @trask @jmacd @jack-berg @jsuereth

[jmacd]

I think I agree with this proposal. The Lightstep metrics SDK which I used for prototyping does have two limits that can be roughly described as @MrAlias has described above.

We might disagree on what "within a collection cycle" means. In my implementation, this "interior" cardinality limit is enforced between any two collection cycles by any two Readers. So -- and I admit this is not very intuitive -- when the interior limit is being reached, one way to address this is for the user to add another Reader with a shorter collection cycle. This will push the cardinality out of the interior data structure into each reader sooner, at which point the per-reader collection limit is well defined.

@utpilla looking for input.

[jack-berg]

Introduce the new "collection limit" to directly set the maximum number of measurements allowed for distinct attribute sets within a collection cycle (1)

If I understand this correctly, this is saying for a given attribute set (i.e. {"shape": "square", "color":"red"}), limit the max number of measurements to some configurable value. Who is asking for this? What's the goal of this?

[MrAlias]

Who is asking for this?

@MrAlias

What's the goal of this?

Is there something unclear with the description?

[trask]

we may want to revisit this issue after #3798 is resolved since seems to be some interdependency between them

[cijothomas]

Introduce the new "collection limit" to directly set the maximum number of measurements allowed for distinct attribute sets within a collection cycle (1)

If I understand this correctly, this is saying for a given attribute set (i.e. {"shape": "square", "color":"red"}), limit the max number of measurements to some configurable value. Who is asking for this? What's the goal of this?

+1

I am confused too... Why do we want to limit the number of measurements allowed for this?

for i=0; i<100; i++)
{
 counter.add(1, shape=square,color=red);
}

// Are we saying we need to have a limit on the number of measurements allowed ? i.e if I have 100 measurements, and limit is 90, then what happens to the other 10 measurements? What is the need of this limiting? Isn't the whole point of Metrics is that output is of fixed, predictable size, so if user has 100 measurements or a million of them, the output is still predictable size.

It may be the case that we didn't understand each other. @MrAlias Can you clarify if my (and Jack's) understanding is correct?

Also #3856 has clarified that the cardinality limit in the spec today is 2 from this issue.
"Refine the definition of "cardinality limit" to only be the maximum number of time-series exported at the end of the collection cycle (2)"

[MrAlias]

I have 100 measurements, and limit is 90

Note from the description:

The maximum number of measurements made for distinct attribute sets within a collection cycle

Which means that if you made 100 measurements for distinct attribute sets, yes you would limit to 90. If you make 100 measurements for the same attribute set you would measure all 100.

This assumes filtering is done in the collection phase.