PASS vs OK&PASS for AnyOf&AllOf

[ArnoldTchegou]

Hi all,

I have some concerns regarding how results are handled in the case of AnyOf and AllOf evaluations:

• AnyOf Evaluation: When the service level requirement passes, instances whose property values contribute to making the test pass are marked as PASS, while those that do not contribute are marked as OK.

• AllOf Evaluation: When the service level requirement passes, all instances (which do not individually contribute to passing the test) are marked as OK.

According to the spec, "The AnyOf and AllOf comparison types can be met using any resource instance within the scope of the schema-level requirement." Based on this, I believe that in both cases (i.e., for both AnyOf and AllOf evaluations), if the service level passes, all instances should have their result marked as PASS, since the requirement is met at the service level regardless of individual contributions.

Would you consider the new idea? if its the case, i am ready to make a pull request.

Reference for AnyOf

Reference for AllOf

[mraineri]

I like that idea; it helps break out which specific instances are contributing to the service-level PASS criteria. I'll ask folks their opinion on the next call.

[mraineri]

For the AllOf output, I noticed the result shows "OK", even though the value was in the AllOf list. Shouldn't that show "PASS" instead since it's contributing to the passing criteria of the test?

[ArnoldTchegou]

Hi mraineri,

Thank you for your reply.

In the validator's current state, the result of each instance for AllOf (regardless of whether an instance contributes or not) will be marked as OK when the requirement/condition passes at the service level.

I have a few concerns about how the results should be displayed at the instance level. Please see the table below for my suggestions:

Service Level	Instance's Contribution	AnyOf (current)	AnyOf (suggested)	AllOf (current)	AllOf (suggested)
PASS	contributes and makes pass individually	PASS		PASS
PASS	contributes and does not make pass individually			OK
PASS	Does not contribute	OK	NO PASS	OK	NO PASS
FAIL	contributes			NO PASS	OK
FAIL	Does not contribute	FAIL	NO PASS	FAIL	NO PASS

I believe the item in bold is the most likely candidate for change compared to the others.

[mraineri]

Here's our recommendation:

Service Level	Instance's Contribution	AnyOf (current)	AnyOf (suggested)	AllOf (current)	AllOf (suggested)
PASS	contributes and makes pass individually	PASS	PASS	PASS	PASS
PASS	contributes and does not make pass individually	<Not possible for AnyOf>	<Not possible for AnyOf>	OK	PASS
PASS	Does not contribute	OK	OK	OK	OK
FAIL	contributes	<Not possible for AnyOf>	<Not possible for AnyOf>	PASS	PASS
FAIL	Does not contribute	FAIL	NO PASS	FAIL	OK

A few things to note for our reasoning with the table:

We don't think showing "FAIL" in the resource's report is appropriate for any of the cases (just show FAIL at the service-level). The reason for this is each resource on its own is allowed to have a value that does not contribute to the overall service-level requirement. Someone seeing a "FAIL" here would get the impression they need to fix that exact resource instance.

We also think showing "PASS" for any value that contributes to the overall service-level requirement is good because it highlights the fact that the issue is not with that resource in question. The reviewer will know to look elsewhere in the service for addressing the issue.

[FHomps]

Hi Mike,

Good points and we agree with your logic. Most important to us is coherence between the different cases, not so much the actual values.
I think you might have made a few typos in your table? Here are our remarks:

Are we OK to start working on a pull request to implement the following (changes in bold):

Service Level	Instance's Contribution	AnyOf (current)	AnyOf (suggested)	AllOf (current)	AllOf (suggested)
PASS	contributes and makes pass individually	PASS	PASS	PASS	PASS
PASS	contributes and does not make pass individually	<N/A>	<N/A>	OK	PASS
PASS	does not contribute	OK	OK	OK	OK
FAIL	contributes	<N/A>	<N/A>	NO PASS	PASS
FAIL	does not contribute	FAIL	OK	FAIL	OK

Best,
François

[mraineri]

You're right on the last row for "AnyOf (suggested)"; that should be OK.

For the fourth row for "AllOf (current)", we did not notice a "NO PASS" result... So we were a bit confused about that specifically... At least from the code today it looks like it's being marked as PASS....

[mraineri]

Closing; PR merged