[uss_qualifier] Add test scenario for validation of op intents

[mickmis]

This adds a test scenario for requirements related to the validation of operational intents.

[BenjaminPelletier]

Just a note: I'm all for documentation when it increases clarity, but I don't think we necessarily need sequence diagrams in every scenario. No change requested here.

[BenjaminPelletier]

I think we'll need to be careful on this one. The previous one (tiny overlap conflict) definitely works, I think, because it isn't valid for a USS to shrink the area the user wants to fly in. But, a USS might expand the area a user wants to fly in when creating the operational intent. One reason for this might be that the USS only supports a certain kind of geometry (e.g., point radius, or rectangular bounding box). Another reason might be that the USS has an agreement with the operator to be helpful and auto-adjust the op intent when the aircraft gets near an edge, so the USS needs to buffer the intended flight area somewhat to enable that functionality. So, if the USS rejects this user intent that is close-but-not-intersecting, I'm not sure we can conclude that a requirement has been violated.

[mickmis]

Mmh indeed yes. Then, since the USS can expand the volume of the intent injected, it is not actually possible to test this case, is it?

[BenjaminPelletier]

We can attempt to test for near-intersections, but we have to carefully consider whether each response is valid, and whether each response validates requirement compliance. In the case of instructing a USS to plan a flight with a user op intent that is very close to an existing op intent but not touching, if the USS does plan that flight and the shared op intent is consistent with the request (encompasses all requested area at least), then we can conclude the USS meets that requirement. If the USS declines to plan the flight, we cannot conclude they failed that requirement nor can we conclude that they meet that requirement under the set of circumstances tested. In that case, we would probably want to invent a separate test to cover that requirement.

For instance, the buffering USS could be instructed to plan a flight with no other flights in the area, which they would do successfully (after buffering the user flight intent to construct a larger shared op intent). Then, uss_qualifier could have a mock USS or other USS, which would use op intents exactly as provided by the user, plan a flight right next to the first USS. Then, the first USS could be instructed to activate their flight. Any buffer was already applied, so presumably this should succeed unless the first USS incorrectly comprehends the close-but-not-intersecting op intent as intersecting. It's possible there might be some USS implementation that would grow a user op intent more when activating than when planning, but I'm not thinking of a good reason to do that, and certainly the class of USSs that would do that is smaller than the class of USSs that would grow a user op intent when planning.

If we can't or don't come up with tests that cover every requirement for every possible USS implementation, that's ok -- when that weird USS is tested by uss_qualifier, uss_qualifier will find that it is unable to verify compliance to requirements X and Y, which means it will fail to confirm that the USS correctly implements a particular capability (e.g., strategic coordination). At that time, the USS will likely come talk to us to understand why we weren't able to verify requirements X and Y and then we can work together to figure out a test that will actually verify those requirements for their implementation, or else the USS can decide to prove compliance to those requirements outside automated testing. We want to minimize the frequency that this happens because uss_qualifier loses some credibility when it fails to validate a compliant implementation, but we certainly can't be perfect.

The most important design criterion is that we avoid declaring that a USS fails a requirement when they actually meet that requirement -- if we do that, it's a bug in uss_qualifier, and the severity of the bug depends on how unusual or unlikely the USS implementation is (failing a requirement for a plain-vanilla compliant USS implementation is a bad bug, failing a requirement for a very strange yet technically-compliant USS is a minor bug). The next most important thing is that we minimize cases where we've checked a requirement and not indicated failure when the USS actually does not meet that requirement. This latter one is more tricky because we cannot prove a negative ("there is no set of circumstances in which the USS implementation will fail to satisfy this requirement") and rather we just say that we have checked some circumstances and the USS did not fail the requirement under those circumstances. Ideally, we will try to arrange the tested circumstances to be similar to difficult real-world circumstances such that satisfying the requirement under our circumstances suggests high likelihood of satisfying the requirement in most real-world circumstances.

[mickmis]

Thank you for the thorough response, I will save it for later reference.