-
Notifications
You must be signed in to change notification settings - Fork 244
Description
Is your feature request related to a problem? Please describe.
There are a number of similar situations for which the "N/A" (aka UTASSERT_CASETYPE_NA) log type is used, but have different nuances:
- When a certain functionality is not implemented at all (e.g. network tests on a system without network stack, the API returns
OS_ERR_NOT_IMPLEMENTED). In this case a single "N/A" entry is used and the whole group of tests is skipped. - When a certain functionality is implemented, but cannot be tested due to external factors, such as the state of the system, how it was booted, etc. (e.g. in the "CDS" test in ES, when calling CFE_ES_RegisterCDS(), if the CDS memory was not cleared before the test, the result may be "CFE_ES_CDS_ALREADY_EXISTS"). In this case the "N/A" entry is used to advise the user that the test is incomplete due these external factors, and that the user should take action to correct this and get a complete test result.
- When there is more than one possible acceptable result from a specific function call. In this case, the test code checks for each acceptable result, but are checked with
UTASSERT_CASETYPE_NAto indicate that although this value was checked and does not match, it does not constitute a failure of the test (the test can go on to check for the other possible acceptable results).
Describe the solution you'd like
These three different "N/A" nuances should ideally have a different case type. (specific words/abbreviations can be discussed). As the traditional meaning of "N/A" is "Not applicable", it likely line up best with the use case (1) above, where a test is truly not applicable on that platform/setup.
For the other two, a separate casetype should be added to better convey the intent:
For item (2) above is to warn the user that the results are incomplete due to external factors, and they need to take action to correct those external factors to get a complete result. This shouldn't necessarily be a failure; the test can still succeed in reduced form, but the report should be very clear that tests were skipped or not complete, and the user needs to take manual action to correct it.
For item (3) above, the intent is allow a "soft" test - where multiple values are acceptable, it is necessary to "PASS" (and log it) if a value does match the acceptable value, but not fail if the value does not match - because there are other acceptable values. Typically these checks wouldn't need to go in the log at all; they may only be of interest to developers implementing the test, it does not provide much value in a final report log to see values that were checked for but didn't match. (In that sense, visibility of these tests should be similar to DEBUG, but we should not overload DEBUG either).
Describe alternatives you've considered
Leave all these case types as N/A
Additional context
Existing overload of N/A is not horribly broken/wrong, but it makes the logs a little harder to process, and not as clear as to what action (if any) the user needs to take for the N/A reports.
Requester Info
Joseph Hickey, Vantage Systems, Inc.