Audit preparation: Verification

[aschemmel-tech]

"not started" topics:

• Testing of the embedded software: present that it is tailored out
  • See: https://eclipse-score.github.io/score/main/platform_management_plan/safety_management.html#tailoring

Actions and Deviations:

• Deviation_13: The method selection for the unit tests seems to be too theoretical or wrong. How is a control flow and data flow analysis made on Unit level? Why is requirements based testing and interface based testing not selected? The table should be reworked in accordance to the mapping of the specified test levels to the ISO 26262 test levels.
  • See test level "1" in https://eclipse-score.github.io/score/main/platform_management_plan/software_verification.html#verification-methods
  • See https://eclipse-score.github.io/score/main/platform_management_plan/software_verification.html#verification-methods
• Action_43: The mapping of the test levels to the ISO 26262-6 test levels shall be shown e.g., in the PMP.
  • see note in https://eclipse-score.github.io/score/main/platform_management_plan/software_verification.html#levels-of-integration-and-verification
• Action_44: Provide an argumentation about which tests have been selected for a feature.
  • unclear action, https://eclipse-score.github.io/process_description/main/process_areas/verification/verification_workproducts.html#wp__verification_feat_int_test describes from which input the feature tests are derived, need to clarify if we want to have a review checklist.
  • Further improvement planned within the PMP Verification plan. See below.
• Action_48: An argument shall be provided which covers the ISO 26262 requirement that all test levels are executed in a target near environment.
  • see https://eclipse-score.github.io/score/main/platform_management_plan/software_verification.html#test-execution-environment-and-reference-hardware
• Action_49: The methods of ISO 26262 shall be explained.
  • see https://eclipse-score.github.io/process_description/main/process_areas/verification/guidance/verification_methods.html
• Action_50: The Coverage target values as listed by table 37 need to described in more detail. E.g. what is meant by “Coverage of detailed/architectural design” (Sequence diagram coverage?)
  • see https://eclipse-score.github.io/score/main/platform_management_plan/software_verification.html#quality-criteria
• Action_51: It needs to be clarified where the reference system is located physically (affecting e.g. access rights).
  • planned, see Improvement: Location of reference hardware score#608
  • Not yet decided which HW to use see also: https://eclipse-score.github.io/score/main/platform_management_plan/software_verification.html#test-execution-environment-and-reference-hardware
  • https://github.com/orgs/eclipse-score/discussions/472#discussioncomment-15122092 is the start to discuss about the reference hardware
  • Follow up planned to ease closing this issue. See below. Topic cannot be concluded until audit.
• Action_53: The tool Bazel needs to be evaluated as it may not be able to see the dependencies of changes in requirements, code and tests and may skip some tests. Define the counter measurements.
  • see Tool management: Audit preparation: Tool management #81
• Action_54: The test shall have an attribute which level the test case belongs to.
  • unclear, linking to requirements is not solving this as IF tests are not linked to these.
    • the test in the link, show the trace to component requirements and also include "CIT" as "component integration Test" as additional identifier in the name of the file. This is optional, but makes life simpler. See: https://github.com/eclipse-score/persistency/blob/main/tests/python_test_cases/tests/test_cit_supported_datatypes.py#L57
    • Another example is feature integration test, where the folder is named accordingly. See: https://github.com/eclipse-score/reference_integration/blob/main/feature_integration_tests/python_test_cases/tests/basic/test_orchestartion_with_persistency.py#L23
  • Second it‘s not clear which test belongs to which level; this shall be fixed.
    • see https://eclipse-score.github.io/score/main/platform_management_plan/software_verification.html#levels-of-integration-and-verification
  • Third, the application of the test methods must be recognizable (e.g. boundary value analysis, equivalence class definition, interface-based testing, ...).
    • see https://eclipse-score.github.io/process_description/main/process_areas/verification/guidance/verification_process_toolreqs.html#gd_req__link_tests
• Deviation_19: The Verification Plan does not define the required review activities. (on Existing tests from the Open-Source Community)
  • see https://eclipse-score.github.io/score/main/platform_management_plan/software_verification.html#pre-existing-test-cases
• Action_55: The independence of the verification team is not yet described in the Verification Plan
  • see https://eclipse-score.github.io/process_description/main/process_areas/verification/guidance/verification_process_toolreqs.html#gd_req__verification_independence
• Action_56: It remains unclear how many Verification reports will be created (one per module?)
  • see https://eclipse-score.github.io/process_description/main/process_areas/verification/verification_workflows.html#wf__verification_mod_ver_report

---

Not covered as part of this Epic:

• ❌ Action_44: Provide an argumentation about which tests have been selected for a feature.
  • Should be further improved in PMP verification plan.
• ❌ Action_51: It needs to be clarified where the reference system is located physically (affecting e.g. access rights).
  Improvement: Location of reference hardware score#608
• ❌ Are (all) Tool Requirements and Dependencies addressed

Follow up in issue #489