Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Guideline

This guideline outlines the responsibilities and procedures for developers performing
verification activities (testcase creation, inspection, and review) for documentation,
Rust and C/C++ elements of the platform and its tooling.
code (like Rust and C/C++) elements of the project/platform and its tooling.

Note that rust, python and gTest are used for test case creation.

Expand Down Expand Up @@ -71,7 +71,7 @@ Simplified in a nutshell:
#. Implement test case
#. Link test case to requirements and specify metatags
#. Confirm requirement test coverage by creating linkage document
#. Set requirement attribute [testcovered=YES] during Sphinx build
#. Set requirement attribute [testcovered=YES] during software build

More information on the concept of requirements verification can be found in :ref:`requirement_verification_workflow`

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ Process Requirements


More information can be found in the :need:`gd_guidl__verification_guide`, :need:`doc_concept__verification_process`,
and :need:`gd_guidl__verification_specification`.
and :need:`gd_guidl__verification_specification` .

.. gd_req:: Linking Requirements to Tests (C++)
:id: gd_req__verification_link_tests_cpp
Expand All @@ -57,30 +57,12 @@ Process Requirements
:complies: std_req__iso26262__support_6432


For linking C++ test suites to requirements **record properties** shall be used. Attributes
For linking C++ tests to requirements **record properties** shall be used. Attributes
which are common for all test cases can be specified in the Setup Function (SetUp()), the other
attributes which are specific for each test case need to be specified within the test case:

.. code-block:: cpp
A more detailed description of how to link code to requirements looks like is available in the :ref:`verification_template_cpp`

class TestSuite : public ::testing::Test{
public:
void SetUp() override
{
RecordProperty("TestType", "<TestType>");
RecordProperty("DerivationTechnique", "<DerivationTechnique>");
...
}
};

TEST_F(TestSuite, <Test Case>)
{
RecordProperty("PartiallyVerifies", "ID_2, ID_3, ...");
RecordProperty("FullyVerifies", "ID_4, ID_5, ...");
RecordProperty("Description", "<Description>");

ASSERT ...
}

.. gd_req:: Linking Requirements to Tests (Python)
:id: gd_req__verification_link_tests_python
Expand All @@ -101,17 +83,8 @@ Process Requirements
For allowed values for test_type & derivation_technique please check :need:`gd_req__verification_link_tests`
Further more, this decorator will also check if your test has a `docstring` which should act as the description of the test.

A more detailed description of how to link code to requirements looks like is available in the :ref:`verification_template_python`

.. code-block:: python

@add_test_properties(
partially_verifies=["tool_req__docs_dd_link_source_code_link"],
test_type="requirements-based",
derivation_technique="requirements-analysis",
)
def test_group_by_need_empty_list():
"""Test grouping empty list of needlinks."""
...

.. gd_req:: Linking Requirements to Tests (Rust)
:id: gd_req__verification_link_tests_rust
Expand All @@ -122,19 +95,7 @@ Process Requirements

For linking Rust tests to requirements **#[record_property]** shall be used:

.. code-block:: rust

use test_properties::record_property;

#[record_property("PartiallyVerifies", "ID_2, ID_3, ...")]
#[record_property("FullyVerifies", "ID_4, ID_5, ...")]
#[record_property("Description", "<Description>")]
#[record_property("TestType", "<TestType>")]
#[record_property("DerivationTechnique", "<DerivationTechnique>")]
#[test]
fn test_case_function() {
...
}
A more detailed description of how to link code to requirements looks like is available in the :ref:`verification_template_rust`

.. gd_req:: Independence
:id: gd_req__verification_independence
Expand All @@ -153,7 +114,7 @@ Process Requirements
:complies:

The tool automation shall automatically generate the Verification reports.
These may be independent documents (i.e. not integrated into sphinx documentation).
These may be independent documents (i.e. not integrated into docs-as-code based repositories).
The content of the reports is specified in :need:`gd_temp__platform_ver_report` and :need:`gd_temp__mod_ver_report`.

.. gd_req:: Verification Report Archiving
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ A test specification contains the following attributes.
- Description
- Helpful links
* - Partially and/or Fully Verifies
- Sphinx-needs Id(s)
- Docs-as-code Id(s) for traceable elements
- Links to one or more requirement Ids. Can be partial or full coverage of a
requirement, as per test property.
-
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,49 @@
Verification Templates
======================

gTest
-----
The sections below are seen as typical ways when writing tests and their specification.
Their usage differs based on the selected testing framework and the implementation language of the module(s).

gTest is the preferred and accepted test framework to write test cases for C/C++ code in the project.
C++
---

For C++ code gTest is a commonly used and accepted test framework to write test cases for C/C++ code.

Each test case requires a link to one or more requirement/design element.
A more detailed description of how to link code to requirements is available here: :need:`gd_req__verification_link_tests_cpp`

Writing test cases using gTest shall follow the recommendations from the official gTest documentation.
For linking C++ tests to requirements **record properties** shall be used. Attributes
which are common for all test cases can be specified in the Setup Function (SetUp()), the other
attributes which are specific for each test case need to be specified within the test case:

Below code is exemplary and can be used as a template when writing test cases.

.. _verification_template_cpp:

C++ Properties Template
^^^^^^^^^^^^^^^^^^^^^^^

.. code-block:: cpp

class TestSuite : public ::testing::Test{
public:
void SetUp() override
{
RecordProperty("TestType", "<TestType>");
RecordProperty("DerivationTechnique", "<DerivationTechnique>");
...
}
};

TEST_F(TestSuite, <Test Case>)
{
RecordProperty("PartiallyVerifies", "ID_2, ID_3, ...");
RecordProperty("FullyVerifies", "ID_4, ID_5, ...");
RecordProperty("Description", "<Description>");

ASSERT ...
}

When writing test cases using gTest, they shall follow the recommendations from the official gTest documentation.
For very basic start follow http://google.github.io/googletest/primer.html
For advanced information follow http://google.github.io/googletest/advanced.html

Expand All @@ -38,20 +72,63 @@ Details on the definition an the test organization in rust can be found here:
https://doc.rust-lang.org/book/ch11-03-test-organization.html

Each test case requires a link to one or more requirement/design element.
A more detailed description of how to link code to requirements is available here: :need:`gd_req__verification_link_tests_rust`

Writing test cases in rust shall follow the recommendations from the official rust documentation.
Below code is exemplary and can be used as a template when writing test cases.

.. _verification_template_rust:

Rust Properties Template
^^^^^^^^^^^^^^^^^^^^^^^^

For linking Rust tests to requirements **#[record_property]** shall be used:

.. code-block:: rust

use test_properties::record_property;

#[record_property("PartiallyVerifies", "ID_2, ID_3, ...")]
#[record_property("FullyVerifies", "ID_4, ID_5, ...")]
#[record_property("Description", "<Description>")]
#[record_property("TestType", "<TestType>")]
#[record_property("DerivationTechnique", "<DerivationTechnique>")]
#[test]
fn test_case_function() {
...
}

When writing test cases in rust, they shall follow the recommendations from the official rust documentation.
https://doc.rust-lang.org/book/ch11-01-writing-tests.html


Python
------

Writing test cases in python should be done using pytest.
When writing test cases in python, this should be done using pytest.
Note that python unittest does not support metatags and therefore should not be considered as test framework.

Each test case requires a link to one or more requirement/design element.
A more detailed description of how to link code to requirements is available here: :need:`gd_req__verification_link_tests_python`

Writing test cases in python shall follow the recommendations from the official python and community documentation.
For linking python tests to requirements **metadata** shall be used.

For allowed values for test_type & derivation_technique please check :need:`gd_req__verification_link_tests`

Below code is exemplary and can be used as a template when writing test cases.

.. _verification_template_python:

Python Properties Template
^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code-block:: python

@add_test_properties(
partially_verifies=["tool_req__docs_dd_link_source_code_link"],
test_type="requirements-based",
derivation_technique="requirements-analysis",
)
def test_group_by_need_empty_list():
"""Test grouping empty list of needlinks."""
...

When writing test cases in python, they shall follow the recommendations from the official python and community documentation.
https://docs.python-guide.org/writing/tests/
12 changes: 2 additions & 10 deletions process/process_areas/verification/verification_concept.rst
Original file line number Diff line number Diff line change
Expand Up @@ -142,16 +142,8 @@ Following aspect should be considered when developing test cases:
Consider not mocking away libraries the unit uses, as long as you can obtain sufficient
structural coverage from the unit testing with included/integrated libraries,
as this reduces effort on integration testing.

Use the following frameworks for unit testing:

* **Rust:** Utilize the built-in testing framework with ``#[test]`` attributes and the ``cargo test`` command.
* **Python:** Use ``pytest`` frameworks.
* **C++:** Use Google Test frameworks.
* **Integration Testing:** Verify the interaction between different components or modules.
For integration testing, the ITF (Integration Test Framework) is used.

For more information, see ``[TODO: Link to ITF documentation once available. Related feature request is #599]``.
Depending on the implementation this can be on component, module or feature level.
* **Platform Integration Testing:** Test the platform with configured features as a whole.
* **Regression Testing:** Ensure that changes do not introduce new defects.
Automate regression tests where possible as they will get executed as part of the CI.
Expand Down Expand Up @@ -204,5 +196,5 @@ generated including:
So if the content of the requirement is altered also the hash will change making it necessary to
revisit the linkage of all test cases to the requirement again.

If the status of the linked test case and the linkage document is valid the attribute *testcovered* shall be set to *YES* during the Sphinx Build.
If the status of the linked test case and the linkage document is valid the attribute *testcovered* shall be set to *YES* during the SW Build.
Further information can also be depicted from the :ref:`requirements_engineering` process.
14 changes: 0 additions & 14 deletions process/process_areas/verification/verification_getstrt.rst
Original file line number Diff line number Diff line change
Expand Up @@ -48,17 +48,3 @@ The workflows can be split into 4 major parts:
available for a specific baseline.

The details of what needs to be done in each part are described in the :ref:`verification_workflows`.

Tooling Support
***************

The following tools may support the verification process (according to the project specific :need:`SW development Plan <wp__sw_development_plan>`):

* **Bazel:** For building the software and running tests (supported by ITF).
* **Google Test:** For unit testing in C++.
* **Rust's built-in testing framework:** For unit testing in Rust.
* **Pytest:** For component to integration testing in Python.
* **ITF (Integration Test Framework):** As framework for integration testing based on pytest.
* **Sphinx-needs:** Tracing artifacts like requirements to test cases.

Additionally, tools will be provided to automatically generate coverage tests based on the unit source code.
Loading