squash: better docs and schema

spec/plans/results.fmf

@@ -82,9 +82,12 @@ description: |
       # key. Fields have the same meaning as fields of the "parent" test result, but
       # relate to each check alone.
       check:
-          # String, outcome of the check execution.
+          # String, the effective outcome of the check execution.
         - result: "pass"|"fail"|"info"|"warn"|"error"|"skip"
 
+          # String, the original outcome of the check execution.
+          original_result: "pass"|"fail"|"info"|"warn"|"error"|"skip"
+
           # String, optional comment to report with the result.
           note: "Things were great."
 
@@ -117,9 +120,12 @@ description: |
           # String, name of the test phase.
         - name: First test case
 
-          # String, outcome of the phase execution.
+          # String, the effective outcome of the phase execution.
           result: "pass"|"fail"|"info"|"warn"|"error"|"skip"
 
+          # String, the original outcome of the phase execution.
+          original_result: "pass"|"fail"|"info"|"warn"|"error"|"skip"
+
           # String, optional comment to report with the result.
           note: "Things were great."
 
@@ -213,9 +219,25 @@ description: |
     original test result in the custom result set to the value known
     to tmt.
 
-    The ``original_result`` key holds the outcome of the test as reported
-    by the test itself, without any additional influence of test checks,
-    subresults or :py:ref:`/spec/tests/result`.
+    The ``original_result`` key holds the outcome of a test, check or
+    subresult as reported by the test, check or subresult itself, without
+    any additional influence.
+    A reported outcome may be a subject to interpretation, and tmt may
+    consider additional inputs and eventually report different effective
+    outcome in the ``result`` key.
+
+    The following rules apply when it comes to test, test check and subresult
+    outcomes and their interpretation and effects:
+
+    * Test outcome is interpreted according to
+      :py:ref:`/spec/tests/result`. It is not yet planned for test check
+      outcomes be interpreted. It is not yet planned for subresult
+      outcomes to be interpreted.
+    * Test checks and subresults do not influence the effective test
+      outcome. It is however planned for test check outcomes to affect
+      the test outcome, see https://github.com/teemtee/tmt/issues/3185
+      for more details. It is not planned for subresults to affect
+      the test outcome, and we do not expect them to gain this effect.
 
     See also the complete `JSON schema`__.
 

tmt/result.py

@@ -118,6 +118,9 @@ class BaseResult(SerializableContainer):
     end_time: Optional[str] = None
     duration: Optional[str] = None
 
+    def __post_init__(self) -> None:
+        self.original_result = self.result
+
     def show(self) -> str:
         """ Return a nicely colored result with test name (and note) """
 
@@ -266,7 +269,6 @@ def from_test_invocation(
             fmf_id=invocation.test.fmf_id,
             context=invocation.phase.step.plan._fmf_context,
             result=result,
-            original_result=result,
             note=note,
             start_time=invocation.start_time,
             end_time=invocation.end_time,

tmt/schemas/results.yaml

@@ -29,6 +29,9 @@ definitions:
       result:
         $ref: "/schemas/common#/definitions/result_outcome"
 
+      original_result:
+        $ref: "/schemas/common#/definitions/result_outcome"
+
       note:
         $ref: "/schemas/common#/definitions/result_note"
 
@@ -63,6 +66,9 @@ definitions:
       result:
         $ref: "/schemas/common#/definitions/result_outcome"
 
+      original_result:
+        $ref: "/schemas/common#/definitions/result_outcome"
+
       note:
         $ref: "/schemas/common#/definitions/result_note"
 
@@ -120,6 +126,9 @@ items:
     result:
       $ref: "/schemas/common#/definitions/result_outcome"
 
+    original_result:
+      $ref: "/schemas/common#/definitions/result_outcome"
+
     context:
       $ref: "/schemas/common#/definitions/context"