Open Test Reporting

New test reporting formats that are agnostic of testing framework and programming language.

Introduction

This repository contains the definition of and tooling for two new XML-based test reporting formats that are agnostic of any testing framework and programming language yet concrete enough to be consumable by a wide range of downstream tools such as IDEs, build tools, and report generators.

The first format is event-based which is not only suitable for writing events to a file but also for streaming events over a local socket or network connection. Instead of having to collect all data for a test only to write it to the report after the test has completed, testing frameworks can emit events as they occur, for example when a test is started or finished.

The second format is meant to be closer to existing hierarchical representations of test results that users are familiar with. For example, when executing tests in IDEs, one is usually presented with a tree of tests and their results. This same structure can be found in files that use this format. Thus, instead of requiring humans or processing tools to piece together all relevant events for a test from the event-based format, this hierarchical format collects all relevant information in a single place for each test. Besides being human-readable, this format is easier to transform into an HTML report by a downstream reporting tool or CI server.

The two formats are designed to complement each other such that the event-based format can be mechanically converted into the hierarchical one. This repository contains a reference implementation of such a converter as an executable and library. This way, testing frameworks can focus on writing the event-based format; build tools may use the converter to write the hierarchical format; and reporting tools can consume whichever format they prefer.

Prior art

In the Java ecosystem, Ant originally defined an XML-based reporting format for tests. Other build tools like Maven and Gradle have adopted the format and, in the case of Maven Surefire, later made changes to add additional data. Many build servers know how to parse the XML-based format, and even non-Java tools sometimes support it. However, it’s based on the concept of test classes and methods, so using it for frameworks and tools where those elements are not present is awkward at best. Moreover, it does not support nested structures beyond a simple parent-child relationship. Finally, it is not extensible: no additional attributes can be added without the risk of breaking existing tools.

For those reasons, many testing frameworks such as TestNG and Spock have defined their own reporting formats. This has given them the flexibility they need, but the number of tools that can parse, display, or transform their custom formats is very limited.

To overcome the limitations described above, this document defines a new format for test reporting. Its goal is to be platform-agnostic so that as many testing frameworks as possible can benefit from it. Moreover, it is designed to be extensible so new data can be added as needed, without breaking consumers. However, all well-known attributes are properly defined so it’s consumable by downstream reporting tools.

Design goals

human-readable

The new format needs to be human-readable so it can be inspected without requiring another tool. A format that represents the test tree via nesting is easier to understand than a flat list of events that reference their parents via IDs.

streamable

Writing report files during test execution should not require keeping state about currently running tests, etc. in memory. Instead, the new format should allow tools to write events to a file as they occur.

machine-readable

At the same time, the format must be machine-readable so it can be easily parsed and transformed on a variety of platforms.

schema-aware

Implementors and tools should be able to validate a given document against a well-known schema.

extensible

Adding additional language- or framework-specific attributes to nodes must be possible without breaking backwards compatibility.

XML vs. JSON

JSON is less verbose than XML, but the latter provides more expressive ways to define schemas. Moreover, XML has typed extensions built-in via the use of multiple schemas. Thus, the new formats use XML with accompanying XML schemas.

Format specification

Both formats share a set of core elements that is defined in core.xsd.

Event-based format

The event-based format is defined in events.xsd.

The following example shows the result of a test run with a single top-level "container" and a "test" child.

<?xml version="1.0" ?>
<e:events
        xmlns="https://schemas.opentest4j.org/reporting/core/0.1.0"
        xmlns:e="https://schemas.opentest4j.org/reporting/events/0.1.0">
    <infrastructure> <!--(1)-->
        <hostName>wonderland</hostName>
        <userName>alice</userName>
    </infrastructure>
    <e:started id="1" name="container" time="2022-02-05T16:30:39.129888Z"/> <!--(2)-->
    <e:started id="2" name="test" parentId="1" time="2022-02-05T16:30:39.137022Z"/> <!--(3)-->
    <e:finished id="2" time="2022-02-05T16:30:39.143013Z"> <!--(4)-->
        <result status="SUCCESSFUL"/>
    </e:finished>
    <e:finished id="1" time="2022-02-05T16:30:39.143292Z"> <!--(5)-->
        <result status="SUCCESSFUL"/>
    </e:finished>
</e:events>

1. attributes of the infrastructure the tests were run on

2. start event of "container" with timestamp

3. start event of "test" with timestamp and reference to its parent "container"

4. finished event of "test" with timestamp and result status

5. finished event of "container" with timestamp and result status

This file contains 4 events: 2 started and 2 finished ones. The test framework reporting these events was able to write them as they occurred. In particular, it did not have to wait for all children of "container" to be finished.

Hierarchical format

The event-based format is defined in hierarchy.xsd.

The following example shows the result of converting the above event-based report into the hierarchical format.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<h:execution xmlns:h="https://schemas.opentest4j.org/reporting/hierarchy/0.1.0"
             xmlns="https://schemas.opentest4j.org/reporting/core/0.1.0">
    <infrastructure> <!--(1)-->
        <hostName>wonderland</hostName>
        <userName>alice</userName>
    </infrastructure>
    <h:root duration="PT0.013404S" name="container" start="2022-02-05T16:30:39.129888Z"> <!--(2)-->
        <result status="SUCCESSFUL"/>
        <h:child duration="PT0.005991S" name="test" start="2022-02-05T16:30:39.137022Z"> <!--(3)-->
            <result status="SUCCESSFUL"/>
        </h:child>
    </h:root>
</h:execution>

1. attributes of the infrastructure the tests were run on (same as in the event-based format)

2. root node of "container" with timestamp, duration, and result status

3. child node of "test" with timestamp, duration, and result status

Java extensions

All schema definitions mentioned so far are language-agnostic. In order to report Java-specific attributes, e.g. the class or method name of a test, an extension schema is defined in java.xsd.

Note	Test frameworks are encouraged to define their own similar framework-specific extensions if they want to report additional information that is not suitable to be added to the core namespace.

Reference implementation

While the reporting formats are language-agnostic, the reference implementation is written in Java due to being the language its authors are most familiar with.

This repository contains the following subprojects:

schema

XML schema definitions of both formats

events

Java API for writing the event-based format without having to deal with Java’s XML APIs

cli

Command-line interface for validating both fomats and converting from the event-based to the hierarchical format

tooling

Java API for validating both formats and converting from the event-based to the hierarchical format (suitable for inclusion in build tools and reporting tools)

API for writing event-based format

Testing frameworks that run on the JVM can use the API provided by the events subprojects as follows.

import org.opentest4j.reporting.events.api.DocumentWriter;
import org.opentest4j.reporting.events.api.NamespaceRegistry;
import org.opentest4j.reporting.events.core.CoreFactory;
import org.opentest4j.reporting.events.root.Events;
import org.opentest4j.reporting.schema.Namespace;

import java.nio.file.Paths;
import java.time.Instant;

import static org.opentest4j.reporting.events.core.CoreFactory.*;
import static org.opentest4j.reporting.events.core.Result.Status.SUCCESSFUL;
import static org.opentest4j.reporting.events.root.RootFactory.finished;
import static org.opentest4j.reporting.events.root.RootFactory.started;

public class DocumentWriterSample {

    public static void main(String[] args) throws Exception {

        NamespaceRegistry namespaceRegistry = NamespaceRegistry.builder(Namespace.REPORTING_CORE) // (1)
                .add("e", Namespace.REPORTING_EVENTS) //
                .add("java", Namespace.REPORTING_JAVA) //
                .build();

        try (DocumentWriter<Events> writer = Events.createDocumentWriter(namespaceRegistry, Paths.get("events.xml"))) {
            writer.append(infrastructure(), infrastructure -> infrastructure // (2)
                    .append(userName("alice")) //
                    .append(hostName("wonderland")));
            writer.append(started("1", Instant.now(), "container")); // (3)
            writer.append(started("2", Instant.now(), "test"), started -> started.withParentId("1")); // (4)
            writer.append(finished("2", Instant.now()), finished -> finished.append(CoreFactory.result(SUCCESSFUL))); // (5)
            writer.append(finished("1", Instant.now()), finished -> finished.append(CoreFactory.result(SUCCESSFUL))); // (6)
        }
    }
}

1. create a registry of all namespaces used in the document along with their prefixes

2. report infrastructure attributes

3. start event of "container" with timestamp

4. start event of "test" with timestamp and reference to its parent "container"

5. finished event of "test" with timestamp and result status

6. finished event of "container" with timestamp and result status

CLI tool for validation and format conversion

The CLI tool provided by the cli subprojects provides subcommands for validating both formats and converting from the event-based to the hierarchical format.

$ ./open-test-reporting convert events.xml
ℹ️ Converted events.xml to hierarchy.xml

Note	Please refer to the CLI tool’s --help option for more information.