TestNG Engine for the JUnit Platform

Allows executing TestNG tests on the JUnit Platform

Usage

Console Launcher

When running without a build tool, you need to download the following jars from Maven Central:

• junit-platform-console-standalone-1.8.2.jar

• testng-engine-1.0.5.jar

• testng-7.10.2.jar

• jcommander-1.72.jar

• jquery-3.5.1.jar

The following samples assume the above jars have been downloaded to the local lib folder and production and test classes to bin/main and bin/test, respectively.

$ java -cp 'lib/*' org.junit.platform.console.ConsoleLauncher \
       -cp bin/main -cp bin/test \
       --include-engine=testng --scan-classpath=bin/test

Thanks for using JUnit! Support its development at https://junit.org/sponsoring

╷
└─ TestNG ✔
   └─ CalculatorTests ✔
      ├─ add(int, int, int) ✔
      │  ├─ [0] 0, 1, 1 ✔
      │  ├─ [1] 1, 2, 3 ✔
      │  ├─ [2] 49, 51, 100 ✔
      │  └─ [3] 1, 100, 101 ✔
      └─ addsTwoNumbers ✔
            2021-07-04T17:43:52.223145 description = `1 + 1 = 2`

Test run finished after 38 ms
[         3 containers found      ]
[         0 containers skipped    ]
[         3 containers started    ]
[         0 containers aborted    ]
[         3 containers successful ]
[         0 containers failed     ]
[         5 tests found           ]
[         0 tests skipped         ]
[         5 tests started         ]
[         0 tests aborted         ]
[         5 tests successful      ]
[         0 tests failed          ]

Gradle

build.gradle[.kts]

dependencies {
    testImplementation("org.testng:testng:7.10.2")
    testRuntimeOnly("org.junit.support:testng-engine:1.0.5") // (1)
}
tasks.test {
    useJUnitPlatform() // (2)
}

1. Add the engine as an extra dependency for running tests

2. Configure the test task to use the JUnit Platform

Maven

pom.xml

<project>
    <!-- ... -->
    <dependencies>
        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>7.10.2</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.support</groupId>
            <artifactId>testng-engine</artifactId>
            <version>1.0.5</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.0.0-M7</version>
            </plugin>
        </plugins>
    </build>
    <!-- ... -->
</project>

Supported versions

TestNG

The engine supports TestNG version 6.14.3 and above.

JUnit Platform

The engine requires at least JUnit Platform 1.5.x.

Configuration Parameters

The following JUnit Platform configuration parameters are supported.

Execution

testng.allowReturnValues (boolean)

whether methods annotated with @Test that have return values should be considered test methods (default: false; see documentation)

testng.dataProviderThreadCount (integer)

maximum number of threads to use for running data providers in parallel, if enabled via @DataProvider(parallel = true) (default: 10; see documentation)

testng.excludedGroups (comma-separated list)

groups to exclude (see Groups vs. Tags)

testng.groups (comma-separated list)

groups to be run (see Groups vs. Tags)

testng.parallel (methods|tests|classes|instances|none)

TestNG’s parallel execution mode for running tests in separate threads (default: "none"; see documentation)

testng.preserveOrder (boolean)

whether classes and methods should be run in a predictable order (default: true; see documentation)

testng.threadCount (integer)

maximum number of threads for running tests in parallel, if enabled via testng.parallel (default: 5; see documentation)

Reporting

testng.listeners (comma-separated list of fully-qualified class names)

custom listeners that should be registered when executing tests (default: ""; see documentation)

testng.outputDirectory (file path)

the output directory for reports (default: "test-output"; see documentation)

testng.useDefaultListeners (boolean)

whether TestNG’s default report generating listeners should be used (default: false; see documentation)

testng.verbose (integer)

TestNG’s level of verbosity for console output (default: 0)

Generating TestNG reports

Console Launcher

$ java -cp 'lib/*' org.junit.platform.console.ConsoleLauncher \
       -cp bin/main -cp bin/test \
       --include-engine=testng --scan-classpath=bin/test \
       --config=testng.useDefaultListeners=true \
       --config=testng.outputDirectory=test-reports

Gradle

build.gradle[.kts]

tasks.test {
    useJUnitPlatform()
    systemProperty("testng.useDefaultListeners", "true")

    val testNGReportsDir = layout.buildDirectory.dir("reports/testng")
    outputs.dir(testNGReportsDir).withPropertyName("testng-reports")
    jvmArgumentProviders += CommandLineArgumentProvider {
        listOf("-Dtestng.outputDirectory=${testNGReportsDir.get().asFile.absolutePath}")
    }
}

Maven

<project>
    <!-- ... -->
    <build>
        <plugins>
            <plugin>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.0.0-M7</version>
                <configuration>
                    <properties>
                        <configurationParameters>
                            testng.useDefaultListeners = true
                            testng.outputDirectory = ${project.build.directory}/testng-reports
                        </configurationParameters>
                    </properties>
                </configuration>
            </plugin>
        </plugins>
    </build>
    <!-- ... -->
</project>

Registering custom listeners

Console Launcher

$ java -cp 'lib/*' org.junit.platform.console.ConsoleLauncher \
       -cp bin/main -cp bin/test \
       --include-engine=testng --scan-classpath=bin/test \
       --config=testng.listeners=com.acme.MyCustomListener1,com.acme.MyCustomListener2

Gradle

build.gradle[.kts]

tasks.test {
    useJUnitPlatform()
    systemProperty("testng.listeners", "com.acme.MyCustomListener1, com.acme.MyCustomListener2")
}

Maven

<project>
    <!-- ... -->
    <build>
        <plugins>
            <plugin>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.0.0-M7</version>
                <configuration>
                    <properties>
                        <configurationParameters>
                            testng.listeners = com.acme.MyCustomListener1, com.acme.MyCustomListener2
                        </configurationParameters>
                    </properties>
                </configuration>
            </plugin>
        </plugins>
    </build>
    <!-- ... -->
</project>

Limitations

Groups vs. Tags

Groups declared via the @Test annotation on test classes and methods are exposed as tags to the JUnit Platform. Hence, you can use tag filter expressions to include or exclude certain groups. For example, given the following test class, including the tag included and excluding excluded will run test a and c but not b and d.

import org.testng.annotations.Test;

public class TestWithGroups {
    @Test(groups = "included")
    public void a() {}
    @Test(groups = {"included", "excluded"})
    public void b() {}
    @Test(groups = "included")
    public void c() {}
    @Test
    public void d() {}
}

However, since tags and therefore groups are filtered rather than selected, @BeforeGroups and @AfterGroups configuration methods are not executed on some versions of TestNG. Moreover, tag filters do not support wildcards such as integration.*. Therefore, instead of using tags, you can use the testng.groups and testng.excludedGroups configuration parameters to specify the groups that should be selected.

Warning

Ideally, you should use either tags or the group configuration parameters. If you specify both, please ensure they match.

Using Tags

Console Launcher

$ java -cp 'lib/*' org.junit.platform.console.ConsoleLauncher \
       -cp bin/main -cp bin/test \
       --include-engine=testng --scan-classpath=bin/test \
       --include-tag=included --exclude-tag=excluded

Gradle

build.gradle[.kts]

tasks.test {
    useJUnitPlatform {
        includeTags("included")
        excludeTags("excluded")
    }
}

Maven

<project>
    <!-- ... -->
    <build>
        <plugins>
            <plugin>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.0.0-M7</version>
                <configuration>
                    <groups>included</groups>
                    <excludedGroups>excluded</excludedGroups>
                </configuration>
            </plugin>
        </plugins>
    </build>
    <!-- ... -->
</project>

Using Configuration Parameters

Console Launcher

$ java -cp 'lib/*' org.junit.platform.console.ConsoleLauncher \
       -cp bin/main -cp bin/test \
       --include-engine=testng --scan-classpath=bin/test \
       --config=testng.groups=included --config=testng.excludedGroups=excluded

Gradle

build.gradle[.kts]

tasks.test {
    useJUnitPlatform()
    systemProperty("testng.groups", "included")
    systemProperty("testng.excludedGroups", "excluded")
}

Maven

<project>
    <!-- ... -->
    <build>
        <plugins>
            <plugin>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.0.0-M7</version>
                <configuration>
                    <properties>
                        <configurationParameters>
                            testng.groups = included
                            testng.excludedGroups = excluded
                        </configurationParameters>
                    </properties>
                </configuration>
            </plugin>
        </plugins>
    </build>
    <!-- ... -->
</project>

Suites

The engine’s main intention is integration with build tools like Gradle and Maven. Hence, custom suites specified via testng.xml files are not supported. However, you can use JUnit’s suite support with this engine.