Sonar Pitest Plugin

Compatibility Matrix

Sonarqube version	Pitest plugin version
Sonarqube 7.1	sonar pitest 1.0-SNAPSHOT
Sonarqube 6.7 (LTS)	sonar pitest 0.9 (download)
Sonarqube 6.5	sonar pitest 0.8 (download)
Sonarqube 5.6.X	sonar pitest 0.7 (download)

Contributing

Pull Request (PR)

To submit a contribution, create a pull request for this repository. Please make sure that you follow the SonarQube Developer Guidelines code style and all tests are passing.

Description / Features

PIT is a mutation testing tool for java. You can check out the official pitest web site for more details on mutation testing and PIT. Long story short, mutation testing is a very smart way to check the relevance of unit tests. The main idea is to alter the tested code and check that at least one unit test fails. An alteration of the code is called a "mutant". A mutant has "survived" the tests if there is no test failure. A mutant is "killed" if there is a test failure when the tests are executed on the mutated code.

The goal of this plugin is to bring PIT results to SonarQube. "Survived mutants" are seen as SonarQube issues. "Killed mutants" show as a coverage measure in the class containing the mutant. If code is not covered by any test, that code will not be mutated; prior test coverage is a precondition for mutation testing. Finally, if the percentage of "Survived mutants" in a source file exceeds a configurable threshold, the plugin creates a SonarQube issue on the source file.

Usage

Notes on mutation testing

Mutation testing should only be executed on true unit tests. Do not try to use it on tests accessing resources such as a database or filesystem, as the mutation may can unexpected consequences.
Mutation testing can be computationally expensive. Henry Coles, creator of PIT, provides the following tips to manage PIT execution time:

• Target only specific portions of your codebase (using the class filters)
• Limit the mutation operators used
• Limit the number of mutations per class
• Tweak the number of threads

Configuration

The sonar-pitest plugin exposes two rules:

• "Survived mutant", which creates an issue (of TYPE BUG and SEVERITY MAJOR) whenever Mutated code does not result in a test failure
• "Insufficient Mutation Coverage", which creates an issue (of TYPE BUG and SEVERITY MAJOR) whenever the percentage of Survived mutants exceeds a configurable threshold (default: 65%)

Both rules are inactive by default

Project build setup

PIT needs to be launched before SonarQube You can launch PIT using the PIT maven plugin or the command line runner. PIT execution must be done before SonarQube analysis. You also need to specify the "reuseReport" mode of the PIT SonarQube plugin. Pit needs to be configured in order to generate XML reports. Be aware that PIT default behavior is to generate HTML reports. Below a simple configuration example for maven :

<!-- inside the build/plugins section -->
<plugin>
  <groupId>org.pitest</groupId>
  <artifactId>pitest-maven</artifactId>
  <version>LATEST</version>
  <configuration>
    <inScopeClasses>
      <param>com.acme.tools.commons*</param>
    </inScopeClasses>
    <targetClasses>
      <param>com.acme.tools.commons*</param>
    </targetClasses>
    <outputFormats>
      <outputFormat>XML</outputFormat>
    </outputFormats>
  </configuration>
</plugin>

inScopeClasses and targetClasses parameters indicated the classes of the system under test where mutations can be performed. In the example above, all the classes from the com.acme.tools.commons package and sub packages may be altered.
Once configured in the maven pom file, you need to run PITusing the following command:

mvn org.pitest:pitest-maven:mutationCoverage

Note : Of course, all the configuration options are clearly explained in the official pitest documentation. Last but not least, you need to run a SonarQube analysis with the PIT plugin activated in "reuseReport" mode (the default). The following command would do the job:

mvn sonar:sonar

By default SonarQube will search the latest PIT report in "target/pit-reports". You can specify another location using property "sonar.pitest.reportsDirectory". You will find below the list of all the available configuration parameters.

Basic configuration properties

Below the exhaustive list of configuration properties of the SonarQube pitest plugin:

Name	Key	Default value	Description
Pitest activation mode	sonar.pitest.mode	reuseReport	Possible values : 'skip' and 'reuseReport'
Path to the pitest reports	sonar.pitest.reportsDirectory	target/pit-reports	Path used to locate pitest xml reports. Pitest creates a new subfolder "timestamp" at each shot. The SonarQube plugin will explore these subfolders and find the newest xml reports generated.

You can check out the quickstart section of the official pitest web site for detailed instructions.