Performance of Pitest/Descartes

[vmassol]

FYI I've implemented on XWiki (in xwiki-commons to be precise) the strategy of breaking the build when the mutation score computed by Pitest/Descartes is below a defined threshold (see https://markmail.org/message/bonb5f7f37omnnog ).

However it has an important performance cost. For example on xwiki-commons:

• Without pitest/descartes: 5:10 mn
• With pitest/descartes: 37:16 mn
• With pitest/decartes and using 4 threads :15:12 mn
• With pitest/decartes and using 8 threads :15:02 mn (just checked and indeed my macbook pro only has 4 cores: https://support.apple.com/kb/SP749?locale=en_US)

[bbaudry]

thanks @vmassol for running this experiment. It will be super interesting to see how the Descartes metrics (score, nb live mutants, nb killing tests, etc.) evolve over time
Performance is a challenge, which we need to discuss next week.

[monperrus]

Hi Vincent,

FYI I've implemented on XWiki (in xwiki-commons to be precise) the strategy of breaking the build when the mutation score computed by Pitest/Descartes is below a defined threshold (see https://markmail.org/message/bonb5f7f37omnnog ).

Excellent! What threshold do you use? It would be great to write a tutorial documenting how to do this! See you next week,

…

--Martin

[vmassol]

You're going a bit fast @monperrus ;)

I still need to validate a few things before I can write about this:

• That it will indeed fail when I introduce a test with lower quality (I'm testing this today and in the coming days)
• That it doesn't bring false positives (ie that you add good tests that will still lower the threshold)

Re thresholds, I use the mutation score computed by pitest/descartes and set is as the threshold. The idea is to progress, and not to reach a given value. Same strategy that we're applying with test coverage and jacoco.

Thx. Stay tuned.

[monperrus]

Hi, Martin, are you thinking of a tutorial internal to STAMP? Or a tutorial/tool demo at a conference, which would also be useful internally, but also gives some visibility externally Tutorial deadline for ASE is April 30 Tool demo deadline for ASE is May 31 Best regards, Brice.

-----Original Message----- From: stamp-request@ow2.org [mailto:stamp-request@ow2.org] On Behalf Of Martin Monperrus Sent: onsdag 28. mars 2018 09.48 To: STAMP-project/docs-forum <reply@reply.github.com>; stamp@ow2.org Cc: Subscribed <subscribed@noreply.github.com> Subject: Re: [stamp] [STAMP-project/docs-forum] Use of Pitest/Descartes (#11) Hi Vincent,

FYI I've implemented on XWiki (in xwiki-commons to be precise) the strategy of breaking the build when the mutation score computed by Pitest/Descartes is below a defined threshold (see https://markmail.org/message/bonb5f7f37omnnog ).

Excellent! What threshold do you use? It would be great to write a tutorial documenting how to do this! See you next week,

…

--Martin

[vmassol]

@monperrus and @oscarlvp (cc @danglotb): I did some tests on score evolution in one xwiki-commons module and reported my findings at STAMP-project/pitest-descartes#54

In short: it seems the strategy is not working that well. It's not completely useless but less interesting than I thought it would be.

Feel free to comment on STAMP-project/pitest-descartes#54

Thx

[monperrus]

Hi Brice, I'm thinking of a written document for the general audience. It would be self-contained and it could be understood by people without the Stamp context. This document would increase the visibility and impact of our work. That said, it's excellent idea to make a "live tutorial" at a conference! Cheers,

…

--Martin

-- Martin Monperrus, Professor, KTH Royal Institute of Technology On Wed, 2018-03-28 at 08:18 +0000, Brice Morin wrote: Hi, Martin, are you thinking of a tutorial internal to STAMP? Or a tutorial/tool demo at a conference, which would also be useful internally, but also gives some visibility externally Tutorial deadline for ASE is April 30 Tool demo deadline for ASE is May 31 Best regards, Brice.

-----Original Message----- From: stamp-request@ow2.org<mailto:stamp-request@ow2.org> [mailto:stamp-request@ow2.org] On Behalf Of Martin Monperrus Sent: onsdag 28. mars 2018 09.48 To: STAMP-project/docs-forum <reply@reply.github.com<mailto:reply@reply.github.com>>; stamp@ow2.org<mailto:stamp@ow2.org> Cc: Subscribed <subscribed@noreply.github.com<mailto:subscribed@noreply.github.com>> Subject: Re: [stamp] [STAMP-project/docs-forum] Use of Pitest/Descartes (#11) Hi Vincent, FYI I've implemented on XWiki (in xwiki-commons to be precise) the strategy of breaking the build when the mutation score computed by Pitest/Descartes is below a defined threshold (see https://markmail.org/message/bonb5f7f37omnnog ). Excellent! What threshold do you use? It would be great to write a tutorial documenting how to do this! See you next week,

--Martin