Statistics shrinked label examples and minimum percentage of labels

[EmilTholin]

Thank you so much for your hard work on fast-check! I've only recently started to experiment with property based testing, and fast-check is so well-written and well-documented and an absolute joy to work with!

I saw John Hughes' talk Building on developers' intuitions and got very excited about the new shrinked label examples and minimum percentage of labels features in QuickCheck. Do you think it would be possible to add these features to this great library?

[dubzzz]

Thanks a lot for those great references, I will have a deeper look into the approach in order to see how I can bring it into fast-check.

It is good to know, that fast-check already has limited support for labelled inputs. fc.statistics is able to show the user how many generated values are generated for each label:

const fc = require('fast-check');

const kvPairArb = fc.tuple(fc.integer(-10, 10), fc.integer());
const kvPairEqual = (a, b) => a[0] === b[0];

fc.statistics(
    // Property you would have passed into fc.assert
    fc.property(
        fc.set(kvPairArb, kvPairEqual),
        kvPairArb,
        () => true // code under test
    ),
    // Labeling function
    // can either return a single string label or an array of multiple string labels
    ([vs, [k, v]]) => {
        const label = vs.length === 0 ? 'empty'
            : Math.min(...vs.map(([kv,vv]) => kv)) >= k ? 'at start'
            : Math.max(...vs.map(([kv,vv]) => kv)) <= k ? 'at end'
            : 'at middle';
        return label +', ' + (vs.find(([kv,vv]) => kv === k) ? 'update' : 'create');
    },
    { numRuns: 10000 });

Running code on Runkit: https://runkit.com/dubzzz/5cabc890d332880012e2dcfa

With this code I got:

at middle, create..38.83%
at end, create.....15.73%
at start, create...15.19%
at middle, update..12.99%
empty, create.......9.99%
at start, update....3.98%
at end, update......3.29%

I'll come back to you asap ;) Thanks a lot for the references

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[moodmosaic]

Hmm... this seems to fail when run inside Deno

statistics test
TypeError: this.arb.withBias is not a function
    at ConverterToNext.generate (https://cdn.skypack.dev/-/fast-check@v2.25.0-uOb65XP5FkKjOw41iBhx/dist=es2019,mode=imports/optimized/fast-check.js:539:48)
    at Property.generate (https://cdn.skypack.dev/-/fast-check@v2.25.0-uOb65XP5FkKjOw41iBhx/dist=es2019,mode=imports/optimized/fast-check.js:962:28)
    at https://cdn.skypack.dev/-/fast-check@v2.25.0-uOb65XP5FkKjOw41iBhx/dist=es2019,mode=imports/optimized/fast-check.js:1738:26
    at https://cdn.skypack.dev/-/fast-check@v2.25.0-uOb65XP5FkKjOw41iBhx/dist=es2019,mode=imports/optimized/fast-check.js:1978:58
    at mapHelper (https://cdn.skypack.dev/-/fast-check@v2.25.0-uOb65XP5FkKjOw41iBhx/dist=es2019,mode=imports/optimized/fast-check.js:35:11)
    at mapHelper.next (<anonymous>)
    at Object.statistics (https://cdn.skypack.dev/-/fast-check@v2.25.0-uOb65XP5FkKjOw41iBhx/dist=es2019,mode=imports/optimized/fast-check.js:1989:14)
    at file:///.../tests/example.test.ts:197:6
    at asyncOpSanitizer (deno:runtime/js/40_testing.js:21:15)
    at resourceSanitizer (deno:runtime/js/40_testing.js:58:13)

// @ts-nocheck
// https://github.com/dubzzz/fast-check/issues/2781

Deno.test("statistics test", () => {
  const kvPairArb = fc.tuple(fc.integer(-10, 10), fc.integer());
  const kvPairEqual = (a, b) => a[0] === b[0];

  fc.statistics(
    // Property you would have passed into fc.assert
    fc.property(
      fc.set(kvPairArb, kvPairEqual),
      kvPairArb,
      () => true // code under test
    ),
    // Labeling function
    // can either return a single string label or an array of multiple string labels
    ([vs, [k, v]]) => {
      const label = vs.length === 0 ? 'empty'
        : Math.min(...vs.map(([kv,vv]) => kv)) >= k ? 'at start'
        : Math.max(...vs.map(([kv,vv]) => kv)) <= k ? 'at end'
        : 'at middle';
      return label +', ' + (vs.find(([kv,vv]) => kv === k) ? 'update' : 'create');
    },
    { numRuns: 1000 });
});

[moodmosaic]

I see a workaround in #2736, unsure if it applies here though.

[dubzzz]

Workaround should work fine for this case in theory