Generate edge cases

[Icelandjack]

Generate edge cases (minBound, maxBound, minBound + 1, ...) with some low frequency.

---

abs always returns a positive number right?

>>> quickCheck (\(n :: Int) -> abs n >= 0)
+++ OK, passed 100 tests.

Alright let's ship it — well — the test fails if we run it on Int8s often enough

>>> quickCheck (\(n :: Int8) -> abs n >= 0)
+++ OK, passed 100 tests.
>>> quickCheck (\(n :: Int8) -> abs n >= 0)
+++ OK, passed 100 tests.
>>> quickCheck (\(n :: Int8) -> abs n >= 0)
+++ OK, passed 100 tests.
>>> quickCheck (\(n :: Int8) -> abs n >= 0)
*** Failed! Falsifiable (after 23 tests):                  
-128

Turns out it doesn't actually hold for Ints

>>> abs @Int minBound
-9223372036854775808

This could be detected for larger sample spaces like Int if minBound were generated. I would like to see this in the Arbitrary Int instance but maybe a modifier is better?

[Icelandjack]

Minimal modifier that produces a counterexample for Int

newtype XTreme a = XTreme a deriving Show

instance (Arbitrary a, Num a, Bounded a) => Arbitrary (XTreme a) where
  arbitrary = let
    prob = [(1, pure minBound), (1, pure (minBound + 1)), (1, pure (minBound + 2)),
            (1, pure maxBound), (1, pure (maxBound - 1)), (1, pure (maxBound - 2)),
            (1000-6, arbitrary)]

    in XTreme <$> frequency prob

>>> quickCheckWith stdArgs { maxSuccess = 1000 } (\(XTreme (a :: Int)) -> abs a >= 0)
*** Failed! Falsifiable (after 94 tests):
XTreme (-9223372036854775808)

[Icelandjack]

Found a similar version in the tests:

newtype Extremal a = Extremal { getExtremal :: a } deriving (Show, Eq, Ord, Num, Enum, Real, Integral)

instance (Arbitrary a, Bounded a) => Arbitrary (Extremal a) where
  arbitrary =
    fmap Extremal $
    frequency
      [(1, return minBound),
       (1, return maxBound),
       (8, arbitrary)]
shrink (Extremal x) = map Extremal (shrink x)

[phadej]

I'd rather do

{-# LANGUAGE ScopedTypeVariables, GeneralizedNewtypeDeriving #-}
import Test.QuickCheck

newtype EB a = EB { getEB :: a }
   deriving (Show, Eq, Ord, Num, Enum, Real, Integral)

instance (Enum a, Bounded a) => Arbitrary (EB a) where
    arbitrary = EB <$> frequency
      [ (20, toEnum <$> choose (i, j))
      , (1, return minBound)
      , (1, return maxBound)
      ]
      where
        i = fromEnum (minBound :: a)
        j = fromEnum (maxBound :: a)

    shrink = fmap (EB . toEnum) . shrink . fromEnum . getEB

[nick8325]

Seems like a good idea!

However, we shouldn't do it for the default Int generator. That deliberately only generates small numbers, as people use Ints as (e.g.) lengths of lists, this kind of thing:

prop_replicate_length :: Int -> String -> Property
prop_replicate_length n x = n >= 0 ==> length (replicate n x) === n

But adding it to the Large modifier might be sensible. To do that we should change arbitrarySizedBoundedIntegral. Then the fixed-size integer types (Int32 and friends) would generate extreme values by default, but not Int.

[Icelandjack]

Thanks for the comments, I think this should be mentioned in the documentation — a user may not know to use a modifier if their tests pass for the default generator. Jotting down some ideas

Alternative reality where Large generates minBound/maxBound:

The default generator for Int generates small numbers, it will miss edge cases like minBound @Int and maxBound @Int.

>>> quickCheck (\(n :: Int) -> abs n >= 0)
+++ OK, passed 100 tests.

that get caught by using a Large modifier

>>> quickCheck (\(Large (n :: Int)) -> abs n >= 0)
*** Failed! Falsifiable (after 21 tests):
Large (-9223372036854775808)

Using a modifier to generate small numbers explicitly makes sense, not suggesting that as a change though

prop_replicate_length :: Small Int -> String -> Property
prop_replicate_length (Small n) x = n >= 0 ==> length (replicate n x) === n

unrelated, given Integral a => Integral (Positive a) we can nest them

prop_replicate_length :: Positive (Small Int) -> String -> Property
prop_replicate_length (Positive (Small n)) x = length (replicate n x) === n

[minad]

Hi,

I had some issues with quickcheck not generating special values, so I made a small package for it which uses a Special newtype similar to what was proposed here.

• http://hackage.haskell.org/package/quickcheck-special
• https://github.com/minad/quickcheck-special

Is this going in the right direction concerning this issue? Maybe something like that could be added to QuickCheck at some point?

[nick8325]

Yes, generating special values is definitely a good idea!

The approach I would take is to change the existing Arbitrary instances for Char, Float etc., rather than having a modifier, so that you always get special values (I don't see any advantage to not generating them).

"Small" special values like Nothing, [], 0 are already generated with high probability, so the important thing is to get those special values that are hard to find currently.

[minad]

In principle I agree that everything should be generated. However for Float, the NaN values are problematic since they violate the typeclass laws.

[nick8325]

Oh yes, good point. NaN should be left out of that. Maybe things like infinity are also problematic.

Perhaps we should be conservative and have the principle that we only change the distribution of generated values, i.e. we generate values that it was unlikely to generate before, but not values that it was impossible to generate before. Probably that means we don't change the distribution of Float, but then have a modifier like in your package. But we can still do something for other types.

[Icelandjack]

Humour me, how about an additional method or subclass of Arbitrary, with a default value like shrink

class Arbitrary a where
  arbitrary :: Gen a

  shrink :: a -> [a]
  shrink _ = []

  extreme :: [a]
  extreme = []

  tooExtreme :: [a]
  tooExtreme = []

Now we have some control over extreme values: how extreme we want them as well as their frequency

instance Arbitrary Float where
  ...
  extreme :: [Float]
  extreme = [0, -0, 2.22044604925031308085e-16, 4.94065645841246544177e-324, 2.22507385850720138309e-308, 1.79769313486231570815e+308]

  tooExtreme :: [Float]
  tooExtreme = [0/0, 1/0, -1/0]

instance Arbitrary Int where
  extreme :: [Int]
  extreme = [0, 1, -1, minBound, maxBound, minBound + 1, maxBound - 1]

instance Arbitrary Char where
  extreme :: [Char]
  extreme = [minBound, maxBound, succ minBound, pred maxBound]

Edit: Just ike specialValues from @minad's package, making it a method of Arbitrary means any data type can be tested with special values but making it a special type class means we know which values support special values... bleh

class SpecialValues a where
  -- | Finite list of special values
  specialValues :: [a]

It could include frequency information, extreme :: [(Int, a)] but that feels arbitrary (pun intended).

[bekh6ex]

Just found out that quickcheck doesn't generate NaN and was quiet surprised.
@nick8325, you said that NaN should be left out.

Can you explain why exactly?

Because for me it seems that this value should definitely present in generation as soon as developers tend to forget about it and then weird things happen and this is what the quickcheck for - to help to reveal edge cases, isn't it?

[Lysxia]

That an input contains no NaN seems like a reasonable precondition to assume implicitly, similar to inputs not being undefined.

[nick8325]

Right, my thinking was that passing NaN to a function means that an error has already occurred - in this respect it seems very similar to undefined. I would expect most numeric algorithms to implicitly have a precondition that the input does not contain NaN.

However, it would definitely be useful to have a modifier so that the programmer can opt into testing with NaN.

[phadej]

👍 to the modifier for NaN