Skip to content

Latest commit

 

History

History
265 lines (198 loc) · 12.3 KB

theories.rst

File metadata and controls

265 lines (198 loc) · 12.3 KB

Using theories

Theories are a powerful tool for test-driven development, allowing you to test a specific behaviour against all permutations of a set of user-defined parameters known as "data points".

Adding theories

Adding theories is done by defining data points and a theory function:

#include <criterion/theories.h>

TheoryDataPoints(suite_name, test_name) = {
    DataPoints(Type0, val0, val1, val2, ..., valN),
    DataPoints(Type1, val0, val1, val2, ..., valN),
    ...
    DataPoints(TypeN, val0, val1, val2, ..., valN),
}

Theory((Type0 arg0, Type1 arg1, ..., TypeN argN), suite_name, test_name) {
}

suite_name and test_name are the identifiers of the test suite and the test, respectively. These identifiers must follow the language identifier format.

Type0/arg0 through TypeN/argN are the parameter types and names of theory theory function and are available in the body of the function.

Datapoints are declared in the same number, type, and order than the parameters inside the TheoryDataPoints macro, with the DataPoints macro. Beware! It is undefined behaviour to not have a matching number and type of theory parameters and datatypes.

Each DataPoints must then specify the values that will be used for the theory parameter it is linked to (val0 through valN).

Assertions and invariants

You can use any cr_assert or cr_expect macro functions inside the body of a theory function.

Theory invariants are enforced through the cr_assume(Condition) macro function: if Condition is false, then the current theory iteration aborts without making the test fail.

On top of those, more assume macro functions are available for common operations:

Macro Description
cr_assume_not(Condition) Assumes Condition is false.
cr_assume_null(Ptr) Assumes Ptr is NULL.
cr_assume_not_null(Ptr) Assumes Ptr is not NULL.
cr_assume_eq(Actual, Expected) Assumes Actual == Expected.
cr_assume_neq(Actual, Unexpected) Assumes Actual != Expected.
cr_assume_lt(Actual, Expected) Assumes Actual < Expected.
cr_assume_leq(Actual, Expected) Assumes Actual <= Expected.
cr_assume_gt(Actual, Expected) Assumes Actual > Expected.
cr_assume_geq(Actual, Expected) Assumes Actual >= Expected.
cr_assume_float_eq(Actual, Expected, Epsilon) Assumes Actual == Expected with an error of Epsilon.
cr_assume_float_neq(Actual, Unexpected, Epsilon) Assumes Actual != Expected with an error of Epsilon.
cr_assume_strings_eq(Actual, Expected) Assumes Actual and Expected are the same string.
cr_assume_strings_neq(Actual, Unexpected) Assumes Actual and Expected are not the same string.
cr_assume_strings_lt(Actual, Expected) Assumes Actual is less than Expected lexicographically.
cr_assume_strings_leq(Actual, Expected) Assumes Actual is less or equal to Expected lexicographically.
cr_assume_strings_gt(Actual, Expected) Assumes Actual is greater than Expected lexicographically.
cr_assume_strings_geq(Actual, Expected) Assumes Actual is greater or equal to Expected lexicographically.
cr_assume_arrays_eq(Actual, Expected, Size) Assumes all elements of Actual (from 0 to Size - 1) are equals to those of Expected.
cr_assume_arrays_neq(Actual, Unexpected, Size) Assumes one or more elements of Actual (from 0 to Size - 1) differs from their counterpart in Expected.

Configuring theories

Theories can optionally recieve configuration parameters to alter the behaviour of the underlying test; as such, those parameters are the same ones as the ones of the Test macro function (c.f. :ref:`test-config-ref`).

Full sample & purpose of theories

We will illustrate how useful theories are with a simple example using Criterion:

The basics of theories

Let us imagine that we want to test if the algebraic properties of integers, and specifically concerning multiplication, are respected by the C language:

int my_mul(int lhs, int rhs) {
    return lhs * rhs;
}

Now, we know that multiplication over integers is commutative, so we first test that:

#include <criterion/criterion.h>

Test(algebra, multiplication_is_commutative) {
    cr_assert_eq(my_mul(2, 3), my_mul(3, 2));
}

However, this test is imperfect, because there is not enough triangulation to insure that my_mul is indeed commutative. One might be tempted to add more assertions on other values, but this will never be good enough: commutativity should work for any pair of integers, not just an arbitrary set, but, to be fair, you cannot just test this behaviour for every integer pair that exists.

Theories purposely bridge these two issues by introducing the concept of "data point" and by refactoring the repeating logic into a dedicated function:

#include <criterion/theories.h>

TheoryDataPoints(algebra, multiplication_is_commutative) = {
    DataPoints(int, [...]),
    DataPoints(int, [...]),
};

Theory((int lhs, int rhs), algebra, multiplication_is_commutative) {
    cr_assert_eq(my_mul(lhs, rhs), my_mul(rhs, lhs));
}

As you can see, we refactored the assertion into a theory taking two unspecified integers.

We first define some data points in the same order and type the parameters have, from left to right: the first DataPoints(int, ...) will define the set of values passed to the int lhs parameter, and the second will define the one passed to int rhs.

Choosing the values of the data point is left to you, but we might as well use "interesting" values: 0, -1, 1, -2, 2, INT_MAX, and INT_MIN:

#include <limits.h>

TheoryDataPoints(algebra, multiplication_is_commutative) = {
    DataPoints(int, 0, -1, 1, -2, 2, INT_MAX, INT_MIN),
    DataPoints(int, 0, -1, 1, -2, 2, INT_MAX, INT_MIN),
};

Using theory invariants

The second thing we can test on multiplication is that it is the inverse function of division. Then, given the division operation:

int my_div(int lhs, int rhs) {
    return lhs / rhs;
}

The associated theory is straight-forward:

#include <criterion/theories.h>

TheoryDataPoints(algebra, multiplication_is_inverse_of_division) = {
    DataPoints(int, 0, -1, 1, -2, 2, INT_MAX, INT_MIN),
    DataPoints(int, 0, -1, 1, -2, 2, INT_MAX, INT_MIN),
};

Theory((int lhs, int rhs), algebra, multiplication_is_inverse_of_division) {
    cr_assert_eq(lhs, my_div(my_mul(lhs, rhs), rhs));
}

However, we do have a problem because you cannot have the theory function divide by 0. For this purpose, we can assume than rhs will never be 0:

Theory((int lhs, int rhs), algebra, multiplication_is_inverse_of_division) {
    cr_assume(rhs != 0);
    cr_assert_eq(lhs, my_div(my_mul(lhs, rhs), rhs));
}

cr_assume will abort the current theory iteration if the condition is not fulfiled.

Running the test at that point will raise a big problem with the current implementation of my_mul and my_div:

[----] theories.c:24: Assertion failed: (a) == (bad_div(bad_mul(a, b), b))
[----]   Theory algebra::multiplication_is_inverse_of_division failed with the following parameters: (2147483647, 2)
[----] theories.c:24: Assertion failed: (a) == (bad_div(bad_mul(a, b), b))
[----]   Theory algebra::multiplication_is_inverse_of_division failed with the following parameters: (-2147483648, 2)
[----] theories.c:24: Unexpected signal caught below this line!
[FAIL] algebra::multiplication_is_inverse_of_division: CRASH!

The theory shows that my_div(my_mul(INT_MAX, 2), 2) and my_div(my_mul(INT_MIN, 2), 2) does not respect the properties for multiplication: it happens that the behaviour of these two functions is undefined because the operation overflows.

Similarly, the test crashes at the end; debugging shows that the source of the crash is the divison of INT_MAX by -1, which is undefined.

Fixing this is as easy as changing the prototypes of my_mul and my_div to operate on long long rather than int.

What's the difference between theories and parameterized tests ?

While it may at first seem that theories and parameterized tests are the same, just because they happen to take multiple parameters does not mean that they logically behave in the same manner.

Parameterized tests are useful to test a specific logic against a fixed, finite set of examples that you need to work.

Theories are, well, just that: theories. They represent a test against an universal truth, regardless of the input data matching its predicates.

Implementation-wise, Criterion also marks the separation by the way that both are executed:

Each parameterized test iteration is run in its own test; this means that one parameterized test acts as a collection of many tests, and gets reported as such.

On the other hand, a theory act as one single test, since the size and contents of the generated data set is not relevant. It does not make sense to say that an universal truth is "partially true", so if one of the iteration fails, then the whole test fails.