MiniZinc Testing

Setup

Requires Python 3 (on Windows 3.8 is required). Make sure you're in the tests/ directory.

pip install -r requirements.txt

Running Test Suite

To run the full test suite:

pytest

An HTML report will be generated at output/report.html.

Use the --solvers option to specify a subset of solvers to use in the tests. Tests which use a solver not in this list will be skipped.

pytest --solvers gecode,chuffed

Use the --driver option to specify the directory containing the minizinc executable. Otherwise, default directories and PATH will be used to locate it.

pytest --driver=../build

Multiple test suites

To facilitate running the test suite with different minizinc options, specs/suites.yml contains configurations for running tests, or a subset of tests using different options.

my-test-suite: !Suite
  includes: ['*'] # Globs for included .mzn files
  solvers: [gecode, chuffed] # The allowed solvers (if the test case itself specifies a different solver it will be skipped)
  strict: false # Allow tests to pass if they check against another solver (default true)
  options:
    -O3: true # Default options to pass to minizinc (merged and overwritten by individual test cases)

For example, to run the optimize-2 and no-mip-domains configurations only:

pytest --suite optimize-2 --suite no-mip-domains

Creating/editing test cases with the web interface

A web interface can be used to create and edit test cases graphically. Python 3.8.1 or newer and Flask is required.

pip install flask

Start the web interface with

cd tests
python -m minizinc_testing.create

Then open http://localhost:5000.

The web interface detects .mzn files in the tests/spec directory. To add a new test, simply create a .mzn containing the model, and then open the file in the web interface. Test cases can then be generated accordingly.

Creating/editing test cases manually

The test cases are defined using YAML inside the minizinc .mzn files. This YAML definition must be inside a block comment like the following:

/***
!Test
expected: !Result
  solution:
    x: 1
***/

Multiple cases can be specified for one .mzn file:

/***
--- !Test
  ...
--- !Test
  ...
--- !Test
  ...
***/

YAML format

The format of the test case spec is as follows:

!Test
solvers: [gecode, cbc, chuffed] # List of solvers to use (omit if all solvers should be tested)
check_against: [gecode, cbc, chuffed] # List of solvers used to check results (omit if no checking is needed)
extra_files: [datafile.dzn] # Data files to use if any
options: # Options passed to minizinc-python's solve(), usually all_solutions if present
  all_solutions: true
  timeout: !Duration 10s
expected: # The obtained result must match one of these
- !Result
  status: SATISFIED # Result status
  solution: !Solution
    s: 1
    t: !!set {1, 2, 3} # The set containing 1, 2 and 3
    u: !Range 1..10 # The range 1 to 10 (inclusive)
    v: [1, 2, 3] # The array with 1, 2, 3
    x: !Unordered [3, 2, 1] # Ignore the order of elements in this array
    _output_item: !Trim |
      trimmed output item
      gets leading/trailing
      whitespace ignored 
- !Error
  type: MiniZincError # Name of the error type
  message: Exact error message # Exact error message string (avoid using this as it's generally not portable)
  regex: .*type-inst must be par set.* # Regex the start of the string must match (run with M and S flags)

For a test to pass, at least one expected result must be a subset of the obtained result. That is, the obtained result can have more attributes, but not less, and corresponding attributes must match.

If a solution is produced that does not match any given expected output, the result is checked using another solver. If this check passes then the test passes with a warning.

Multiple solutions

When setting all_solutions: true and the order of the returned solutions often does not matter, use !SolutionSet for the list of solutions:

!Result
  status: ALL_SOLUTIONS
  solution: !SolutionSet
  - !Solution
    x: 1
  - !Solution
    x: 2

Testing FlatZinc output

Use type: compile on a test to enable only flattening. Then !FlatZinc filename.fzn to give files with expected results.

!Test
solvers: [gecode]
type: compile
expected: !FlatZinc expected.fzn

TODO

• Tool for generating test cases
• Tweak YAML parsing so not so many !Tags are required
• Better documentation of how the framework operates