Unit Testing Package

FORTPY: Unit Testing

Quickstart

Instead of reading the entire document from start to finish, here is a quick checklist of how to enable unit tests for a subroutine.

1. Create an output template that describes how to compare the output files that will be created by the tests.
2. Alter the fortran code to add a Fortpy XML element (as a comment) on the first line of the output files that need to be compared.
3. Determine what the method needs to run successfully: list any methods that need to be run first, variables that need values initialized etc.
4. Create an XML file with the same name as the fortran code file that contains the method to be tested. Add a testing group to it with tags for the pre-reqs and variables that the method needs to run.

Once those steps have been completed, you can run the unit testing script located in fortpy/scripts/runtests.py. PS: you may need to read some of the documentation after all if you don’t know how to do any of the things in the list. To get you started, here is the tag that needs to be at the top of output files to compare. If it isn’t there, they are assumed to be version 1.

 <fortpy version="2"></fortpy>

Overview

The python package fortpy.testing allows unit tests to be dynamically generated using XML tags that decorate a subroutine or function. This document outlines the various XML tags and attributes available and how they change the behavior of the test generation and execution.

Other more specific documentation for unit testing is at:

• Testing Group Tags and Attributes
• File Comparison Across Versions

Steps Performed to Generate a Test

Whenever the unit testing framework runs, it performs the following in order:

• Parse all modules in the specified directory as well as any dependencies of those modules.
• Look through all the XML doctags to find testing groups.
• If a testing group was found, complete the steps in the next list; otherwise do nothing.

For each module method that has a testing group:

• Create a folder for the module.method.
• Examine the code files that the subroutine or function needs to compile and see which have been modified.
• Copy any new/modified files across to the folder.
• Generate a [test name].f90 fortran program.
• Generate a Makefile for the program with the module in the correct order for dependencies.
• Compile the program; if it compiles run all tests specified in the testing group.

When the fortran program is generated, the framework determines which variables are needed and initializes them based on the directives in the testing group. Any executable dependencies (i.e. methods that need to be run before the one being tested) are executed in the correct order. Results of comparing the output files to the model output files are saved to a filename.results file in the test directory.

<globals> Tag

It is often the case that many unit tests in a module access the same files (for example random seeds). This leads to a lot of duplication in the <group> and <test> tags for each subroutine being unit tested. The <globals> tag allows tags and attributes to be accessible by any unit test in the same XML file. For background information, see Simplifying Unit Testing XML.

Testing Groups as Decorators

The heart of all unit testing is the <group> tag with attribute purpose="testing". Any subroutines/functions without a testing group (i.e. the group tag with purpose set to testing) are ignored by the unit test generator.

The following elements have meaning within the group tag. Each is described in its own section below.

• <prereq> specifies an executable dependency that needs to be run before the unit test will execute.
• <assignment> specifies that an instance of a variable needs to have its value changed or one of its methods run.
• <test> specifies a set of execution tests to run and how to evaluate the results.
• <mapping> changes the default matching of parameter names to local program variables.
• <global> specifies a custom variable that needs to be initialized in the fortran program for use by all the methods in the unit test.

Because testing groups can become large and obstruct the readability of the code, the framework supports writing docstrings in a separate file that has the same name as the fortran code file, but an extension of xml instead of f90. See an example of a testing group and the program it generates.

Additional Parameter Attributes

The unit testing framework gives meaning to these additional attributes when they appear on a <parameter> tag of a method. See the XML Documentation Page for full parameter attribute information.

• range specifies a list of min:max values of the same type as the parameter for each dimension. Right before the method is tested, auto-generated code loops through the variables values to make sure that each one falls within the range.
• random specifies that for unit-testing purposes, the value of the auto-generated variable for this parameter should be random. For a multi-dimensional variable, random values are assigned to each element in the array. If a range is specified, all the random values will fall within the range.
• regular when "true", the framework will automatically create a variable of the same name and type in the calling program's context. If random is not specified, the value will need to be initialized manually using a <prereq> or <instance> tag. If multiple methods define a regular variable with the same name, the framework will croak if they are not of the same type with the same modifiers. In that case, either terminate the pre-req chain or remove the regular from one of the parameters.

When the documentation refers to auto-matching, it means the auto-generation of variables based on the name and type of the <parameter> that had the regular attribute.

NOTE: the range and random attributes haven't been implemented yet.