add test directory with README explaining the benchmarking suite and …

…an initial incflo.tests.ini file

README.md

@@ -34,8 +34,8 @@ Clone and build incflo
 ```shell
 > git clone http://github.com/AMReX-Codes/incflo.git
 > cd exec
-> make -j
-> ./incflo3d.gnu.MPI.ex inputs.double_shear_layer
+> make -j4
+> mpirun -np 4 incflo3d.gnu.MPI.ex inputs.channel_cylinder
 ```
 
 # Contributing

test/README.md

@@ -0,0 +1,96 @@
+# incflo regression tests
+
+Regression tests for incflo are managed using [the regression testing tools that are distributed as part of AMReX](https://github.com/AMReX-Codes/regression_testing).  
+The tools are a custom set of python scripts that check the
+results produced by the code in the remote repository (i.e. not in your local, working copy).  
+For information on how to set up a remote repository, see the Setting Up the Tests section below.
+
+The regression test scripts
+1. "git pull" AMReX and incflo
+2. For each defined test,
+   * build an executable according to compile-time parameters/defines
+   * run in serial or parallel
+   * compare the results (typically in the form of a plotfile) with a
+      "benchmark" reference solution.
+3. Assemble the results of all the tests, formatted in html.
+   Optionally, if any of the tests fail, an email is generated and sent
+   to a specified list of recipients.
+
+A key design feature of the regression suite is that the reference
+solutions can be updated manually at any time.  This is necessary
+when, for example, a bug is discovered or an algorithm change results
+in improved solutions or modified error metrics.  For this reason, the
+initial set of benchmarks needs to be created manually before the first
+test is executed (more info on how to do this in the Setting Up the
+Tests section).
+
+
+## Setting Up the Tests
+
+1. Choose/create an area/directory to run the regression tests.  It is
+recommended to make a special "scratch" area for the exclusive use of
+the regression tester.  This is because the testing suite may optionally
+checkout specific branches or SHA1 commits of the needed repositories,
+and while it always restores the repository to it's original state, it's
+best to let the tester work on its own where it won't risk confusing
+you. We assume REGTEST_SCRATCH is defined to point to this location.
+
+2. Clone the required repos into the scratch area.
+
+    ```
+    git clone https://github.com/AMReX-Codes/amrex.git ${REGTEST_SCRATCH}/amrex
+    git clone https://github.com/AMReX-Codes/regression_testing.git ${REGTEST_SCRATCH}/regression_testing
+    git clone https://github.com/AMReX-Codes/incflo.git ${REGTEST_SCRATCH}/incflo
+    ```
+
+3.  Move to the location where the tests will be built/run, create
+testing area expected by reg test scripts:
+
+    ```
+    cd ${REGTEST_SCRATCH}; mkdir test_data
+    ```
+
+4.  Edit the config file, ${REGTEST_SCRATCH}/incflo/test/incflo-tests.ini
+to set the AMReX and incflo scratch clone locations and desired branch/SHA.
+In that file, also set
+
+     testTopDir =  ${REGTEST_SCRATCH}/test_data/incflo
+
+5. (optional) Create symbolic links 
+    ```
+    ln -s ${REGTEST_SCRATCH}/regression_testing/regtest.py .
+    ln -s ${REGTEST_SCRATCH}/incflo/Test/incflo-tests.ini .
+    ```
+
+6.  Generate the initial benchmark solutions for all the tests listed
+in the .ini configuration file.  Rerunning this at any time will
+overwrite the previous versions of the benchmarks
+
+    ```
+    ./regtest.py --make_benchmarks "<a useful comment>" incflo-tests.ini
+    ```
+
+5. Upon some trigger event, re-run the tests and format the results in
+html.  In this case, the results will appear as
+test_data/incflo/web/index.html
+
+    ```
+    ./regtest.py incflo-tests.ini
+    ```
+
+NOTE: The regtest.py script takes a handy option "--no_update All",
+which instructs the tester to work with the scratch repositories as
+they currently exist.  Without this option specified, the branch of
+each repository that is specified in the .ini config file is "git
+pull"'d to obtain its most recent version; the original state of the
+repositories are restored when the tests complete.  Using this
+feature, a user can checkout any specific branch of any of the
+repositories in the scratch area and run the complete set of tests.  A
+user may wish to do this prior to issuing a "pull request", for
+example.
+
+More information on available options is given by
+    ```
+    ./regtest.py -h
+    ```
+which prints a verbose description of usage and setup. 

test/incflo-tests.ini

@@ -0,0 +1,80 @@
+[main]
+# the parent directory for all the test suite stuff.  
+# The benchmarks and run directories will live under here
+testTopDir     = /path/to/regtest/test_data
+
+# location of the tests' output, written in html
+webTopDir      = /path/to/regtest/test_data
+
+MAKE = make
+sourceTree = C_Src
+# how many simultaneous build jobs (through the -j flag of make) are to
+# be done
+numMakeJobs = 8
+
+# the names of the compilers for C++ and Fortran.  These should be
+# names that the AMReX build system can interpret, as these variables
+# as passed directly to the make command.
+COMP = g++
+FCOMP = gfortran
+
+# additional options added to C++ make command
+add_to_c_make_command = TEST=TRUE USE_ASSERTION=TRUE
+
+# after the test is run and the comparisons are made, do we keep the
+# plotfiles around?  If this is 1, then any plot or checkpoint files,
+# except the one that was used for the comparison will be deleted.
+# Otherwise, all the output files will be tar/gzipped.
+purge_output = 1
+
+# suiteName is the name prepended to all output directories
+suiteName = incflo
+
+reportActiveTestsOnly = 1
+
+# Add "GO UP" link at the top of the web page?
+goUpLink = 1
+
+# email
+sendEmailWhenFail = 0
+# emailTo = user@domain
+# emailBody = incflo regression test failure
+
+# MPIcommand should use the placeholders:
+#   @host@ to indicate where to put the hostname to run on
+#   @nprocs@ to indicate where to put the number of processors
+#   @command@ to indicate where to put the command to run
+#
+# only tests with useMPI = 1 will run in parallel
+# nprocs is problem dependent and specified in the individual problem
+# sections.
+
+#MPIcommand = mpiexec -host @host@ -n @nprocs@ @command@
+MPIcommand = mpirun -np @nprocs@ @command@
+MPIhost = 
+
+# Specify the source code repositories.  Each git repo is
+# given its own section.  
+
+[AMReX]
+# This should be a separate directoy from one you do
+# day-to-day work in, to ensure that there are no conflicts
+# when the test suite does git pulls
+dir = /path/to/regtest/amrex
+branch = development
+
+[source]
+dir = /path/to/regtest/incflo
+branch = development
+
+# individual problems follow
+
+[double_shear_layer] 
+buildDir = exec
+inputFile = inputs.double_shear_layer
+dim = 3
+restartTest = 0
+useMPI = 1
+numprocs = 8
+compileTest = 0
+doVis = 0