This is the documentation of the MonetDB regression testing environment.

For further information about the automatic nightly multi-platform regression
testing of MonetDB, please also see
http://monetdb.cwi.nl/Development/TestWeb/index.html


- The testing environment consists of the following tools:
  + Mtest.py	a python script to run tests
  + Mdiff	a C program to show the differences of two files as HTML document
  + Mapprove.py	a python script to approve recent test output (see below)

  When called with command-line option "--help", each of these tools
  provides the respective usage information.


- In each directory of all MonetDB-related source trees, tests may be
  provided in a subdirectory called "Tests". Of course, the tests should
  deal with the part/modules where they are located and they should be
  provided and maintained by the respective developer (see also below).


- Each "Tests" directory must contain a file "All" that contains the names
  of all tests in that directory (one name per line).


- Each test named TST consists of
  + a test script which is ONE of the following:
    * an arbitrary executable (e.g., a shell script)        (TST.sh)        (1,3)
       on Windows, executables must be ".(exe|com)"         (TST.(exe|com)) (1,3)
                   and scripts must be ".(bat|cmd)"         (TST.(bat|cmd)) (1,3)
    * a Python script                                       (TST.py)        (1,3)
    * a MAL script to be executed by mclient -lmal          (TST.malC) (2,3,4)
    * a SQL script to be executed by mclient -lsql          (TST.sql)  (2,3,4)
    * a sqllogictest-compatible test script                 (TST.test)
   (1) In case the name of an arbitrary executable or Python script is
        suffixed with ".MAL" (i.e., TST.MAL[.(sh|exe|com|bat|cmd|py)]),
        Mtest.py starts an mserver5 with a MAL-listerner on MAPIPORT
        in the background before executing the test. The test can then
        connect via MAPI (on MAPIPORT) to that mserver5.
       Similar,
       with suffix ".SQL", an mserver5 with an SQL-listener on
        MAPIPORT is started;
       In both cases, Mtest.py stops the mserver5 again, once the test has
        finished.
   (2) For mclient, if several files are present named
        (1) TST_sXX.(malC|sql) (XX={00,01,...,99})
            these are executed sequentlially using the same mserver5
        (2) TST_pXX.(malC|sql) (XX={00,01,...,99})
            these are executed concurrently using the same mserver5
   (3) For each test file TST[.*] involved,
        if a file called
	TST[.*].src  exists instead of  TST[.*],
	TST[.*].src  is expected to contain a single line
	giving the original location of the test file to be used.
   (4) For each test file TST.* involved,
        if a file called
	TST.*.in  exists instead of  TST.*,  all environment variables in
	TST.*.in  are replaced by their current value
	when copying  $TSTSRCDIR/TST.*.in  to  $TSTTRGDIR/TST.* .
  + for frontend tests with MonetDB5 (mserver5), extra options with
    which mserver5 is started
     (optional)                                             (TST.options5) (4,5)
  + a file that contains a list of modules (one per line)
     required by the test  (optional)                       (TST.modules)  (4,5)
  + a file that contains a list of tests (one per line) that must
     have run successfully prior to the test (optional)     (TST.reqtests)
  + a file that if it exists indicates that the --forcemito flag
     must not be passed to the server                       (TST.nomito)
  + a set of files used by the arbitrary executable
     (optional)                                              (TST.*)       (4,5)
  + stable (i.e. correct) versions of stdout and stderr
     of the test                                            (TST.stable.{out,err})
    operating system (OS) specific stable output can be
    provided by adding the suffix ".<OSname>" (".`uname`"),
    ".<OSname><OSversion>" (".`uanme``uname -r`"), or
    ".<OSname><OSversion>.(64|32)bit"
    to the respective filename
  + a file that contains a single integer number which is used as a factor
    to extend the default TIMEOUT for this test, only.      (TST.timeout)

  Using this naming conventions, the test environment can automatically
  decide what to do when called as "Mtest.py TST" (see below), i.e. execute
  one of the following:
  * TST[.sh|.exe|.com|.bat|.cmd] TST
  * python TST.py
  * mserver5 --dbpath=GDK_DBFARM/TSTDB --set mapi_port=$MAPIPORT "[<TST.options5]" &
         mclient -lmal < TST[_(s|p)XX].malC
     or  TST.MAL.(sh|exe|com|bat|cmd) TST
     or  python TST.MAL.py
  * mserver5 --dbpath=GDK_DBFARM/TSTDB --set mapi_port=$MAPIPORT "[<TST.options5]" &
         mclient -lsql < TST[_(s|p)XX].sql
     or  TST.SQL.(sh|exe|com|bat|cmd) TST
     or  python TST..SQL.py

  (mserver5 is called with "--debug=$GDK_DEBUG $setMONETDB_MOD_PATH --set monet_prompt=".
   See below and "mserver5 --help" for details.)

  (mclient is called with "--port=$MAPIPORT".
   See "mclient --help" for details.)


- Mtest.py uses two directory trees based at TSTSRCBASE and TSTTRGBASE,
  respectively. These trees have similar purpose as the SOURCE and PREFIX
  trees when configuring and compiling MonetDB: the original tests are found
  in TSTSRCBASE and Mtest.py writes the test output to TSTTRGBASE.
  The default settings are as follows:
	TSTSRCBASE=`<PACKAGE>-config --source`  (where you checked out <PACKAGE>'s source tree)
	TSTTRGBASE=`<PACKAGE>-config --prefix`  (where you told configure to install <PACKAGE>)
  You can overrule any of these defaults by setting the respective
  environment variable, or by giving, e.g., --TSTTRGBASE=/tmp as command
  line option to Mtest.py.

  IMPORTANT NOTE:

  In any case, all directories must be given as ABSOLUTE PHYSICAL PATHS,
  i.e., starting with "/" on Unix (or "[<drive-letter>:]\" on Windows, and
  not containing any symbolic link(s).
  The latter is a restriction that might be released in the future, but for
  the time being, using paths that contain symbolic links will cause
  problems with Mtest.py.
  Hence, if you compiled MonetDB in a path that contains a symbolic link,
  e.g., ~/dev/monet -> /net/myhost/export/scratch1/myname/monet, you should
  overrule the default settings of TSTSRCBASE and TSTTRGBASE (e.g.) as
  follows
	TSTSRCBASE=/net/myhost/export/scratch1/myname/monet
	TSTTRGBASE=/net/myhost/export/scratch1/myname/monet/$SYSTVER
  either by setting the respective environment variables or by using the
  respective command line options to Mtest.py (see above).


- Each test is identified by its name and its directory path relative to
  TSTSRCBASE, but without the trailing "/Tests".


- Running one or several tests is done by simply calling
  + "Mtest.py <names...>" to run tests <names...> in
                          a) ".",       if "." is a "Tests"-directory and
                                           there is an "All"-file in "."
                          b) "./Tests", if "./Tests/All" exists
  + "Mtest.py -r"         to run all tests found in any subdirectory of "."

  If Mtest.py is not called in (a subdirectory of) $TSTSRCBASE, Mtest.py
  changes ("cd's") to $TSTSRCBASE before starting its work.
  For more details call "Mtest.py --help".


- Environment variables:
  + Mtest.py doesn't need any special environment variables to be set.
  + However, the following environment variables can be set to overrule the
    default settings:
	name		default					description
	----		-------					-----------
	TSTSRCBASE	`<PACKAGE>-config --source`		base of test source tree
	TSTTRGBASE	`<PACKAGE>-config --prefix`		base for test output
	MALCLIENT	mclient -lmal -ftest -tnone		mal-client program	`)
	SQLCLIENT	mclient -lsql -ftest -tnone		sql-client program	`)
	SQLDUMP	        msqldump				sql-dump program	`)

    `) Alternative MAL & SQL client programs need to accept (or ignore)
       "--config=", "--host=", & "--port=" options.

    The following variables are currently still used legacy, but they will disappear soon...
	MONETDB_MOD_PATH	`<PACKAGE>-config --modpath([45])`

    The setting of all these environment variables may be overruled
    by commandline options with the same names, e.g., "--MALCLIENT=mclient.py".

    NOTE:
    All paths must be given as ABSOLUTE PHYSICAL PATHS (see "IMPORTANT NOTE"
    above).

  + Further environment variables that Mtest.py exports (to be used in test
    scripts) are:
  	name		description
	----		-----------
	HOST		hostname
	SYST		`uname`
	SYSTVER		`uname``uname -r`
	MDIFF		"Mdiff"
	MSERVER		"mserver5 --config=$MONETDB_CONF --debug=$GDK_DEBUG $setMONETDB_MOD_PATH $setGDK_DBFARM --set mapi_port=$MAPIPORT --set monet_prompt="
	MAL_CLIENT	"$MALCLIENT --config=$MONETDB_CONF --host=$HOST --port=$MAPIPORT"
	SQL_CLIENT	"$SQLCLIENT -u monetdb -P monetdb --host=$HOST --port=$MAPIPORT"
	SQL_DUMP	"$SQLDUMP -u monetdb -P monetdb --host=$HOST --port=$MAPIPORT"
	MAPIPORT	the MAPI   port (random number between 30000 and 39999)
	TST		name of the current test
  	TSTDB		name of default test database, unique for each directory
	TSTDIR		current test directory without $TSTSRCBASE/ & /Tests
	TSTSRCDIR	$TSTSRCBASE/$TSTDIR/Tests
	TSTTRGDIR	$TSTTRGBASE/mTests/$TSTDIR
	GDK_DBFARM	as read from $MONETDB_CONF or set by --dbfarm=<path>
	GDK_DEBUG	debug bitmask for mserver5 (default: 10)
	MONETDB_MOD_PATH	new module search path of mserver5 (if set via --monet_mod_path=<pathlist>)
	setMONETDB_MOD_PATH	"--set monet_mod_path=$MONETDB_MOD_PATH"	(if --monet_mod_path=<pathlist> is used)
	setGDK_DBFARM	"--set gdk_dbfarm=$GDK_DBFARM"	(if --dbfarm=<path> is used)


- For each directory, Mtest.py creates a database $TSTDB in $GDK_DBFARM
  before running any test.


- A test is skipped (i.e. not executed),
  if $TST.modules exists and contains modules that are not available, or
  if $TST.BATs exists and contains BATs that are not available in the
  current database $TSTDB.


- While executing test "$TSTSRCDIR/TST", Mtest.py uses $TSTTRGDIR as working
  directory; for convenience, all files "$TSTSRCDIR/TST*" are copied to
  $TSTTRGDIR.


- Mtest.py stores the output of test "$TSTSRCDIR/TST" in
  $TSTTRGDIR/TST.test.{out,err} and checks it against
  $TSTSRCDIR/TST.stable.{out,err}. If system specific stable output files
  ($TSTSRCDIR/TST.stable.{out,err}[.($SYST|$SYSTVER|$DIST|$DISTVER)][.((64|32)bit|$ARCH)][.int128][.single])
  exist, the most specific stable output is used. Mtest.py stores the
  respective Mdiff results in $TSTTRGDIR/TST.{out,err}.diff.html.
  For each test directory, a common entry point to the test results is
  created as $TSTTRGDIR/.index.html.
  Here, $DIST is the name of the distribution (e.g. Fedora, Ubuntu,
  Windows), $DISTVER is the name of the distribution with the
  distribution version number, and $ARCH is the CPU architecture as
  reported by mserver5 on startup ("Compiled for").  The other
  variables are as described above.
  For the whole test run, a global entry point to the test results is
  created as $TSTTRGBASE/mTests/index.html.


- Mfilter.py is used to filter out variable date form the test output to
  avoid no-error differences when running Mdiff. Mfilter.py prefixes all
  lines enclosed in two line containing only "#~BeginVariableOutput~#" and
  "#~EndVariableOutput~#", respectively, with "#~". These lines are then
  ignored during Mdiff.

  MAL Example:
	io.printf("#~BeginVariableOutput~#\n");
	io.print(somebat);
	io.printf("#~EndVariableOutput~#\n");


- A tool Mapprove.py is provided to copy the recent output from
  $TSTTRGDIR/TST.test.{out,err} to $TSTSRCDIR/TST.stable.{out,err}. Lines
  starting with a '!' (i.e. error messages) are eliminated during copying.
  Of course, $TSTTRGDIR/TST.test.{out,err} have to be checked and claimed
  correct before being approved!
  See "Mapprove.py -?" for command-line options.


- A development session is then supposed to proceed as follows:
  (assuming TSTSRCBASE=`<PACKAGE>-config --source`)

	cd $TSTSRCBASE
	run  hg pull --update
	loop
		loop
			loop
				# write & compile your code
				cd $TSTSRCBASE/$TSTDIR
				edit source
				cd `<PACKAGE>-config --build`/$TSTDIR
				[g]make
				[g]make install
			endloop
			loop
				# write, adapt, run your tests ...
				cd $TSTSRCBASE/$TSTDIR
				edit/create  Tests/All
				edit/create  Tests/<names...>.*
				run  Mtest.py <names...>
				# ... and check their results
				check 	$TSTTRGDIR/<names...>.test.{out,err}
					$TSTTRGDIR/<names...>.{out,err}.diff.html
					(cf. $TSTTRGBASE/mTests/index.html)
			endloop
			# approve new/modified correct output of your tests
			run  Mapprove.py <names...>
		endloop
		# run all MonetDB tests...
		cd $TSTSRCBASE
		run  Mtest.py -r
		# ... and check their results
		check $TSTTRGBASE/mTests/index.html
	endloop
	# approve "intended" impact of your changes on other tests' output
	run  Mapprove.py <names...>
	# checkin all correct new stuff to hg
	run  hg commit
	run  hg push


--
| Stefan Manegold     | mailto:Stefan.Manegold@cwi.nl |
| CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
| 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       |
| The Netherlands     | Fax : +31 (20) 592-4312       |