PSI/J comes with a testing infrastructure that allows users to contribute tests of the PSI/J library itself on resources they have access to and where running such tests is appropriate.
Contributed tests take the form of normal pytest runs for which results are uploaded to a test aggregation service. The compatibility of various PSI/J branches with various resources can then be assessed and is available at https://testing.exaworks.org
There are a number of ways to run tests. Invoking pytest
directly,
running the integration tests, and through cron
(or a similar tool).
This is the preferred way of running tests since it allows the PSI/J team to keep a constant eye on the state of the library on various resources. To set up the Cron job (or an alternative method), you can either use the provided setup script:
./psij-ci-setup
or manually set up the CI runner (psij-ci-run
) with Cron or your
favorite scheduler.
Note: If you need to set up an environment module (such as module load python/cpython-x.y.z
) or something similar, such as loading a conda or
virtual environment, please run the relevant commands before invoking
psij-ci-setup
.
Note: To upload data to the test server, you will need an authentication key. You can obtain such a key from https://testing.psij.io/auth.html. Obtaining a key requires a valid email.
The CI runner is a convenience script which can clone the PSI/J repository and run tests on several git branches. To use the CI runner:
- Install dependencies:
pip install --user -r requirements-dev.txt
-
Edit
testing.conf
and customize as needed. -
Run tests:
./psij-ci-run
or, alternatively, using built-in make
target:
make ci-tests
This is the most direct way to run the tests and it must be done from the main psij directory. Use the following steps:
- Install dependencies:
pip install --user -r requirements-dev.txt
- Run tests:
PYTHONPATH=$PWD/src pytest tests
or, alternatively, using the built-in make
target:
make tests
A number of custom options are available for tests. They are described in
detail in the testing.conf
file. In order to use these options with
pytest
, they can be passed as double-dashed command line arguments with
underscores converted to hyphens. For example, to run tests and
automatically upload results to the default aggregation server, you can
run:
PYTHONPATH=$PWD/src pytest --upload-results tests
or, using make
, prefix all options with a double dash ("--"):
make tests -- --upload-results
Care must, however, be taken since it is impossible to preserve
whitespace and certain special characters (such as the double quotes)
when passing arguments through make
.