Create a new interface for tests

[jsolbrig]

Requested Update

Description

We should consider adding a new interface for integration tests called integration_tests and convert the current integration test repositories to be plugins for integration_tests.

Background and Motivation

Currently, we have a number of different integration test repositories. They contain data that is used for testing GeoIPS functionality and must be installed to a common directory that is referenced via the GEOIPS_TESTDATA_DIR environment variable.

Additionally, the test scripts that go with the test data are not housed with the test data. They are housed in the main GeoIPS repository. This makes it difficult to determine which test scripts go with which data.

Proposed Solution

I would like to create a new interface called integration_tests that would load plugins from the geoips.integration_tests entry point. Associated plugins would need to implement a Plugin module similar to other interfaces (see #78 for current configuration). The plugin module could perform the tests itself using Python or it could call an external script and examine its exit status.

We should also implement setup.py for the integration test repositories to allow it to be installable and discoverable as a GeoIPS plugin repository.

Note that this may need to wait until we have completed implementation of data file discovery for non-python files (#3). How the plugin itself is implemented may depend on the status of #75.

When implementing, we should also implement a CLI command for integration-tests or something similar that allows us to run installed integration tests at the command line.

Alternative Solutions

Status-quo?

Checklist for Completion

• Wait for Set up YAML files to work with package_data in setup.py for geoips and plugins #3 to be complete
• Add a new IntegrationTestsInterface class
• Add new Plugin classes as needed, depending on the status of Consider how to make Plugin classes work more smoothly #75
• Migrate tests scripts to the test data repositories
• Implement Python-based test for each test script
• Implement setup.py
• Test installing and running tests
• Consider implementing CLI for running integration tests

[mindyls]

Note we decided not to store the test scripts and test outputs with the test data because the test data is fairly static and does not require frequent updates (nor does it really need to be tracked).

We should think about it before we start storing files that we expect to periodically change with very large static datasets

[jsolbrig]

This is superseded by #513, #514, and #515