Background

Automated testing is an important part of a software development lifecycle. It is used to automatically catch bugs and regressions both during development and prior to release or deployment. GeoIPS currently makes use of three layers of testing:

1. Linting: Checks to ensure that code conforms to the GeoIPS code standard.
2. Unit testing: Light-weight tests for individual “units” of code (e.g. functions) to ensure that existing code units continue to work correctly.
3. Integration testing: Expensive full system tests to ensure that the products output by GeoIPS don’t change.

Current State

GeoIPS testing suite should be improved to ensure the stability and usability of the holistic system. We will assess the current state each of the three testing layers to determine what updates should be made.

Unit testing is currently severely lacking. It covers only a small fraction of the code and will require significant development to become useful. GeoIPS was not originally built with unit testing in mind and retroactively adding unit tests is a difficult, but important, task. We have begun adding unit tests as a part of our development workflow, but a more sustained effort is required to construct a reasonable architecture for unit tests, achieve better code coverage, and better support the development process.

Integration testing is currently fairly comprehensive. It is comprised of a set of scripts that produce a variety of products to test GeoIPS from end-to-end. The resulting products are compared, pixel-by-pixel, against pre-generated versions of the same products. However, the current test suite is a cumbersome and would be improved by creating a new “tests” plugin interface and reconstructing the tests as plugins. This would enable easy calling of tests from all plugin packages without special knowledge and allow development of efficiency profiling and coverage testing to better ensure that updates don’t cause GeoIPS to become less efficient.

Technical Approach

• Integration testing

  • Investigate methods to improve performance of integration tests. For example:
    • Research and implement methods of parallelizing the tests either via internal parallelization or by running the individual tests in parallel.
    • Research and develop methods of automatically producing and using smaller test datasets rather than complete datasets.
  • Create a new tests plugin interface for GeoIPS.
  • Convert current tests to plugins.
  • Add code profiling to help identify regressions in efficiency.
  • Assess code coverage from integration tests and add tests as needed to ensure more complete coverage.
  • Add documentation describing how to construct integration tests for new product types.

• Unit testing

  • Assess currently available unit testing tools to determine whether Pytest is the appropriate tool for GeoIPS.
  • Research unit test structures and methods employed by other Python packages.
  • Use lessons learned from other packages to formalize and document the structure and methods employed in the GeoIPS unit testing suite.
  • Update the pull-request review process to require that unit tests are added for all new features.
  • Identify and implement automated methods to ensure that unit tests are added when required. For example, require that test coverage doesn’t drop due to a new pull-request.

• While developing major new features like the order-based procflow and the CLI, refactor functions as needed to implement unit tests.

Top-level issue for this milestone: #513