Acceptance Testing

This module provides a simple acceptance testing library.

Structure

An acceptance test is structured fairly simple. It exposes a way to implement tests that can be run by the bazel test framework. At the core are the TestBase and TestTopogen classes, which serve as base classes for the individual tests. TestTopogen is the base class for tests that operate on a SCION topology generated by the topogen tool, which is the common case. For tests that require a different setup, the TestBase class is used as a base class.

Each test is an individual executable that is executed (either directly or via bazel) with commandline parameters to run the test. Tests in this framework consist of three phases:

• setup
• run
• teardown

When no subcommand is specified, the entire sequence is executed. Using the subcommands for the individual phases, the test setup can easily be reproduced for interactive exploration.

Tests scripts can offer commandline parameters. The base classes define the following switches:

• --artifacts-dir <path>; directory for test artifacts, defaults to environment variable TEST_UNDECLARED_OUTPUTS_DIR
• --executable <name>:<path>; specifies path for an executable used in the test. This can be used to run executables that are built by bazel.
• --docker-image <path>; load the specified container images tars. This can be used to load images that are built by bazel.
• --topo <path>; path to the .topo file for topogen tests
• --setup-params <args>; additional parameters for topogen.

Each test directory contains a BUILD.bazel file with the rules to execute the acceptance test script(s) in this directory. The test framework provides two bazel macros, raw_test and topogen_test, corresponding to the TestBase and TestTopogen base classes respectively.

Writing Your Own Test

Write your own test by adding a directory to acceptance, e.g. acceptance/new_test. Add an executable python file test.py and BUILD.bazel. The test script contains a subclass to TestBase or TestTopogen, where the _run method implements the actual test. Additionally, the methods setup, setup_prepare, setup_start and teardown can be overridden to perform additional setup or cleanup.

Tests can use:

• self.artifacts: the specified directory for test outputs, created during setup.
• self.get_executable(<name>): returns an executable specified using the --executable switch.
• self.dc: a wrapper for docker compose, instantiated during TestTopogen.setup.

The base.main function is the main entry point to run the tests and must be invoked in __main__.

Example

load("//acceptance/common:topogen.bzl", "topogen_test")

topogen_test(
    name = "test",
    src = "test.py",
    topo = "//topology:tiny.topo",
)

#!/usr/bin/env python3

from acceptance.common import base, scion

class Test(base.TestTopogen):
    """
    Fill in the test description here.
    """

    # optional: override this (or setup, setup_start) to do additional setup
    def setup_prepare(self):
        super().setup_prepare()

        # Example:
        # Modify the logging config for all beacon servers
        scion.update_toml({"log.file.level": "debug"}, self.artifacts // "gen/*/bs*.toml")

    def _run(self):
        # run some tests here

    # optional: override this for additional cleanup
    def teardown(self):
        super().teardown()

if __name__ == "__main__":
    base.main(Test)