Test Suite for Array API Compliance

This is the test suite for array libraries adopting the Python Array API standard.

Note the suite is still a work in progress. Feedback and contributions are welcome!

Quickstart

Setup

To run the tests, install the testing dependencies.

$ pip install -r requirements.txt

Ensure you have the array library that you want to test installed.

Specifying the array module

You need to specify the array library to test. It can be specified via the ARRAY_API_TESTS_MODULE environment variable, e.g.

$ export ARRAY_API_TESTS_MODULE=numpy.array_api

Alternately, change the array_module variable in array_api_tests/_array_module.py line, e.g.

- array_module = None
+ import numpy.array_api as array_module

Run the suite

Simply run pytest against the array_api_tests/ folder to run the full suite.

$ pytest array_api_tests/

The suite tries to logically organise its tests. pytest allows you to only run a specific test case, which is useful when developing functions.

$ pytest array_api_tests/test_creation_functions.py::test_zeros

What the test suite covers

We are interested in array libraries conforming to the spec. Ideally this means that if a library has fully adopted the Array API, the test suite passes. We take great care to not test things which are out-of-scope, so as to not unexpectedly fail the suite.

Primary tests

Every function—including array object methods—has a respective test method. We use Hypothesis to generate a diverse set of valid inputs. This means array inputs will cover different dtypes and shapes, as well as contain interesting elements. These examples generate with interesting arrangements of non-array positional arguments and keyword arguments.

Each test case will cover the following areas if relevant:

• Smoking: We pass our generated examples to all functions. As these examples solely consist of valid inputs, we are testing that functions can be called using their documented inputs without raising errors.

• Data type: For functions returning/modifying arrays, we assert that output arrays have the correct data types. Most functions type-promote input arrays and some functions have bespoke rules—in both cases we simulate the correct behaviour to find the expected data types.

• Shape: For functions returning/modifying arrays, we assert that output arrays have the correct shape. Most functions broadcast input arrays and some functions have bespoke rules—in both cases we simulate the correct behaviour to find the expected shapes.

• Values: We assert output values (including the elements of returned/modified arrays) are as expected. Except for manipulation functions or special cases, the spec allows floating-point inputs to have inexact outputs, so with such examples we only assert values are roughly as expected.

Additional tests

In addition to having one test case for each function, we test other properties of the functions and some miscellaneous things.

• Special cases: For functions with special case behaviour, we assert that these functions return the correct values.

• Signatures: We assert functions have the correct signatures.

• Constants: We assert that constants behave expectedly, are roughly the expected value, and that any related functions interact with them correctly.

Be aware that some aspects of the spec are impractical or impossible to actually test, so they are not covered in the suite

Interpreting errors

First and foremost, note that most tests have to assume that certain aspects of the Array API have been correctly adopted, as fundamental APIs such as array creation and equalities are hard requirements for many assertions. This means a test case for one function might fail because another function has bugs or even no implementation.

This means adopting libraries at first will result in a vast number of errors due to cascading errors. Generally the nature of the spec means many granular details such as type promotion is likely going to also fail nearly-conforming functions.

We hope to improve user experience in regards to "noisy" errors in #51. For now, if an error message involves _UndefinedStub, it means an attribute of the array library (including functions) and it's objects (e.g. the array) is missing.

The spec is the suite's source of truth. If the suite appears to assume behaviour different from the spec, or test something that is not documented, this is a bug—please report such issues to us.

Configuration

By default, tests for the optional Array API extensions such as linalg will be skipped if not present in the specified array module. You can purposely skip testing extension(s) via the --disable-extension option, and likewise purposely test them via the --enable-extension option.

The tests make heavy use Hypothesis. You can configure how many examples are generated using the --max-examples flag, which defaults to 100. Lower values can be useful for quick checks, and larger values should result in more rigorous runs. For example, --max-examples 10000 may find bugs where default runs don't, but will take a much longer time.

Contributing

Remain in-scope

It is important that every test only uses APIs that are part of the standard. For instance, when creating input arrays you should only use the array creation functions that are documented in the spec. The same goes for testing arrays—you'll find many utilities that parralel NumPy's own test utils in the *_helpers.py files.

Tools

Hypothesis should always be used for the primary tests, and can be useful elsewhere. Effort should be made so drawn arguments are labeled with their respective names. For st.data(), draws should be accompanied with the label kwarg i.e. data.draw(<strategy>, label=<label>).

pytest.mark.parametrize should be used to run tests over multiple arguments. Parameterization should be preferred over using Hypothesis when there are a small number of possible inputs, as this allows better failure reporting. Note using both parametrize and Hypothesis for a single test method is possible and can be quite useful.

Error messages

Any assertion should be accompanied with a descriptive error message, including the relevant values. Error messages should be self-explanatory as to why a given test fails, as one should not need prior knowledge of how the test is implemented.

Generated files

Some files in the suite are automatically generated from the spec, and should not be edited directly. To regenerate these files, run the script

./generate_stubs.py path/to/array-api

where path/to/array-api is the path to a local clone of the array-api repo. Edit generate_stubs.py to make changes to the generated files.

Future plans

Keeping full coverage of the spec is an on-going priority as the Array API evolves.

Additionally, we have features and general improvements planned. Work on such functionality is guided primarily by the concerete needs of developers implementing and using the Array API—be sure to let us know any limitations you come across.

• A dependency graph for every test case, which could be used to modify pytest's collection so that low-dependency tests are run first, and tests with faulty dependencies would skip/xfail.

• In some tests we've found it difficult to find appropaite assertion parameters for output values (particularly epsilons for floating-point outputs), so we need to review these and either implement assertions or properly note the lack thereof.