Separate build and test configuration

[matthew-brett]

#417 has brought up a subject we haven't resolved, which is how to separate build and test configurations.

At the moment, Mac doesn't distinguish; we source the files named in ENV_VARS_PATH and CONFIG_PATH, and both build and test run with those configurations.

Manylinux does distinguish, in that a) we don't pass, or source ENV_VARS_PATH in the test phase, and b) the user could, if they wanted, change CONFIG_PATH before the test phase, to get a different test configuration.

Two questions for the company:

• Should we, in fact, pass and source ENV_VARS_PATH in the Manylinux test phase? This would be a back-compatibility break, but it seems reasonable given we effectively have that behavior on Mac already.
• Should we add a new function called something like init_test_phase, which could switch any configurations. In that function we could:
  • Set an environment variable MB_PHASE to 'test, for use by conditionals in config.sh and similar.
  • Switch CONFIG_PATH and ENV_VARS_PATH to (if present) new environment variables TEST_CONFIG_PATH and TEST_ENV_VARS_PATH.

What do y'all think?

[mattip]

I think these days we should encourage testing outside of multibuild. The point of multibuild is to build wheels, there should be an option to leave testing outside the scope of multibuild. Many CI platforms are better than we can be about getting newer versions of OS and Python out to users. Github actions has nightly builds of CPython and PyPy.