[TESTING] Fix the error when running tests with default targets

[tqchen]

The previous testing.py causes problem of not being able to run the test with the default set of targets.

We might also want to update the code further to memoize the list of targets in the first run

cc @tkonolige @tmoreau89 @jroesch

[tkonolige]

Which target strings is this failing with?

It seems like we want to test if the full target string is enabled with tvm.runtime.enabled. For example llvm -mtriple=nonexistant should not be enabled (but it will be with this patch).

[tqchen]

opencl -device=xyz will fail on my local mac when running tests.

This happens when I directly run python tests/python/integration/test_ewise.py

There is a difference between whether a target is enabled(can generate code), versus whether a runtime is available(can run). Most of the runtime enabled code only takes the target_kind as input without the additional strings.

For runtime availability testing, we should always use the target kind for now. We are also in the process of upgrading the Target specifications, so we might want to look into that later.

But my take is that most of the cases should be about runtime availability.

[tkonolige]

This test fails: TVM_TEST_TARGETS="llvm -mtriple=nonexistant" python3 -m pytest tests/python/relay/test_op_level2.py.

Many of the tests used to use tvm.runtime.enabled on the full target string before #6346, which is why I kept the check on the full string. I think the main problem is that checking if something is enabled is not consistent between target/devices.

[tqchen]

It is a bit confusing to do runtime.enabled("llvm -mtriple=nonexistant") test.

Because llvm -mtriple=nonexistant is a target(whether we can generate code for the target), but not necessarily mean that the particular runtime is enabled. e.g. we can generate ARM code on an x86 machine, or opencl code in a machine without opencl devices. But it does not mean that we can run the code on the same env.

In most of the testcases we are interested right now, we are asking about whether a runtime is enabled(so we can run the generated code). I agree that we might want to improve the logic further. Or modify the testing logic, to use TVM_TESTING_RUNTIME and only include cpu/gpu/opencl/metal/ variations for now.

[tqchen]

That is also why in the original test cases the LLVM arm code generator tests are done in a different way(always enabled because llvm covers these cases), and we don't run the code, but instead just inspect the generated asms

[tkonolige]

I think this change is fine. The real problem seems to be a conflation of targets and devices in a lot of the tests. I think we might also have to improve tvm.runtime.enabled and tvm.context().exist. It appears that tvm.context("llvm -mtriple=something").exist is not accurate.