[SYCL][NFC] Drop on-device tests in favor of llvm-test-suite

[sergey-semenov]

The original motivation for introducing on-device tests was two-fold:

• Provide a way to test product changes together with E2E test changes
  for features in active development, these tests would then be moved to
  llvm-test-suite once the feature is considered stable. Frequently,
  instead of using on-device tests (or in case an existing llvm-test-suite
  test has to be modified) developers would create a pull request in each
  repo, link them via a comment, then they would be either merged
  simultaneously (while relying on local testing and intel/llvm post-commit
  results for the modified/added tests) or one after the other.
• As a way to have some sanity E2E checks in intel/llvm. On-device
  tests haven't been used in this manner, we've been relying on component
  tests instead.

This patch drops in-tree on-device tests in favor of llvm-test-suite
in order to reduce in-tree test running time.

[bader]

Thanks!
I think we should continue working on make it easier to run llvm-test-suite tests in local work spaces.

[vladimirlaz]

We should not drop on-device lit infrastructure as it is used for internal testing.
Also while developing of features for device code it is really handy to have atomic change for product/tests. Removing this functionality look like a regression. It is especially important for CUDA or ROCm backends.

It is developers responsibility to move tests to E2E test suite once feature development finished.

[bader]

We should not drop on-device lit infrastructure as it is used for internal testing.

What do you mean by "internal testing"? Are there any tests other than removed in this PR use this infrastructure?

Also while developing of features for device code it is really handy to have atomic change for product/tests. Removing this functionality look like a regression. It is especially important for CUDA or ROCm backends.

I don't think so. According to my observations most of the developers contribute to llvm-test-suite bypassing "on-device" tests. Maintaining another infrastructure and machine infrastructure to run them doesn't sounds like a reasonable choice to me.

It is developers responsibility to move tests to E2E test suite once feature development finished.

If developers has configured environment for running "on-device" tests, they can easily run E2E tests as well, so it's better to make it easy to integrate E2E tests to developer's workflow than support two different test suites.

[vladimirlaz]

1. There is some internal testing which uses this infra. They can be disabled afterwards.
2. It brings risks of failures in post-commit for CUDA/ROCm platforms which are not widely used.
3. The idea is that these tests are run in CI on all supported platforms. No need to use several local builds to test on different backends. There is no way to run uncommitted test changes with uncommitted product change.

[bader]

1. There is some internal testing which uses this infra. They can be disabled afterwards.

I suggest move them to llvm-test-suite as it's done with "on-device" tests.

2. It brings risks of failures in post-commit for CUDA/ROCm platforms which are not widely used.

ROCm is not tested yet, so there are no risks added to existing risks.
CUDA: half of the existing few tests are disabled for CUDA and we run only 17 tests.
I'm sure that risk can be mitigated by adding unittests if they do not cover tested functionality already.
Another alternative, run llvm-test-suite in pre-commit to reduce the risk to 0.

3. The idea is that these tests are run in CI on all supported platforms. No need to use several local builds to test on different backends. There is no way to run uncommitted test changes with uncommitted product change.

The idea behind LLVM projects in-tree tests is that they should not have extra dependencies like specific HW or SW. We are trying to align with such requirement and move tests requiring GPU/FPGA HW and OpenCL/CUDA driver to llvm-test-suite.
check-sycl target should be able to validate SYCL runtime library on arbitrary system with minimum requirements for LLVM projects like python + C++ runtime libraries.
If your intention to test execution on specific platform, I suggest running llvm-test-suite instead.

[vladimirlaz]

https://github.com/intel/llvm/blob/sycl/CONTRIBUTING.md should be also updated to remove reference to deleted infra