[meta] Plan for Writing Convenient and Distributable Tests

[cwfitzgerald]

The Problems

1. We've witnessed a vast increase in bug reports as a result of the 0.8 release. While most of these were patched quickly due to @kvark and other's effort and diligence, it indicates that our current testing process is not sufficient to catch all the bugs we have.

2. In addition, the backends that are less manually tested tend to have a large number of uncaught bugs. I've heard from many people that they can only run their project on Vulkan or maybe Vulkan/Metal but no others. This also makes it painfully clear that we need a way to reliably test these other backends.

3. Our current wgpu-level tests are extremely incomplete. We primarily use the wgpu-rs examples as a testing platform, so they are polluted with many extraneous features, and running them all on all backends is an extremely tedious job. An additional step in our release testing process involves porting forward many applications and testing them for bugs on the platforms that we have available. This is a very spot-check-based process and has many holes. It is also dependant on the maintainers of the projects to have time to do the porting process.

4. Writing play-tests are a very tedious process because there are many different things that must all align (such as buffer IDs) and generally can't utilize things like helper functions which can simplify the writing of lots of simple tests easily.

5. There are numerous platform-specific issues that are caused by either driver bugs or wgpu bugs. We have no way of catching these as we only regularly test a small set of platforms and APIs.

The Solutions

There are quite a large set of problems that we need to solve. Thankfully there is an incremental path to a solution.

Create Easy-To-Write Easy-To-Modify Testing Infrastructure

While play-tests have their use and I do not recommend getting rid of them, their inconvenience to write makes it difficult to find the time and energy to write new play-tests. As such I propose a new system of testing that programs directly against wgpu-core. While using wgpu-core directly can be a pain, a lot of the operations the tests are doing are repetitive and can easily abstract into helper functions. Once a collection of these functions are built up, adding new tests becomes extremely easy. I will mock up some simple tests in the coming weeks.

Tests can be filtered based on the presence of features/limits/downlevel capabilities. If a test is filtered it will "pass" but print to stdout why it was rejected.

Once we have the infrastructure in place, we should recruit help in writing the tests detailed in #807.

Run Tests on All Backends and GPUs

With the above in place, it should be easy to test all different backends and cards on the system. The ideal testing flow should look something like this.

WGPU_BACKEND=vulkan cargo test
WGPU_BACKEND=dx12 cargo test
..
WGPU_BACKEND=vulkan WGPU_ADAPTER='RX 580' cargo test
WGPU_BACKEND=dx12 WGPU_ADAPTER='RX 580' cargo test

This likely could be automated with a simple helper program which would also be necessary for the final step.

Run Tests in CI

The work set out by #1365 should allow us to test vulkan, dx12, dx11, and opengl in CI. We need to make sure this happens using the above infrastructure to test all the things.

Split Play Tests into Individual Tests

This might be a bit weird, but if we are trying to programmatically extract data from tests, it would be significantly easier if each playtest was its own rust test. We can make each test a call to the play function with a path to the trace. This would be no more difficult than the current solution with listing everything in a ron file.

Build Distributed Testing Network

Once we have a reliable and functional test suite, we should build out a testing network using hardware volunteered by their owners, and given to contributors by the gfx-rs hardware fund. This involves developing a server to manage and view all the tests and the data and clients who run the tests and report the results.

The details of this are out of scope for this issue but will be expanded on when it is time to build these systems.