Improve testing architecture

[ludfjig]

We have reached a point where we have a pretty large matrix of tests and PR tests might take too long (even more w/ adding different skews for Intel/AMD). We should consider improving the tests (running unit tests only on Linux/Windows), and maybe adding merge queues back again.

In addition, we should think about which combination of cargo features we want to test compile and/or test the entire test suite with, and when we want to run these.

[danbugs]

Consider #204 w/ this issue