Speed up CI

[emilk]

It's time to take another pass on this.

Maybe some tricks can be learned from the CI section of
https://matklad.github.io/2021/09/04/fast-rust-builds.html

For instance, it suggest setting CARGO_INCREMENTAL: 0 in our env to disable incremental compilation, which should shrink the /target folder and maybe improve caching.

A lot of the tricks in #5332 also apply to this, like:

• Doing much less stuff on each PR

[emilk]

• Duplicate of Tracking issue: Speed up CI #3033