So, by and large, compiletest is untested, as far as I know. That is -- there are no 'self tests' to make sure that it's acting as it should, invoking revisions the way we expect, and so forth. This could be a bit of a tricky thing to do, but it's definitely worth the effort.

For example, an early version of #47605 had a subtle bug where it appeared to be working, but in fact was not passing the revision info through to the final test. I don't think this would have bothered travis one bit, as that would have happily run the same revision over and over.

cc @spastorino @oli-obk ... we need like a compiletest team, don't why? :)