Compiler and rustdoc being re-compiled during CI testing stage

[adpaco-aws]

CI now runs for about ~30mins more than last week. For some reason, it appears the compiler is getting re-built and rustdoc is being built too.

[adpaco-aws]

By looking at the logs, we think this has been caused by #512

Probably has to do with some config flags not being set correctly, but a first look did not help to find this.