Reduce CI time

[zanieb]

Running tests in CI is now >8 minutes (and was previously <1 minute). We've done a lot to optimize this previously, e.g.:

• Use a larger runner for tests #889
• Use the larger runner for Windows clippy runs #3044
• Use nextest test runner in CI #875
• Use mold for linking in CI tests #887
• Explore using a dev drive for windows CI #3508
• Use Windows 2019 runner instead of 2022 #1830
• Disable Windows defender in CI #1832
• Add tests to Windows Defender exclusions #2933
• ... more examples coming

But, CI time is always growing as we add more features and coverage. This is a tracking issue to improve the situation and discuss sources of slowness.

See also:

• Reduce build and check times #5711
• Slow tests #878
• Add fast build profile without debuginfo #2628 (comment)

[zanieb]

Would it be absurd to introduce "CI" time regression checks in CI like the CodSpeed benches? Unfortunately the GitHub Runner performance is super noisy so it might not work.

[helderco]

Wonder if Dagger could help here 🙂

[samypr100]

Arguably self-hosted runners could possibly help here too, but the maintenance/security/cost burden is likely too large

[zanieb]

I strongly considered self-hosted runners, but it seemed painful to orchestrate Windows runners in particular.

[zanieb]

I also very much want to look into something like #609 again to cache our network traffic — I think that'd help a lot.

[samypr100]

I also very much want to look into something like #609 again to cache our network traffic — I think that'd help a lot.

At least for python, maybe worth using something like https://github.com/hauntsaninja/nginx_pypi_cache and saving a copy of the cache as a github cache and reloading it. I actually use it locally a lot to speed up tests. Not sure how it would perform in CI.

[charliermarsh]

That strikes me as a good idea...

[zanieb]

Ah that might be easier than using mitmproxy or rolling our own proxy in Rust. Thanks for the link!

edit: I created a published image at https://github.com/astral-sh/nginx_pypi_cache/pkgs/container/nginx_pypi_cache we can use in our jobs if someone wants to trial it in Ubuntu (or locally even, to start)

[zanieb]

Some progress with:

• Use a larger runner for macOS test jobs #5874
• Use a larger runner for Linux test jobs #5883
• Use a larger runner for Windows test jobs #5873

[zanieb]

Linux is acceptably fast now. We're at the limit for macOS machine size without going to alternative runner providers. There are larger Windows runners, maybe I should test one (#5890). Cost may be a problem at some point, alternative runner providers may be cheaper (often saying things like a 2x cost reduction).

[ChannyClaus]

have we considered using https://bazel.build/? (correct me if i'm wrong but cargo test doesn't seem to skip the tests when none of their dependencies changed)

admittedly adding bazel would significantly increase the complexity though 🙀

[eth3lbert]

Yes, I've heard a lot about complaints regarding bazel's complexity.

[ChannyClaus]

Yes, I've heard a lot about complaints regarding bazel's complexity.

this is definitely true - i've seen it improving the CI time drastically at the same time though so maybe something to consider down the line if there's no other option...