Improve CI build and test times

[zanieb]

Without cache hit source

Compile time for Linux tests 7m 23s
Compile time for Windows tests 8m 56s
Runtime for docs check 2m 33s

With cache hit source

Compile time for Linux tests 3m 13s
Compile time for Windows tests 4m 48s
Runtime for docs check 35s

CI takes an additional 2 minutes to run the tests in both environments

We only check documentation on Linux.

It'd be great if these were faster. I'm not sure it's possible but I'd love to hear ideas.

[charliermarsh]

Is that the documentation check, or doc tests?

[zanieb]

It's the cargo doc command (and most of the time is spent compiling) so it should just be documentation checks.

Considering how slow doctests are generally though, an action item here may be to investigate the time of doctests :)

[zanieb]

Related exploration of nextest

• Explore use of cargo nextest for test running #3752
• Use nextest to run tests in CI #6979

[zanieb]

We could consider paying for runners with more CPUs (https://docs.github.com/en/actions/using-github-hosted-runners/about-larger-runners/about-larger-runners) although I would like to chase free options first :)

[zanieb]

Related:

• Splitting up the ruff_linter crate for faster compile times #1820

[BurntSushi]

I haven't looked at CI and the build in detail, but if any of it is building in release mode with fat LTO enabled, then that is likely taking some sizeable chunk of time. Fat LTO is notorious for being horrendously slow. #8544 switches over to thin LTO. The other thing potentially worth experimenting with is bumping up the codegen-units from 1 to something a little higher. It lets compilation use more parallelization at the expense of potentially giving up some code optimizations.

In addition to the other ideas mentioned, particularly in the non-cache case, a more involved approach might be to reduce the number of dependencies overall. Ruff has quite a lot of them. However, it's not clear to me how much the non-cache case truly matters. (I imagine most folks are installing Ruff as a wheel of some kind? Apologies for the deployment ignorance here on my part.)

[charliermarsh]

(I imagine most folks are installing Ruff as a wheel of some kind? Apologies for the deployment ignorance here on my part.)

No worries at all. That's right, the vast majority of downloads are pre-compiled wheels that we upload to PyPI. (These are basically compiled binaries that get wrapped up with some other metadata in a zip archive.) I would say slow release builds are mostly a problem for development (especially profiling). And slow debug builds are definitely a problem for development.

[konstin]

I believe the most impactful - but also most effort - change would be splitting up the ruff_linter crate (#1820), the crate is very slow to compile compared to other crates i've used, even in debug mode.