Feature Request: Improve CI speed of CUDA on Windows

[NeoZhangJianyu]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

The whole CI of a PR takes about 50+minutes.
Following two tasks take 50+ and 40+mins:

    windows-latest-cmake-cuda (12.2.0, cuda)
    windows-latest-cmake-cuda (11.7.1, cuda)

Other tasks takes time less than 25ms.

Motivation

If the CI time is reduced to 30min from 50+mins, it will save more time of developers.
Speed up the progress of project.

Possible Implementation

No response

[JohannesGaessler]

The long compilation time is a result of templating. There is a large number of combinations of compute capabilities, quantization formats, and tensor shapes for which different template instances are needed for optimal performance. I think a long compilation time is inevitable unless you're willing to trade in some performance or features. If the only goal is to assert the correctness of the code it is possible to significantly speed up the compilation time but the resulting binaries would not be suitable for distribution. In either case, if you are interested in developing and maintaining a flag for faster compilation I am willing to assist (in the easiest case it would be enough to compile with --device-debug which disables all optimization of device code).

[JohannesGaessler]

Also just to make sure: you said CI so I was assuming you're talking about Github, but if you are compiling locally you can significantly speed up the compilation by using multiple threads and ccache.

[JohannesGaessler]

Looking at the log of one of those compilations, it seems that it takes 13 minutes to install CUDA on the Windows runner. Unfortunately looking at the repository for Github runners the request of providing an image with CUDA pre-installed was raised in the past and rejected.

[slaren]

I guess we could build only one architecture.

[JohannesGaessler]

I would say the most important architecture to have in CI is Turing since that also covers Ampere, Hopper, and Ada Lovelace. I think part of the CI is running on a machine with a V100 though so Volta should also be included. If we only remove old CUDA architectures I think an error will be raised automatically when trying to use the code with an old GPU.

[slaren]

I meant for the PRs only. For the releases we should still include all the architectures.

We could to skip building all the examples in the PRs, since that's tested elsewhere (ie. build only the backend). But it would only save a couple of minutes.

[slaren]

The compilation time with one arch (+ ninja) drops to ~12 minutes https://github.com/ggerganov/llama.cpp/actions/runs/12012342821/job/33483343472?pr=10456

[NeoZhangJianyu]

Yes, I means the CI case on github action.
It's good to reduce the compile time to speed up PR merge.
~12mins is good enough!

[slaren]

Should be faster now.