nix: ci: fit into the new limits

[SomeoneSerge]

Most (all) of the nix-build jobs are being cancelled in progress since the quotas have changed. Adjust the workflows to fit in the new limits.

Context: since #6243 the ci jobs are grouped by refs and cancelled together. The existing "Nix CI" job wasn't prepared for this for two reasons:

• It builds many variants of llama.cpp in a single job.
• It only pushes the results to cachix after all of the builds have ended (not sure if it does the push in the "destructor" step after the cancellation).
• PRs from forks don't have access to the repo secrets so they don't push to cachix. However, it's plausible that these could make up the majority of all jobs?
• We're running pure nix-builds, meaning we can only cache store paths (results of complete and successful builds) not e.g. intermediate object files. This provides a strong guarantee that a passing CI means the build can be reproduced locally, but this also limits how much we can reuse between the CI jobs

References:

• Make IQ1_M work for QK_K = 64 #6327 (comment)
• ci: apply concurrency limit for github workflows #6243

CC @philiptaron @Green-Sky

Potential solutions

• Make onPush builds (.#checks) less pure
  • ccacheStdenv
  • check-pointing
  • Run pure builds onSchedule instead
• More granular jobs: generate individual github jobs for individual attributes

Questions

• How effective is the caching right now?
  • PRs from forks aren't allowed to push to cachix

[SomeoneSerge]

Hi @mscheong01! Please advice, could we afford more lax limits for onSchedule jobs and on the less frequent occasions when we change the infra (e.g. cmakelists or nix expressions)?

• It would be nice to build the windows, cuda, and rocm stuff e.g. when updating the lock file (currently once a week): nix: update flake.lock #6402
• The aarch64/qemu job is onSchedule too: https://github.com/ggerganov/llama.cpp/blob/eb7828a3d8a2b1f3fbf1174826f32de99c7c25b3/.github/workflows/nix-ci-aarch64.yml#L5-L18

Even if we implement e.g. ccache, it would be nice to run the pure builds once in a while

[mscheong01]

could we afford more lax limits for onSchedule jobs

this could be done by checking the github.event_name context and assigning a unique group to the workflow if the value is 'schedule'.
more about contexts: https://docs.github.com/ko/actions/learn-github-actions/contexts#github-context

and on the less frequent occasions when we change the infra (e.g. cmakelists or nix expressions)?

I can't think of a straightforward solution that wouldn't unnecessarily complicate the workflow settings 🤔. One way would be to add a manual trigger that doesn't get cancelled by other runs, but I don't think it's the best option.

[slaren]

HIP is probably is the most important build for nix, since it is not tested in any other workflow, as far as I know. Most of the other nix builds are redundant.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.