Incremental meson build on CI

[user202729]

Problem Description

Currently, on each CI run, Sage need to be rebuilt.

In practice, it isn't all that bad because there is ccache. Which means (if it works correctly) only the pyx → c step need to be redone each time, the c → shared library step is reused.

However, we can still speed it up more. Currently it is sometimes fast (6 minutes), but sometimes slow (20+ minutes). See also #40957

[*] it's puzzling how the cache size is only 60MB, I'd have thought there be much more. In particular if I do du -ch $(find . -name "*.o") in my local build directory, I get 232MB, and du -ch $(find . -name "*.p") gives 909MB.

Proposed Solution

After each build, use tar to store the state of the build folder, then in the next CI run download it.

Before #39641, test-new use a docker image which preserves the filesystem, thus also preserve the timestamp needed for make to do dependency tracking. Afterwards, it is not reimplemented. (Although moving from make to meson also gain performance because make is not as efficient, I'd say we lose one feature to gain another feature, so it nets out.)

Side note,

• tar should be faster than docker, require less ad hoc code, and doesn't require the possibly-confusing nested layer of filesystem.
• assume ninja track dependency updates correctly (unlike make), the incremental build shouldn't cause an issue. Although this assumption may be overoptimistic.

(I don't know what's the advantage of docker over tar. Maybe @kwankyu can comment?)

Alternatives Considered

Additional Information

No response

Is there an existing issue for this?

• I have searched the existing issues for a bug report that matches the one I want to file, without success.

[tobiasdiez]

Would be nice indeed to improve the speed even further. A few boundary conditions:

• Should be stable under updates of conda packages (i.e. updating a package should not lead to build issues due to reusing the old build dir)
• Should not come close to the 10GB cache limit (otherwise caches are evicted to often, which would make the change counterproductive)

Cython does support a --cache flag, but I never played around with it (and have no idea how to add it via meson)

[user202729]

There are two options: cache allows enabling the cache by setting it to either True or the path of a cache directory, and cache_size allows to change the maintained size of the LRU cache.

from cython/cython#6070

a problem is… cython cache may not be as reliable as meson/ninja update check.

(e.g. cython/cython#6565 )

or should I say ccache is too good that everything else looks buggy in comparison?

anyway there's also a method of making a manual button that someone can press to clear the cache if things go wrong, I suppose.