Caching in autocomment-atime-results

[criticic]

Hey @Anirban166 @tdhock

As discussed in tdhock/atime#85, I focused on adding the caching step as part of the Github Action, I initially looked at https://github.com/actions/cache, which had almost everything needed, however a major problem with this was that it did not support accessing the cache outside of a given branch/PR (see https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows#restrictions-for-accessing-a-cache)

So I decided to just used Github Artifacts instead, and with the help of an action https://github.com/dawidd6/action-download-artifact/ which allows accessing artifacts from other runs, I was able to cache the packages.

I built upon the work I did for allowing PRs from forks, and added upon it in https://github.com/criticic/data.table-test-work-workflow/blob/master/.github/workflows/atime-run.yml . There are a couple of things that need to be ironed out, for example currently the entire library folder is getting cached, perhaps only the atime built packages should be cached, as well as a few other areas where we can cut our time.

Before caching: https://github.com/criticic/data.table-test-work-workflow/actions/runs/15456059561/job/43508458301 took 14m 5s.

After Caching: https://github.com/criticic/data.table-test-work-workflow/actions/runs/15456248247/job/43508959078 took 5m 18s using roughly 200mb of cached libgit2 build and r libraries.

[tdhock]

great thanks
please post daily updates on issues and tag me to ask for reviews on PRs

[Anirban166]

So I decided to just used Github Artifacts instead, and with the help of an action https://github.com/dawidd6/action-download-artifact/ which allows accessing artifacts from other runs, I was able to cache the packages.

I think the trade-off between using a globally shared cache across all PRs and branches vs the speed and idiomatic nature of actions/cache is justified when we think of optimizations made in universal dependencies like libgit2 as well. The main safety net we must maintain then is cache correctness, so as to prevent a bug in one feature branch from poisoning the cache for all other branches. Also, the artifact-caching approach does not look atomic so potential race conditions can arise if multiple workflows run concurrently.

[Anirban166]

@criticic Good work but like Toby said above, please regularly update us here. Weekly is a bare minimum (for all GSoC students), and it has been 3 weeks..I looked around and found this post but it should be readily accessible here without having me to dig. Keeping us in the loop is crucial as communication isn't optional during the coding period with regards to sharing progress in a timely manner. Make sure you link your PRs here too for ease of reference, like Anirban166/Autocomment-atime-results#44

[criticic]

i apologize for the lack of communication on my end, I have had to travel back and forth from multiple places more than I initially expected, some due to circumstances outside my hands. Though last week I did not post any updates due to being accepted in a offline hackathon in a different city.

I have fixed some of the major concerns on that PR. I'll also update my blog asap.

[criticic]

One method I found for using actions/cache with PRs, is to have another action update the cache on the base branch, and this can then be accessed in the PR context. This approach is explained here https://medium.com/@everton.spader/how-to-cache-package-dependencies-between-branches-with-github-actions-e6a19f33783a

[criticic]

Should I proceed with this method?

[Anirban166]

Yes, you can try it as I believe actions/cache is more stable and this would prevent cache pollution from other PRs, but it seems like we would have to go back to having two workflows.

The time savings with the current caching approach you went for looks pretty decent to me, but then this method means no time spent in artifact zipping, uploading, downloading, etc. so maybe slightly more gains?
Let's see if it bests but I certainly think its worth exploring 👍🏻