Investigate caching .git data on Travis/AppVeyor

[alexcrichton]

The rust-lang/rust repo itself takes awhile to clone but we somewhat mitigate that with --depth=1 clones. Our submodules, however, are much larger and unfortunately cannot be cloned with a --depth argument due to how the branches work. This typically means that cloning the LLVM repo takes quite a long time! Unfortunately this also increases our chances to network problems by requiring a lot of data to move over the network.

When playing around with CircleCI recently I found that they automatically cached git repository data which greatly sped up cloning the repository and checking out submodules. Overall it felt quite nifty! We should investigate to see if a similar strategy can apply to Travis and/or AppVeyor. I'm not personally familiar with how CircleCI's git caching works, so some investigation there would be needed (and comments if you're familiar with it would be most welcome!)

Overall I would expec this change to:

• Reduce timeouts when cloning LLVM because very little data would transfer over the network
• Speed up builds (slightly). It'd save ~10 minutes on AppVeyor and ~2 minutes on Travis from the look of today's timings.

Any help to implement this would be very much appreciated!

[sfackler]

On the Travis side, it should just be a matter of enumerating all of the directories you want: https://docs.travis-ci.com/user/caching/#Arbitrary-directories

[alexcrichton]

Yeah unfortunately I don't actually know what directories are cached here. Is it just .git? (or maybe more?)

If it's just .git then this would make a great PR!

[aidanhs]

I've made a stab in #40780. It's not as simple as caching the .git directory - by the time the script runs, you already have a branch checked out and it's not really clear how you'd move the objects around. It's also not as simple as just copying .git/modules - I'd be pretty paranoid about that breaking if someone updates the branch of a gitmodule, as I doubt git is robust at handling people messing around in there.

Instead, I've tried to go the route of using the --reference flag, which explicitly uses another repo as a cache. Unfortunately this isn't particularly well integrated with submodules, so I've had to do some workarounds.

As an aside, I'm a bit suspicious that depth: 1 (added in #35085) doesn't do anything useful. Here's the initial clone log from my travis build:

$ git clone --depth=1 https://github.com/rust-lang/rust.git rust-lang/rust
Cloning into 'rust-lang/rust'...
remote: Counting objects: 9861, done.
remote: Compressing objects: 100% (8740/8740), done.
remote: Total 9861 (delta 5307), reused 2301 (delta 1034), pack-reused 0
Receiving objects: 100% (9861/9861), 8.01 MiB | 10.89 MiB/s, done.
Resolving deltas: 100% (5307/5307), done.
$ cd rust-lang/rust
$ git fetch origin +refs/pull/40780/merge:
remote: Counting objects: 627478, done.
remote: Compressing objects: 100% (117810/117810), done.
remote: Total 627478 (delta 510690), reused 622389 (delta 505626), pack-reused 0
Receiving objects: 100% (627478/627478), 285.82 MiB | 24.97 MiB/s, done.
Resolving deltas: 100% (510690/510690), completed with 3765 local objects.
From https://github.com/rust-lang/rust
 * branch                  refs/pull/40780/merge -> FETCH_HEAD
$ git checkout -qf FETCH_HEAD

Seems like it has to fetch the whole thing anyway in order to get my branch? Maybe worth raising an issue on the travis issue tracker to ask if this is expected?

[aidanhs]

FWIW, CircleCI is a bit vague about how their git caching works:

https://circleci.com/docs/1.0/how-cache-works/#git-cache

There’s another type of caching that CircleCI does. This is to cache the Git repository of your project. This feature allows for speedier git clones of your project. Control of this cache isn’t user accessible and shouldn’t need to be. In rare scenarios, if something seems to be wrong with your source cache (for example, certain files are in your repo that shouldn’t be), please contact support.

This opacity combined with complaints about the caching ([1], [2]) is not reassuring. I could hazard some guesses at how they're doing it, but it's likely going to be more complicated and therefore less reliable than an approach tailored to the rust repo.

[nagisa]

We used to do this on buildbot didn't we? IIRC corrupted .git was pretty common occurrence.

[aidanhs]

I assume you're referring to #34595, which looks like it happens because the cache may contain a partially cloned/corrupt git submodule. This was a problem even without caching (network failure then retry is unhappy because of this same corrupted state) and was seemingly resolved by deinit before update (#39055) - the same fix may have fixed the buildbots (had they still been around).

That said, I can think of two approaches for validating before caching off the top of my head (the second problem with the buildbots, aside from not doing deinit, was caching continuously rather than only when the cache was valid) - if we see issues (or you want a more paranoid approach to begin with) I'll be able to put something in place fairly quickly.

[alexcrichton]

Thanks for the investigation @aidanhs! I'll take a look at #40780 soon.

Also yeah @nagisa I'd want to be careful about a solution here. Bad caching can cause unending problems, so I'd want to make sure we're always in a situation where the tool we're caching is relatively robust to odd cache entries. For example docker I trust to do the right thing with caching (and naturally rebuilding when necessary), and same with sccache. I was hoping that this would be solved by just persisting some git database that's resilient to naturally being updated and having old cache entries, but we'll see soon!

[notriddle]

Call me stupid (since I probably am missing something obvious), but why do Travis CI and Circle CI workers even download the commit history? GitHub has the "download a tarball from https://github.com/rust-lang/rust/archive/master.tar.gz" option, which is probably way faster.

[aidanhs]

@notriddle if you already have a repo of N commits, downloading the N+1th commit with git pull will be a few KB. Downloading the entire repo every time is a constant ~6MB. Regardless, we need some kind of git clone for submodules to work (as the commitish they're pinned to lives in the .git directory and downloaded .tar.gz contain neither .git nor checked out submodules).

That said, Travis definitely has something sub-optimal with cloning - it should be doing a shallow clone of the branch, but instead it does a shallow clone of master and then a full clone of PR branches (defeating the point of the shallow clone).

[aidanhs]

After a bit of a shaky start (merge completed successfully, then subsequent merges failed because appveyor caching was broken) just the appveyor part was rolled back and so just travis builds are currently using the new repo caching and it seems to be working ok.

In the middle of trying to fix appveyor last night, I realised that there is a way for me to test appveyor - fork rust, then comment out all CI that actually does any rust compilation etc. That way I'll be able to just test the caching. I'll be back soon with a tested PR for appveyor.