Make CI runs faster

[thomaseizinger]

Describe your use case

We are using this in CI and it contributes about ~1-2min to the CI run of each crate despite:

• cargo semver-checks being installed
• cargo doc being cached

See https://github.com/thomaseizinger/rust-libp2p/actions/runs/3653859145/jobs/6173748557 for an example.

Is there a way for how this can be improved? When I run it locally, it is a lot faster (after the first invocation) so I am guessing there is a cache somewhere? Which files do I need to cache?

Describe the solution you'd like

cargo semver-checks to finish within seconds, not minutes.

Alternatives, if applicable

No response

Additional Context

No response

[obi1kenobi]

I'm not familiar with your setup for caching cargo doc — does it work for only the current state of the package, or does it also cache the cargo doc run for the baseline (crates.io) version as well? Based on the log, it seems like the actual checking of the rustdoc files finished in 0.044s, so the slow parts are building one or both of the rustdoc JSON files.

#208 should merge very soon, and might make it easier to not have to cache the cargo install step.

Since the baseline version isn't changing all that often (especially not with every PR), it should be possible to cache the baseline rustdoc JSON file across runs. This is how the GitHub Action builds that file, and in principle it could be improved to cache the file as well. cargo-semver-checks can be directed to use a pre-generated baseline rustdoc JSON file with the --baseline-rustdoc <file> option.

After running cargo check --all-features or cargo test --all-features, building the rustdoc JSON for the current version should be quite fast since everything is already mostly checked and built. So running cargo-semver-checks in the same runner where the tests already ran would probably make it much faster.

In general, I believe your cargo-semver-checks setup is the most sophisticated one in existence (at least that I know of). If you're interested, I'd be thrilled to work with you on building all these capabilities into the cargo-semver-checks GitHub Action itself rather than each of us maintaining a separate workflow.

[thomaseizinger]

My workflow doesn't build it for the baseline version, no. I thought that one was downloaded from docs.rs or something?

Does it build the baseline version also locally?

I am happy to collaborate on an/the action. Fitting that one out with some pre-configured caching would be really nice actually :)

#208 is nice to see!

[obi1kenobi]

docs.rs doesn't yet host the JSON, so for now the baseline is built locally (based on the registry, a gitrev, or a filesystem path) unless the baseline JSON is explicitly provided via the --baseline-rustdoc flag.

This is what the action does at the moment: https://github.com/obi1kenobi/cargo-semver-checks-action/blob/main/action.yml#L29

In a v2 of the action, we'd want to remove the version-selection logic from the action (since it incorrectly selects pre-releases, unlike the logic already present in cargo-semver-checks) and make cargo-semver-checks aware of the possibility of the baseline rustdoc JSON already being generated so it only rebuilds when necessary.

Perhaps something like a --baseline-cache <PATH> option, where cargo-semver-checks will move built baseline JSON files into that directory, or use existing files if they are there?

What do you think? Also, looping in @epage in case he has ideas.

[thomaseizinger]

make cargo-semver-checks aware of the possibility of the baseline rustdoc JSON already being generated so it only rebuilds when necessary.

I think this is the MVP. As long as it is somewhere in the target directory, CI caching should pick that up. Configuration options to control the path can come later once someone asks for it!

[thomaseizinger]

I just saw that there is already target/semver-checks but that directory is 21GB after a cargo clean and running cargo semver-checks check-release --workspace. We can't cache that 😅

[thomaseizinger]

Perhaps something like a --baseline-cache <PATH> option, where cargo-semver-checks will move built baseline JSON files into that directory, or use existing files if they are there?

Just separating them from the current compilation output would already be good. That would allow us to clean that target directory of artifacts and cache it. These should never need to change so unless someone messes with the target directory, I don't think there should be any bugs.

[obi1kenobi]

I just saw that there is already target/semver-checks but that directory is 21GB after a cargo clean and running cargo semver-checks check-release --workspace. We can't cache that 😅

I'm a bit disk-space-limited on the machine I have with me right now, so if you don't mind, I'd love to ask you to check something in that target/semver-checks directory:

• Ignore (or delete) the target/semver-checks/target directory entirely.
• All the other directories in target/semver-checks should be prefixed with registry, right? Those are registry (crates.io) based baseline builds, which each contain both the usual compilation artifacts (debug directory) and the rustdoc JSON doc directory.
• How big are all the JSON files from those doc directories? That's the only part we absolutely need, and we can avoid caching everything else.

The cache would be compressed with zstd in GitHub Actions, so it would probably be reduced to around one-third or one-fourth the raw JSON files' size.

Just separating them from the current compilation output would already be good. That would allow us to clean that target directory of artifacts and cache it.

We can put the cache somewhere like target/semver-checks/cache. How does that sound?

[thomaseizinger]

• How big are all the JSON files from those doc directories? That's the only part we absolutely need, and we can avoid caching everything else.

> stat -c %s target/semver-checks/**/*.json | paste -sd+ | bc
110568535

That is a 110MB!

[thomaseizinger]

Just separating them from the current compilation output would already be good. That would allow us to clean that target directory of artifacts and cache it.

We can put the cache somewhere like target/semver-checks/cache. How does that sound?

Sounds good to me. I guess to make this really effective, cargo semver-checks would have to move the JSON files into that cache directory itself and clean the target directory afterwards otherwise we have a manual step again of cleaning those target directories.

[obi1kenobi]

That is a 110MB!

Whew, I'm glad it's that little :) It's also JSON so it should be highly compressible with zstd which GitHub Actions will use by default.

to make this really effective, cargo semver-checks would have to move the JSON files into that cache directory itself and clean the target directory afterwards

I think cargo-semver-checks should move the JSON files into the cache directory, but I think we should leave the cleaning to its GitHub Action I don't think that cargo-semver-checks itself should clean the target directory, since that might be undesirable for local use.

Let me know if you have any concerns!

[thomaseizinger]

That is a 110MB!

Whew, I'm glad it's that little :) It's also JSON so it should be highly compressible with zstd which GitHub Actions will use by default.

I'll test it locally!

to make this really effective, cargo semver-checks would have to move the JSON files into that cache directory itself and clean the target directory afterwards

I think cargo-semver-checks should move the JSON files into the cache directory, but I think we should leave the cleaning to its GitHub Action I don't think that cargo-semver-checks itself should clean the target directory, since that might be undesirable for local use.

Let me know if you have any concerns!

Hmm, I think that is inconsistent. We are creating those JSON files locally because we can't download from docs.rs.

The fact that we need to build them is an implementation detail IMO. Using as little disk space as possible is also desirable locally, especially when we are talking as much as 22GB for a single repository!

Alternatively, we can also say that the program is completely ignorant of where the data is stored and the action moves around the JSON files and passes them in as baseline etc

[thomaseizinger]

That is a 110MB!

Whew, I'm glad it's that little :) It's also JSON so it should be highly compressible with zstd which GitHub Actions will use by default.

I'll test it locally!

Default zstd yields an archive 13 MB in size :)

[thomaseizinger]

to make this really effective, cargo semver-checks would have to move the JSON files into that cache directory itself and clean the target directory afterwards

I think cargo-semver-checks should move the JSON files into the cache directory, but I think we should leave the cleaning to its GitHub Action I don't think that cargo-semver-checks itself should clean the target directory, since that might be undesirable for local use.
Let me know if you have any concerns!

Hmm, I think that is inconsistent. We are creating those JSON files locally because we can't download from docs.rs.

The fact that we need to build them is an implementation detail IMO. Using as little disk space as possible is also desirable locally, especially when we are talking as much as 22GB for a single repository!

Alternatively, we can also say that the program is completely ignorant of where the data is stored and the action moves around the JSON files and passes them in as baseline etc

I am happy to have a go at implementing this if you agree that it should happen within cargo semver-checks. It currently contributes to about 20% of our CI time which I am keen to optimize :)