Description

We believe the Bazel cache that we store in a GCP bucket and use for CI is not working correctly.

The sw_build job is set up to use the cache by running ci/bazelisk.sh for its build (which connects Bazel to the GCP bucket as its cache) and loading the GCP write credentials when run on the master branch so that it can write to the cache. As far as we can tell, the sw_build job never loads from the cache and always rebuilds from scratch.

Locally reproducing

I tried to set up a local environment that replicates the sw_build job to read from the cache too, but no success.

These are the (convoluted) steps I used (thanks to @nbdd0121 for helping with this):

1. Added the following to ~/.bazelrc (and ~root/.bazelrc for good measure) to replicate what ci/bazelisk.sh builds:

build --remote_cache=https://storage.googleapis.com/opentitan-bazel-cache
build --remote_upload_local_results=false
build --remote_default_exec_properties=OSVersion="Ubuntu 20.04.6 LTS"

2. Ensured my local environment variables matched CI for those set in rules/nonhermetic.bzl:
   • Set HOME="/root" - this is what the CI uses and the HOME env var is non-hermetic, so may have influenced the cache?
   • Set PATH=/tools/verilator/v4.210/bin:/tools/verible/bin:/root/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin which is what sw_build uses - we extracted this by running env in this PR's CI run.
   • $XILINXD_LICENSE_FILE already matched, and the other vars were unset.
3. Execute the same build command that sw_build does but using ./bazelisk.sh instead of ci/bazelisk.sh to ensure our own .bazelrc is used.

We did not observe the cache being used at all. There must be something else contributing to the cache that we didn't account for.

Related

Not every job in the CI workflow uses the cache: some aren't using ci/bazelisk.sh so won't read from it, and some aren't loading the GCP write credentials and won't write to it.

I've opened #20836 to address these issues, but they won't make much difference if the cache isn't working.