CI test caching only stores most recently tested BIB hash causing repeat builds

[achilleas-k]

The CI cache is used to decide if an image should be rebuilt in testing based on whether the image definition or one of the test dependencies has changed. The following items are considered:

1. Manifest ID (hash of image definition + content).
2. osbuild version.
3. bootc-image-builder container ID.
4. Boot test success (this will rebuild an image if we add it to the image types that can boot test and no information about a previous boot test exists).

The decision is made in the check_for_build() function:

images/test/scripts/imgtestlib.py

Line 246 in 5ecb18e

def check_for_build(manifest_fname, build_info_path, errors):

The manifest ID and osbuild version are part of the path of the cached results, so any change in one of those elements creates a new entry in the cache directory tree, keeping the old result intact.
However, the bootc-image-builder container ID is only part of the metadata in the info.json file for each entry. This means that the following sequence of events can occur and cause unnecessary rebuilds:

1. Cache entry exists at path <distro>/<arch>/<osbuild-version>/<manifest-id> and the info.json contains "bib-id": "sha256:aaaa".
2. PR#A to update bib ID is created (automatically, daily). The CI run finds the cache entry, sees that the new bib ID, sha256:bbbb doesn't match the one in info.json and rebuilds the image and boot tests it.
3. <distro>/<arch>/<osbuild-version>/<manifest-id>/info.json is updated and now contains "bib-id": "sha256:bbbb".
4. PR#B is created (before the previous one is merged) with the bib ID defined in the main branch. The CI run finds the cache entry, sees that the bib ID, sha256:aaaa doesn't match the one in info.json and rebuilds the image and boot tests it.
5. <distro>/<arch>/<osbuild-version>/<manifest-id>/info.json is updated and now contains "bib-id": "sha256:aaaa" again.
6. PR#A is added to the merge queue and rebuilds the image again like in step 2, changing the bib ID in the cache back to sha256:bbbb.

Slightly different sequences of events and more PRs can cause more rebuilds, but this demonstrates a simple version of the issue.
A quick solution would be to include the bib ID in the path like we do with the osbuild version, but only a single image type depends on this in testing, so it wouldn't make sense in other cases.
Another solution would be to separate the bib metadata so we could have:

• <distro>/<arch>/<osbuild-version>/<manifest-id>/info.json
• <distro>/<arch>/<osbuild-version>/<manifest-id>/bib-aaaa.json: with content {"boot-success": true}
• <distro>/<arch>/<osbuild-version>/<manifest-id>/bib-bbbb.json: with content {"boot-success": true}

Also, we could be a bit smarter with what we test. If the manifest ID and osbuild version haven't changed, and all we need to do is boot test the image, we could download the image itself from the cache instead of rebuilding it and only run the boot test.