CI failing with `403 Forbidden` errors pushing Docker image

[smklein]

CI started going red after some simple dependabot PRs:

These PRs really shouldn't be changing how we push docker images, but they're resulting in failures:

Here's an example failure: https://github.com/oxidecomputer/omicron/runs/8211105499?check_suite_focus=true

#22 exporting to image
#22 exporting layers
#22 exporting layers 4.0s done
#22 ...

#23 [auth] oxidecomputer/omicron:pull,push token for ghcr.io
#23 DONE 0.0s

#22 exporting to image
#22 exporting manifest sha256:cd232512a422c64b9c3d97340c63a275d31dcb9c4034299e3bfc1b646c0796f4 done
#22 exporting config sha256:dd0f52eed5c77b6576923886f1e057615c51d9cb64e788f31607f43521964161 done
#22 pushing layers
#22 pushing layers 0.4s done
#22 ERROR: unexpected status: 403 Forbidden
------
 > exporting to image:
------
WARNING: local cache import at /tmp/.buildx-cache not found due to err: could not read /tmp/.buildx-cache/index.json: open /tmp/.buildx-cache/index.json: no such file or directory
error: failed to solve: unexpected status: 403 Forbidden
Error: buildx call failed with: error: failed to solve: unexpected status: 403 Forbidden

[smklein]

This appears to be flaky, subsequent PRs are passing on main

[iliana]

Various similar issues on the docker/build-push-action repo suggest contacting GitHub Support, so I have (ticket #1776735).

Deep in the weeds of their issues, there is a suggestion that there may be rate limiting happening; all four of those were merged at roughly the same time. Another pattern of that appears at https://github.com/oxidecomputer/omicron/actions/runs/2877061460 https://github.com/oxidecomputer/omicron/actions/runs/2877057057 https://github.com/oxidecomputer/omicron/actions/runs/2877054158. A suggestion on that repo's issues is to set max-parallel: 4, but that seems like it'd be too high, too.

[iliana]

Support referred to this particular incident as an "outage", writing:

Engineering have identified the cause, and have implemented mitigations. As a result, you should no longer be experiencing any issues, but please let us know if this is not the case.

Engineering will be deploying a permanent fix for this problem shortly.

No other details on root cause.

[jgallagher]

A different failure at the same step happened on https://github.com/oxidecomputer/omicron/actions/runs/3989136170/jobs/6841193236:

#22 [stage-1 9/9] COPY --from=cargo-build /usr/src/omicron/target/release/sled-agent-sim /usr/bin/sled-agent-sim
#22 DONE 0.1s

#23 exporting to image
#23 exporting layers
#23 ...

#24 [auth] oxidecomputer/omicron:pull,push token for ghcr.io
#24 DONE 0.0s

#23 exporting to image
#23 exporting layers 2.4s done
#23 exporting manifest sha256:51ecc495a48b87429d25fa22497367d90987cfd0eedc7a21ea215178fb419a4f done
#23 exporting config sha256:71b5aab2b23137637bcd0d3d2385eb9b6029d8c6a2f586[1687](https://github.com/oxidecomputer/omicron/actions/runs/3989136170/jobs/6841193236#step:7:1688)c5e935c09d4ada done
#23 exporting attestation manifest sha256:2479151fff061748881c729a12008d29d0b269f7897960818379db6a534f0698 done
#23 exporting manifest list sha256:9c962525030511957fc71e8bad54d6486dd91c2c7f57d7d18d5f730f626ab6b9 done
#23 pushing layers
#23 pushing layers 0.9s done
#23 ERROR: failed to push ghcr.io/oxidecomputer/omicron:mgs-ascii-serial-number: failed to copy: io: read/write on closed pipe
------
 > exporting to image:
------
ERROR: failed to solve: failed to push ghcr.io/oxidecomputer/omicron:mgs-ascii-serial-number: failed to copy: io: read/write on closed pipe
Error: buildx call failed with: ERROR: failed to solve: failed to push ghcr.io/oxidecomputer/omicron:mgs-ascii-serial-number: failed to copy: io: read/write on closed pipe

[iliana]

I am considering trying out https://github.com/marketplace/actions/retry-step for flaky steps.

The benefit of this over a whole-action retry is that, for this failure case, the docker image is already built, and docker buildx should see that and just retry the push.

The disadvantage would be that if there's a compilation failure, it would rebuild the entire project for however many retries, further eating into the quota of runners for a public repository. (We could potentially make this, and other actions, rely on the success of the check style action.)

[iliana]

(We're also pinning to a 2021 version of docker/build-push-action, so maybe I'll try upgrading that and see if the flakiness improves.)

[smklein]

Due to #2691 , I'm going to close this