Fail to push image to registry after 2023.01.21

[can019]

Troubleshooting

Before submitting a bug report please read the Troubleshooting doc.

Behaviour

Steps to reproduce this issue

1. 2023.01.16 was the last success and had not failed before.
2. Nothing was changed in my action.yml but suddenly started to fail after 2023.01.21
3. Pushing images to registry by manual cli has been successed

Expected behaviour

Success to push images to registry

Actual behaviour

Success to build image but failed to push images to registry

Configuration

• Repository URL (if public): private
• Build URL (if public): private

Not work after 2023.01.21

   - name: Set up QEMU
        uses: docker/setup-qemu-action@v2

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

      - name: Login to Docker Hub
        uses: docker/login-action@v2
        with:
          registry: registry.***
          username: ${{ secrets.HARBOR_USER_NAME }}
          password: ${{ secrets.HARBOR_PASSWORD }}
        
      - name: Build and push image to harbor
        uses: docker/build-push-action@v3
        with:
          push: true
          tags: registry.***/***/node:latest
          target: production
          cache-from: type=gha
          cache-to: type=gha,mode=max

This yml success

      - name: Set up QEMU
        uses: docker/setup-qemu-action@v2

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

      - name: Login to Docker Hub
        uses: docker/login-action@v2
        with:
          registry: registry.***
          username: ${{ secrets.HARBOR_USER_NAME }}
          password: ${{ secrets.HARBOR_PASSWORD }}
        
      - name: Build image manualy
        run: docker build --tag registry.***/***/node:latest --target production .

      - name: Check docker image
        run: docker image ls
      
      - name: Push to harbor manualy
        run: docker push registry.***/***/node:latest

Logs

buildx failed with: ERROR: failed to solve: failed to push registry.***/***/node:latest: failed commit on ref
 "manifest-sha256:3dd1ff***": unexpected status: 500 Internal Server Error

[jedevc]

Do you have harbor-side logs? The 500 error indicates this is a registry issue - something there is likely failing.

I imagine this is the newest release of buildx/buildkit - you can try disable provenance attestations, with provenance: false to see if that helps to resolve your issue.

[crazy-max]

Also what kind of registry is this?

[jedevc]

@crazy-max I assume harbor:

      - name: Login to Docker Hub
        uses: docker/login-action@v2
        with:
          registry: registry.***
          username: ${{ secrets.HARBOR_USER_NAME }}
          password: ${{ secrets.HARBOR_PASSWORD }}

[koljamaier]

same issue, also fixed via provenance: false. Thx for the support!

[can019]

Do you have harbor-side logs? The 500 error indicates this is a registry issue - something there is likely failing.

I imagine this is the newest release of buildx/buildkit - you can try disable provenance attestations, with provenance: false to see if that helps to resolve your issue.

core.logs of harbor

Jan 21 11:39:28 ip-172-19-0-1.ap-northeast-2.compute.internal core[3877]: 
2023-01-21T11:39:28Z [WARNING] [/core/middlewares/sizequota/handler.go:48]: 
Error occurred when to handle request in size quota handler: unsupported content type for manifest: application/vnd.oci.image.manifest.v1+json

[can019]

Do you have harbor-side logs? The 500 error indicates this is a registry issue - something there is likely failing.

I imagine this is the newest release of buildx/buildkit - you can try disable provenance attestations, with provenance: false to see if that helps to resolve your issue.

@crazy-max
@jedevc
Fixed via provenance:false thanx

#23 exporting to image
[330](https://github.com/***/***/actions/runs/4042086994/jobs/6949372253#step:6:334)
#23 pushing layers 33.1s done
[331](https://github.com/***/***/actions/runs/4042086994/jobs/6949372253#step:6:335)
#23 pushing manifest for registry***/***/node:latest@sha256:5925***
[332](https://github.com/***/***/actions/runs/4042086994/jobs/6949372253#step:6:336)
#23 pushing manifest for registry***/***/node:latest@sha256:5925*** 0.7s done
[333](https://github.com/***/***/actions/runs/4042086994/jobs/6949372253#step:6:337)
#23 DONE 48.2s

I won't close the issue.

[jacek-jablonski]

Action v3.3.0 also broke my GCP Cloud Run jobs. GH Actions can successfully push docker image to Artifact Registry, but then Cloud Run is unable to load these images with error:

  - type: Started
    status: 'False'
    reason: ContainerMissing
    message: Image 'europe-west4-docker.pkg.dev/xxx' not found.

Setting provenance to false fixed this problem.

[jedevc]

@jacek-jablonski this sounds like docker/buildx#1533, not this issue.

@can019, what version of harbor registry are you using? I suspect it's likely you're using version 1? Harbor v2 has been out since 2020, and was one of the first registries to support the OCI image and distribution standards. I'd recommend an upgrade to the latest version - this should fix your issue without needing to set provenance: false.