Annotations not saved when outputting to OCI .tar file

[emirb]

I have a multiple-step process for building OCI images and pushing them to registry:

1. build an image with buildctl, save it to OCI format tar file
2. scan the .tar with container analysis tools
3. push the .tar to registry using crane:

Annotations are important here to preserve metadata on images and use them for further analysis/validation when they're running in Kubernetes.

I'm using a Docker image of master version of buildctl (moby/buildkit:master-rootless).

First step runs buildctl with --output "type=oci,dest=image.tar:

buildctl-daemonless.sh build --frontend=dockerfile.v0 --local context=$CONTEXT --progress=plain --local dockerfile=. --opt filename=$DOCKERFILE_PATH \
        --opt build-arg:"COMMIT_SHA='$COMMIT_SHA'" \
        --opt build-arg:"COMMIT_TAG='$COMMIT_TAG'" \
        --output "type=oci,dest=image.tar,annotation.org.opencontainers.image.source='$SOURCE_SHA',annotation.org.opencontainers.image.revision='$COMMIT_SHA',annotation.org.opencontainers.image.version='$COMMIT_TAG',annotation.org.opencontainers.image.created='$CREATED_DATE'" 

The image is then saved to image.tar. I'm setting annotations as described here.

When all 3 steps are done, when I pull the final image from remote registry and inspect it, it doesn't have the annotations I've set when building the image at all:

$ docker buildx imagetools inspect $IMAGE_URL
Name:      $IMAGE_URL/hidden
MediaType: application/vnd.docker.distribution.manifest.v2+json
Digest:    sha256:f3b4abbb5b9c65e3f2498644de9f9f8e1f26e11afa33036758809c62ac9f65be
$ docker buildx imagetools inspect $IMAGE_URL --raw|jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 1095,
    "digest": "sha256:ed1fd146a0d1a0e9c4b6c40183680dfd3024b8d59c135ccb989566d5cd012f6b"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 2817409,
      "digest": "sha256:79e9f2f55bf5465a02ee6a6170e66005b20c7aa6b115af6fcd04fad706ea651a"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 1024,
      "digest": "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
    }
  ]
}
$ docker image inspect $IMAGE_URL
...
  "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "/bin/sh"
            ],
            "Image": "",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": null
        },
...

However, building the same image with buildkit indirectly, using docker buildx, with --label options, and also saving to an OCI .tar file, does produce these annotations. The only difference is label vs annotation here.
The doc above says Annotations are similar to, but not a replacement for image labels.. I also have no problem using labels, but buildkit doesn't support them.

docker buildx build $CONTEXT \
--progress=plain \
--output type=oci,dest=image.tar \
-f $DOCKERFILE \
--label 'org.opencontainers.image.source="$SOURCE"' \
--label 'org.opencontainers.image.revision="$SHA'' 
...

After pushing, and pulling:

 ...
 "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "/bin/sh"
            ],
            "Image": "",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "org.opencontainers.image.created": "2022-11-28 22:17:11",
                "org.opencontainers.image.revision": "fda5b05f4aa217feadb3f7e27819c82ee203dd73",
                "org.opencontainers.image.source": "foo",
                "org.opencontainers.image.version": "bar"
            }

Maybe this is case of PEBKAC? :) and maybe annotations are meant to be saved when using --output type=image only?

The problem is, I need an OCI .tar archive of an image as I do the image pushing in an another environment. AFAIK there is no way to run --output type=image and then save it to a file using buildctl.

[mjrist]

I am experiencing the same issue. Annotations do not appear to be working with the oci export.