@sha256@sha256 strikes again! (on 0.11.1.1)

[wking]

[//]: # kind bug

And in case Prow notices:

/kind bug

Description

As seen back in #877, but now on 0.11.1.1.

Steps to reproduce the issue:

I haven't worked out a minimal reproducer yet, but we've had a number of folks using this script (openshift/installer#933) see:

$ sudo podman inspect quay.io/openshift-release-dev/ocp-release:4.0.0-8 -f '{{ index .RepoDigests 0 }}'
quay.io/openshift-release-dev/ocp-release@sha256@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777

Describe the results you expected:

I expected the RepoDigests entry to only contain a single @sha256.

Additional information you deem important (e.g. issue happens only occasionally):

Some OpenShift Installer runs do not see this problem, but some (using the same podman and update payload image) do. Runs that see it once can reproduce it reliably. So there's some way to stumble into the broken state, and once you do you stay there. But there's also a way to avoid the broken state.

Output of podman version:

@robszumski was kind enough to work through all of this, and he had:

$ podman version
Version:       0.11.1.1
Go Version:    go1.10.2
OS/Arch:       linux/amd64
$ podman info
host:
  BuildahVersion: 1.5-dev
  Conmon:
    package: podman-0.11.1.1-3.git594495d.el7.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: ce5c8112ef76bda67e00829952505aa447b343c1-dirty'
  Distribution:
    distribution: '"rhcos"'
    version: "4.0"
  MemFree: 5730500608
  MemTotal: 8368660480
  OCIRuntime:
    package: runc-1.0.0-57.dev.git2abd837.el7.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.0'
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 2
  hostname: ip-10-0-6-138
  kernel: 3.10.0-957.1.3.el7.x86_64
  os: linux
  rootless: false
  uptime: 2h 8m 39.48s (Approximately 0.08 days)
insecure registries:
  registries: []
registries:
  registries:
  - registry.access.redhat.com
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.centos.org
store:
  ContainerStore:
    number: 5
  GraphDriverName: overlay
  GraphOptions:
  - overlay.override_kernel_check=true
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
  ImageStore:
    number: 14
  RunRoot: /var/run/containers/storage

Additional environment details (AWS, VirtualBox, physical, etc.):

AWS. Using:

$ cat /etc/os-release
NAME="Red Hat CoreOS"
VERSION="4.0"
ID="rhcos"
ID_LIKE="rhel fedora"
VERSION_ID="4.0"
PRETTY_NAME="Red Hat CoreOS 4.0"
ANSI_COLOR="0;31"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat 7"
REDHAT_BUGZILLA_PRODUCT_VERSION="4.0"
REDHAT_SUPPORT_PRODUCT="Red Hat"
REDHAT_SUPPORT_PRODUCT_VERSION="4.0"
OSTREE_VERSION=47.235

as installed by the v0.8.0 installer. From the broken state:

$ sudo podman inspect quay.io/openshift-release-dev/ocp-release:4.0.0-8
[
   {
       "Id": "e3cd6917002af26debf8052e7f1d95ef7ed912111593248cff8de9ca076569e8",
       "Digest": "sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777",
       "RepoTags": [
           "quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777",
           "quay.io/openshift-release-dev/ocp-release:4.0.0-8"
       ],
       "RepoDigests": [
           "quay.io/openshift-release-dev/ocp-release@sha256@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777",
           "quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777"
       ],
       "Parent": "",
       "Comment": "Release image for OpenShift",
       "Created": "2018-12-22T22:18:35.744591Z",
       "ContainerConfig": {
           "Env": [
               "OPENSHIFT_BUILD_NAME=cluster-version-operator",
               "OPENSHIFT_BUILD_NAMESPACE=ci-op-zlhfgrm2",
               "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
           ],
           "Entrypoint": [
               "/usr/bin/cluster-version-operator"
           ],
           "Labels": {
               "io.openshift.release": "4.0.0-8",
               "io.openshift.release.base-image-digest": "sha256:57ea7826cf3c73d2dd29dfff375f5c09da087cb0e6a72b998bb74a9d18adeaa7"
           }
       },
       "Version": "1.13.1",
       "Author": "",
       "Architecture": "amd64",
       "Os": "linux",
       "Size": 291664680,
       "VirtualSize": 291664680,
       "GraphDriver": {
           "Name": "overlay",
           "Data": {
               "LowerDir": "/var/lib/containers/storage/overlay/57266b1eabbfe64e1c3330b16f0534b77c76af1ba2e28d69ecbc59779785dd1d/diff:/var/lib/containers/storage/overlay/ccbac139cf89893ed2760507dc7f3e19210593e72a9a2d8f05c486e0bb77e7a6/diff:/var/lib/containers/storage/overlay/071d8bd765171080d01682844524be57ac9883e53079b6ac66707e192ea25956/diff",
               "MergedDir": "/var/lib/containers/storage/overlay/5c857d77af584de126b963367139d6546cbfe34208c92cbeee967e2ceaaaee10/merged",
               "UpperDir": "/var/lib/containers/storage/overlay/5c857d77af584de126b963367139d6546cbfe34208c92cbeee967e2ceaaaee10/diff",
               "WorkDir": "/var/lib/containers/storage/overlay/5c857d77af584de126b963367139d6546cbfe34208c92cbeee967e2ceaaaee10/work"
           }
       },
       "RootFS": {
           "Type": "layers",
           "Layers": [
               "",
               "",
               "",
               ""
           ]
       },
       "Labels": {
           "io.openshift.release": "4.0.0-8",
           "io.openshift.release.base-image-digest": "sha256:57ea7826cf3c73d2dd29dfff375f5c09da087cb0e6a72b998bb74a9d18adeaa7"
       },
       "Annotations": {},
       "ManifestType": "application/vnd.docker.distribution.manifest.v1+prettyjws",
       "User": ""
   }
]

Comparing that with a local:

$ podman pull quay.io/openshift-release-dev/ocp-release:4.0.0-8
$ podman inspect quay.io/openshift-release-dev/ocp-release:4.0.0-8

yielded:

--- good
+++ broken
@@ -3,9 +3,11 @@
        "Id": "e3cd6917002af26debf8052e7f1d95ef7ed912111593248cff8de9ca076569e8",
        "Digest": "sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777",
        "RepoTags": [
+            "quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777",
            "quay.io/openshift-release-dev/ocp-release:4.0.0-8"
        ],
        "RepoDigests": [
+            "quay.io/openshift-release-dev/ocp-release@sha256@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777",
            "quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777"
        ],
        "Parent": "",

With the same podman, pulling a fresh, similar image (-7) avoided the broken state:

$ podman pull quay.io/openshift-release-dev/ocp-release:4.0.0-7
$ podman inspect quay.io/openshift-release-dev/ocp-release:4.0.0-7
[
    {
        "Id": "72d96fea0c4fbcc527f509963c11b7c9c9b0bf9ba5641cd18968eb9150f54464",
        "Digest": "sha256:3e70b2d2004c9335dabc24d6259c5b03739fd86eae435c191ee2b8592e86efec",
        "RepoTags": [
            "quay.io/openshift-release-dev/ocp-release:4.0.0-7"
        ],
        "RepoDigests": [
            "quay.io/openshift-release-dev/ocp-release@sha256:3e70b2d2004c9335dabc24d6259c5b03739fd86eae435c191ee2b8592e86efec"
        ],
        "Parent": "",
        "Comment": "Release image for OpenShift",
        "Created": "2018-12-15T06:41:33.392131088Z",
        "ContainerConfig": {
            "Env": [
                "OPENSHIFT_BUILD_NAME=cluster-version-operator",
                "OPENSHIFT_BUILD_NAMESPACE=ci-op-40fz390h",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Entrypoint": [
                "/usr/bin/cluster-version-operator"
            ],
            "Labels": {
                "io.openshift.release": "4.0.0-7",
                "io.openshift.release.base-image-digest": "sha256:f9585cce1889d934b6f9c8361bb1fcc532235ddc4b88ab8614be40545e6bf5d8"
            }
        },
        "Version": "1.13.1",
        "Author": "",
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 292396852,
        "VirtualSize": 292396852,
        "GraphDriver": {
            "Name": "overlay",
            "Data": {
                "LowerDir": "/var/lib/containers/storage/overlay/639b39d8f4d99083f7276b1915cc51b4c24db7e7e5cb9a0817e8b414843b105a/diff:/var/lib/containers/storage/overlay/eb0c639d1332b895a66623afad255f15d2ac507a51f4d54a8604d292660a7d45/diff:/var/lib/containers/storage/overlay/071d8bd765171080d01682844524be57ac9883e53079b6ac66707e192ea25956/diff",
                "MergedDir": "/var/lib/containers/storage/overlay/7ce4431b6f986e887ce043bab47c2e36838e460a3fa2ea7353ad5a30f85fe7c7/merged",
                "UpperDir": "/var/lib/containers/storage/overlay/7ce4431b6f986e887ce043bab47c2e36838e460a3fa2ea7353ad5a30f85fe7c7/diff",
                "WorkDir": "/var/lib/containers/storage/overlay/7ce4431b6f986e887ce043bab47c2e36838e460a3fa2ea7353ad5a30f85fe7c7/work"
            }
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "",
                "",
                "",
                ""
            ]
        },
        "Labels": {
            "io.openshift.release": "4.0.0-7",
            "io.openshift.release.base-image-digest": "sha256:f9585cce1889d934b6f9c8361bb1fcc532235ddc4b88ab8614be40545e6bf5d8"
        },
        "Annotations": {},
        "ManifestType": "application/vnd.docker.distribution.manifest.v1+prettyjws",
        "User": ""
    }
]

Also in this space is #1139, but I don't understand RepoDigests generation well enough to know if this is a dup of that issue or not.

[mheon]

The SHA256s seem to match here, where they did not in #877 - which seems to indicate we're pulling the right image, which is good.

This could just be a display issue (or we could be setting a nonsensical name when we save the image)

[robszumski]

I seem to be able to reproduce this each time I try to boot a cluster, although I have no idea why/how.

[rhatdan]

I don't see how this could happen in libpod code, I would guess it would be in containers/image perhaps?

@mtrmac @vrothberg WDYT?

[baude]

I would think it actually comes from libpod ... maybe the image side of things ... but I need a replicator to trip it. There is actually one spot where we inject sha256 into the image name. But like i said, this would be vastly easier with a reproducer because it would allow us to figure out whats really going on and add a test to prevent regressions.

[mtrmac]

$ sudo podman inspect quay.io/openshift-release-dev/ocp-release:4.0.0-8

…

   "RepoTags": [
       "quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777",

^^^

       "quay.io/openshift-release-dev/ocp-release:4.0.0-8"
   ],
   "RepoDigests": [
       "quay.io/openshift-release-dev/ocp-release@sha256@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777",
       "quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777"
   ],

There are actually very few RepoDigests references throughout podman, pretty sure this is
https://github.com/containers/libpod/blob/c9d63fe89d0a79b069b56249aaa4c168b47649c0/libpod/image/image.go#L311 when Image.Names contains a name@digest value.

This is not exactly #1139 , this is just a local detail of how Names+Digest are combined into `RepoDigests; #1139 is related in the sense the code combining the two IMHO should not exist at all, because it is often enough correct but wrong in general.

[wking]

... when Image.Names contains a name@digest value.

That certainly looks like a likely culprit to me. @robszumski has a reliable reproducer (even if it's a bit complicated for a libpod unit test ;), is there anything he can check for to dig into this possibility?

[mtrmac]

I suspect a

$ podman pull quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777
$ podman pull quay.io/openshift-release-dev/ocp-release:4.0.0-8

would recreate the situation (but I didn’t try it!).

[wking]

I suspect a

$ podman pull quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777
$ podman pull quay.io/openshift-release-dev/ocp-release:4.0.0-8

would recreate the situation (but I didn’t try it!).

Not for me:

$ podman pull quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777
Trying to pull quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777...Getting image source signatures
Copying blob sha256:a02a4930cb5d36f3290eb84f4bfa30668ef2e9fe3a1fb73ec015fc58b9958b17
 71.68 MB / 71.68 MB [=====================================================] 21s
Copying blob sha256:47e4121c7dbd4743f868526085b5bb36f4dff5cec8c1a3f992a6b7f2bf06403c
 9.57 MB / 9.57 MB [========================================================] 2s
Copying blob sha256:ca9c3b8314517310a56ef66741850ee0acce144bb241778cc13635b62fb990b0
 10.34 MB / 10.34 MB [======================================================] 9s
Copying blob sha256:0f3ada1e64e347b6c2617705d16ba9cc1abc287d7b527b979a55c287ddbde2c9
 140.15 KB / 140.15 KB [====================================================] 0s
Writing manifest to image destination
Storing signatures
e3cd6917002af26debf8052e7f1d95ef7ed912111593248cff8de9ca076569e8
$ podman pull quay.io/openshift-release-dev/ocp-release:4.0.0-8
Trying to pull quay.io/openshift-release-dev/ocp-release:4.0.0-8...Getting image source signatures
Skipping fetch of repeat blob sha256:a02a4930cb5d36f3290eb84f4bfa30668ef2e9fe3a1fb73ec015fc58b9958b17
Skipping fetch of repeat blob sha256:47e4121c7dbd4743f868526085b5bb36f4dff5cec8c1a3f992a6b7f2bf06403c
Skipping fetch of repeat blob sha256:ca9c3b8314517310a56ef66741850ee0acce144bb241778cc13635b62fb990b0
Skipping fetch of repeat blob sha256:0f3ada1e64e347b6c2617705d16ba9cc1abc287d7b527b979a55c287ddbde2c9
Writing manifest to image destination
Storing signatures
e3cd6917002af26debf8052e7f1d95ef7ed912111593248cff8de9ca076569e8
$ podman inspect quay.io/openshift-release-dev/ocp-release:4.0.0-8
[
    {
        "Id": "e3cd6917002af26debf8052e7f1d95ef7ed912111593248cff8de9ca076569e8",
        "Digest": "sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777",
        "RepoTags": [
            "quay.io/openshift-release-dev/ocp-release:4.0.0-8",
            "quay.io/openshift-release-dev/ocp-release:none"
        ],
        "RepoDigests": [
            "quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777",
            "quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777"
        ],
...
$ podman version
Version:       0.11.1.1
Go Version:    go1.10.3
Git Commit:    "594495db2637a6e485fffd8aa57c193244e16578"
Built:         Fri Jan  4 16:26:59 2019
OS/Arch:       linux/amd64

Although there is an unnecessary duplicate in RepoDigests.

Should the line you linked earlier be using reference.Parse or similar instead of its current colon splitting? Or were you saying that it should somehow be dropped altogether (although I'm not sure where that leaves RepoDigests)?

[mtrmac]

I suspect a

$ podman pull quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777
$ podman pull quay.io/openshift-release-dev/ocp-release:4.0.0-8

would recreate the situation (but I didn’t try it!).

Not for me:

$ podman inspect quay.io/openshift-release-dev/ocp-release:4.0.0-8
        "RepoTags": [
            "quay.io/openshift-release-dev/ocp-release:4.0.0-8",
            "quay.io/openshift-release-dev/ocp-release:none"
        ],
        "RepoDigests": [
            "quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777",
            "quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777"
        ],

Ah, right. That’s the other bug. In that case, the digested RepoTags value could have been created by CRI-O pulling an image using a digest reference (they do now share the image DB, don’t they?).

Although there is an unnecessary duplicate in RepoDigests.

Sure; the reported RepoDigests array, in the current implementation, has always exactly as many entries as RepoTags, all with the same digest value.

Should the line you linked earlier be using reference.Parse or similar instead of its current colon splitting?

Yes (reference.ParseNormalizedNamed to be consistent with c/image), that would help not mess up the syntax in the short term.

Or were you saying that it should somehow be dropped altogether (although I'm not sure where that leaves RepoDigests)?

Per #1139, instead of actually recording in long-term storage only a Names==RepoTags array, and a single Digest value, I think we should be recording a Names array and a RepoDigests array (actually recording the repo@digest values for every pull, whether pulled by tag or digest) — or maybe embed the repo@digest values into Names==RepoTags directly; then deprecate Digest as ambiguous. At that point the code combining Names+Digest would go away entirely.