installer not completing : incorrect ocp default registry format registry@sha@sha

[kikisdeliveryservice]

Version

$ openshift-install version
bin/openshift-install v0.7.0-master
and
bin/openshift-install v0.7.0

Platform (aws|libvirt|openstack):

aws

What happened?

Ran the installer and got errors related to pulling an image. The location seems to be incorrect and has 2 shas:
Dec 17 19:44:54 ip-10-0-9-101 bootkube.sh[3794]: unable to pull quay.io/openshift-release-dev/ocp-release@sha256@sha256:4f02d5c7183360a519a7c7dbe601f58123c9867cd5721ae503072ae62920575b: error getting default registries to try: invalid reference format

What you expected to happen?

I expect the installer to run

How to reproduce it (as minimally and precisely as possible)?

run the installer, note that it hangs at the below DEBUG lines for 20 minutes:

DEBUG added kube-scheduler.157140aa428fc565: ip-10-0-3-94_a271882c-024e-11e9-ae96-024b8209fb6a became leader DEBUG added kube-controller-manager.157140ab00eeb507: ip-10-0-3-94_a26c1b70-024e-11e9-98fb-024b8209fb6a became leader

until proceeding to the error:

ERROR: logging before flag.Parse: E1217 15:14:13.773270 1521 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=3, ErrCode=NO_ERROR, debug="" WARNING RetryWatcher - getting event failed! Re-creating the watcher. Last RV: 148 WARNING Failed to connect events watcher: Get https://k-api.devcluster.openshift.com:6443/api/v1/namespaces/kube-system/events?resourceVersion=148&watch=true: dial tcp 52.37.184.199:6443: connect: connection refused

the pastebin logs below are occuring at the time that the WARNINGs in the main terminal appear.

Anything else we need to know?

Output from running journalctl -b -u bootkube --no-pager on bootstrap:
http://pastebin.test.redhat.com/685235

[kikisdeliveryservice]

cc: @abhinavdahiya @wking

[cgwalters]

Probably related to this code:

# convert the release image pull spec to an "absolute" form if a digest is available - this is
# safe to resolve after the actions above because podman will not pull the image once it is 
# locally available
if ! release=$( podman inspect {{.ReleaseImage}} -f '{{"{{"}} index .RepoDigests 0 {{"}}"}}' ) || [[ -z "${release}" ]]; then
	echo "Warning: Could not resolve release image to pull by digest" 2>&1
	release="{{.ReleaseImage}}"

Hmm. Why is this a warning instead of fatal? (And shouldn't that be 1>&2?)

[kikisdeliveryservice]

If it's an invalid reference format and that format/target isn't going to change it does seem to make more sense to just make that a fatal error rather than giving Warnings for such a long time only to eventually fail because of it.

So basically there are 2 issues: 1 - double sha256 and 2 that it takes forever for it to finally permanently fail due to a bad format.

[wking]

Excerpted from the linked logs:

Dec 17 18:50:00 ip-10-0-9-101 bootkube.sh[4397]: Pulling release image...
Dec 17 18:50:02 ip-10-0-9-101 bootkube.sh[4397]: Trying to pull quay.io/openshift-release-dev/ocp-release:4.0.0-4...Getting image source signatures

Huh, update payload 4.0.0-4 was installer v0.6.0. Are you sure these logs are from installer v0.7.0?

Here's the hung wait in those logs:

Dec 17 18:57:38 ip-10-0-9-101 bootkube.sh[4397]: Pod Status:openshift-cluster-version/cluster-version-operator-7bff475b99-l7qfb        Pending
Dec 17 19:17:08 ip-10-0-9-101 bootkube.sh[4397]: Error: error while checking pod status: timed out waiting for the condition

And on the next bootkube.service run there's the doubled @sha256 issue:

Dec 17 19:17:14 ip-10-0-9-101 systemd[1]: Started Bootstrap a Kubernetes cluster.
Dec 17 19:17:14 ip-10-0-9-101 bootkube.sh[13694]: unable to pull quay.io/openshift-release-dev/ocp-release@sha256@sha256:4f02d5c7183360a519a7c7dbe601f58123c9867cd5721ae503072ae62920575b: error getting default registries to try: invalid reference format

Both v0.6.0 and v0.7.0 have the same code @cgwalters excerpted above. And once the wrapping templating is expanded, it seems to be working fine for me with my random Podman build from last week:

$ podman version
Version:       0.12.2-dev
Go Version:    go1.10.3
Git Commit:    "ec4cada3d1eabc77d9691a71fe2c99e3bf9343d6-dirty"
Built:         Wed Dec 12 23:58:05 2018
OS/Arch:       linux/amd64
$ podman pull quay.io/openshift-release-dev/ocp-release:4.0.0-4
$ podman inspect quay.io/openshift-release-dev/ocp-release:4.0.0-4 -f '{{ index .RepoDigests 0 }}'
quay.io/openshift-release-dev/ocp-release@sha256:4f02d5c7183360a519a7c7dbe601f58123c9867cd5721ae503072ae62920575b

The next time you see this, can you grab podman version and that inspect command of the hung bootstrap node?

Re: "why the warning?", @smarterclayton motivated it with a reference to broken registries, although I'm still not clear on the details there.

[kikisdeliveryservice]

@wking Haven't run into this since that day, so closing the issue, but will reopen if I see the behaviour again.

[lveyde]

Jan 09 16:27:24 test1-bootstrap bootkube.sh[24200]: unable to pull quay.io/openshift-release-dev/ocp-release@sha256@sha256:e237499d3b118e25890550daad8b17274af93baf855914a9c6f8f07ebc095dea: error getting default registries to try: invalid reference format

[lveyde]

podman version

Version: 0.11.1.1
Go Version: go1.10.2
OS/Arch: linux/amd64

[lveyde]

podman pull quay.io/openshift-release-dev/ocp-release:4.0.0-9
Trying to pull quay.io/openshift-release-dev/ocp-release:4.0.0-9...Getting image source signatures
Skipping fetch of repeat blob sha256:a02a4930cb5d36f3290eb84f4bfa30668ef2e9fe3a1fb73ec015fc58b9958b17
Skipping fetch of repeat blob sha256:47e4121c7dbd4743f868526085b5bb36f4dff5cec8c1a3f992a6b7f2bf06403c
Skipping fetch of repeat blob sha256:ca9c3b8314517310a56ef66741850ee0acce144bb241778cc13635b62fb990b0
Skipping fetch of repeat blob sha256:6266d9236cf1f855894ae5b171bed487a1453703cea1265dae6f6cf1a64b2e76
Writing manifest to image destination
Storing signatures
d0bc83da1db0c2fa8af4ac4176bd3fff1607b18178b38732b5408d72ab7c0784

podman inspect quay.io/openshift-release-dev/ocp-release:4.0.0-9 -f '{{ index .RepoDigests 0 }}'
quay.io/openshift-release-dev/ocp-release@sha256@sha256:e237499d3b118e25890550daad8b17274af93baf855914a9c6f8f07ebc095dea

[wking]

containers/podman#2106

[lveyde]

@wking Thanks for the update!