update registry addon documentation

[tliron]

The exact command to reproduce the issue:

minikube start --addons=registry --container-runtime=cri-o

The full output of the command that failed:

Pods fail to pull image from the registry, e.g.:

image: localhost:5000/mycontainer

Note that I can push to the registry from outside via docker://$(minikube ip):5000/mycontainer.

Switching to Docker runtime allows this to work.

The operating system version:

Fedora 31

[afbjorklund]

You are saying that the registry works with --container-runtime=docker, but not with --container-runtime=cri-o ? (the above command-line is more talking about CNI options, rather than CRI...)

From the outside, it is normally more like: https://minikube.sigs.k8s.io/docs/tasks/docker_registry/

[tliron]

Ugh, so sorry, did a wrong copy and paste. Will edit the original issue,

[afbjorklund]

Do you have the error output, like if you do a describe pod on it ?

[tliron]

These are the events:

16s         Normal    Pulling             pod/mycontainer-f7dcbd7f6-tx9nc    Pulling image "localhost:5000/mycontainer"
28s         Warning   Failed              pod/mycontainer-f7dcbd7f6-tx9nc    Failed to pull image "localhost:5000/mycontainer": rpc error: code = Unknown desc = error pinging docker registry localhost:5000: Get http://localhost:5000/v2/: dial tcp 127.0.0.1:5000: i/o timeout

Nothing much more in describe, just the fail reason being ImagePullBackOff.

[afbjorklund]

There doesn't seem to be anything in the registry logs, so I guess it is getting stuck in registry-proxy

It works from the outside, but just hangs from the inside. Even with curl, so something is wrong...

The internal IP are fine:

$ curl http://10.1.0.2:5000/v2/
{}$ 
$ curl http://10.1.0.3:80/v2/
{}$ 

But not the localhost.

$ curl http://192.168.99.100:5000/v2/
{}$ 
$ curl http://localhost:5000/v2/
curl: (7) Failed to connect to localhost port 5000: Connection timed out

[afbjorklund]

Looks similar to cri-o/cri-o#1804

[tliron]

Thank you for verifying. Any ideas of a temporary workaround?

[afbjorklund]

Note that it has to be localhost, since that is the hack used to get an insecure registry...

Failed to pull image "192.168.99.100:5000/test-img": rpc error: code = Unknown desc = error pinging docker registry 192.168.99.100:5000: Get https://192.168.99.100:5000/v2/: http: server gave HTTP response to HTTPS client

The workaround is configuring the minikube IP as an insecure registry, in the crio.conf

# List of registries to skip TLS verification for pulling images. Please   
# consider configuring the registries via /etc/containers/registries.conf before
# changing them here.                                                          
#insecure_registries = "[]"                                                      

EDIT: Apparently we are using /etc/containers/registry.conf for this as well, so:

[registries.search]
registries = ['docker.io']

[registries.insecure]
registries = ['192.168.99.100:5000']                                   

This would be for the track where you use the IP instead of the localhost proxy:

kubectl run test-img --image=$(minikube ip):5000/test-img

---

Or deploy a proper https registry (left as an exercise for the reader :-) )

https://docs.docker.com/registry/deploying/#run-an-externally-accessible-registry

Instead of the horrible hack, that is the kubernetes registry-proxy container...

kubernetes/kubernetes@d6918bb

The private registry runs as a Pod in your cluster. It does not currently support SSL or authentication, which triggers Docker's "insecure registry" logic. To work around this, we run a proxy on each node in the cluster, exposing a port onto the node (via a hostPort), which Docker accepts as "secure", since it is accessed by localhost.

[tliron]

Hm, so it looks like cri-o doesn't support Docker's "localhost is allowed as an insecure registry" hack? If that's the case, I think it should be documented that the registry addon doesn't work with cri-o, at least this time.

[afbjorklund]

Yeah, it seems like hostPort has some issues with cri-o and that should probably be documented...

[afbjorklund]

Hm, so it looks like cri-o doesn't support Docker's "localhost is allowed as an insecure registry" hack?

That part is supported just fine, since it tries to be bug-compatible at least for the docker:// protocol

[tliron]

I found a simple workaround. Get the registry service's cluster IP:

kubectl get svc registry --output=jsonpath='{.spec.clusterIP}' --namespace=kube-system

And then use that address, with port 80, for the image URL, e.g.:

image: 10.102.27.206:80/mycontainer

[priyawadhwa]

Hey @tliron awesome. We should update the documentation to include that workaround when trying to use the registry addon with cri-o.

[tliron]

@priyawadhwa Thanks, but it's an awkward solution. The problem is that users would have to update all their specs to use this varying IP address. I hope we can find a way to enable localhost:5000 for CRI-O (which, we hope, will become minikube's default runtime).

[tliron]

Here is a version of my workaround above in Go code:

import (
	"context"
	"fmt"
	meta "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/client-go/kubernetes" // v0.18.0
)

func GetInternalRegistryURL(kubernetesClient *kubernetes.Clientset) (string, error) {
	if service, err := kubernetesClient.CoreV1().Services("kube-system").Get(context.TODO(), "registry", meta.GetOptions{}); err == nil {
		return fmt.Sprintf("%s:80", service.Spec.ClusterIP), nil
	} else {
		return "", err
	}
}

[charandas]

The same thing happens with ingress addon. No go with cri-o runtime. The minikube ip does not host any loadbalancer, and it fails silently, while internally, the ingress objects bind to minikube ip just fine.

[medyagh]

I would be happy to review any PR that fixes this for CRI-O

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[CtrlZvi]

/remove-lifecycle rotten
To the best of my knowledge, this is still a valid issue.

[tliron]

I think an acceptable solution is to document this in Minikube, probably here. To be honest, that whole page could use a rewrite, as it's more a collection of notes than a guide. As it stands right now, the only way to really learn what --addons=registry does is to look at blogs.

I'd like to help, but I'm unfamiliar with the project interactions. Could someone who knows please take responsibility for resolving this issue?

[medyagh]

I think an acceptable solution is to document this in Minikube, probably here. To be honest, that whole page could use a rewrite, as it's more a collection of notes than a guide. As it stands right now, the only way to really learn what --addons=registry does is to look at blogs.

I'd like to help, but I'm unfamiliar with the project interactions. Could someone who knows please take responsibility for resolving this issue?

I agree with you @tliron that page is getting very messy and hard to read, I would accept a PR that organizes that page on our webiste and makes it easier to read

[tliron]

It sounds like I'm being volunteered. :) I may try to take a stab at it if I can find the time.

It's all very hard, to be honest. The challenge is not just to allow Kubernetes's container runtime access to the private registry (with authentication? authorization?) but also how a user/developer can access it push images to it. After all, that is the goal of having such a built-in registry in the first place. None of this work is trivial. But at least it can be documented specifically for Minikube, which does have some advantages in that it can run in "insecure" mode (self-signed cert).

On that note, I ended up creating a generic solution to the problem of handling and working with private registries, a project I call Reposure. Reposure has explicit support for Minikube's registry add-on.

[medyagh]

It sounds like I'm being volunteered. :) I may try to take a stab at it if I can find the time.

It's all very hard, to be honest. The challenge is not just to allow Kubernetes's container runtime access to the private registry (with authentication? authorization?) but also how a user/developer can access it push images to it. After all, that is the goal of having such a built-in registry in the first place. None of this work is trivial. But at least it can be documented specifically for Minikube, which does have some advantages in that it can run in "insecure" mode (self-signed cert).

On that note, I ended up creating a generic solution to the problem of handling and working with private registries, a project I call Reposure. Reposure has explicit support for Minikube's registry add-on.

@tliron thank you That does sounds like a good idea to update our docs and maybe refactor our docs on registry addon with better examples and better categorization

[k8s-triage-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten