crio: Unable to pull images form internal registry

[kameshsampath]

minikube version: v1.16.0
commit: 9f1e482427589ff8451c4723b6ba53bb9742fbb1

Steps to reproduce the issue:

1. Start Minikube as

minikube start -p $PROFILE_NAME \
  --memory=$MEMORY --cpus=$CPUS \
  --disk-size=50g \
  --insecure-registry='10.0.0.0/24'

2. Apply the registry and registry-aliases add-on
3. Build and push an image to the Minikube registry as example.com/demo/greeter
4. Deploy a pod using the command

kubectl run demo-greeter -n tektontutorial \
 --generator='run-pod/v1' \
 --image='example.com/demo/greeter' && \
kubectl expose pod demo-greeter -n tektontutorial --port 8080 --type=NodePort

The kubectl get events shows the crashed pod with the following logs:

Full output of failed command:

0s          Warning   Failed      pod/demo-greeter                        Failed to pull image "registry.minikube/rhdevelopers/tekton-tutorial-greeter": rpc error: code = Unknown desc = error pinging docker registry registry.minikube: Get https://registry.minikube/v2/: dial tcp 10.101.166.84:443: connect: connection refused

With --insecure-registry specified, the pod still tries to resolve the url as https than http

CC: @afbjorklund

[kameshsampath]

some more contextual info

minikube ssh -- cat /etc/hosts

127.0.0.1       localhost
127.0.1.1 tektontutorial
192.168.64.1    host.minikube.internal
192.168.64.20   control-plane.minikube.internal
10.101.166.84   example.org
10.101.166.84   example.com
10.101.166.84   test.com
10.101.166.84   test.org
10.101.166.84   registry.minikube

kubectl get svc -n kube-system

AME       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10      <none>        53/UDP,53/TCP,9153/TCP   79m
registry   ClusterIP   *10.101.166.84*   <none>        80/TCP,443/TCP           78m

kubectl get cm coredns -n kube-system -o yaml | yq eval '.data' -

Corefile: |-
  .:53 {
      errors
       rewrite name example.org  registry.kube-system.svc.cluster.local
  rewrite name example.com  registry.kube-system.svc.cluster.local
  rewrite name test.com  registry.kube-system.svc.cluster.local
  rewrite name test.org  registry.kube-system.svc.cluster.local
  rewrite name registry.minikube  registry.kube-system.svc.cluster.local

      health {
         lameduck 5s
      }
      ready
      kubernetes cluster.local in-addr.arpa ip6.arpa {
         pods insecure
         fallthrough in-addr.arpa ip6.arpa
         ttl 30
      }
      prometheus :9153
      forward . /etc/resolv.conf {
         max_concurrent 1000
      }
      cache 30
      loop
      reload
      loadbalance
  }

[afbjorklund]

Worked here.

  Type     Reason     Age              From               Message
  ----     ------     ----             ----               -------
  Normal   Scheduled  4s               default-scheduler  Successfully assigned default/busybox to minikube
  Normal   Pulled     4s               kubelet            Successfully pulled image "registry.minikube/busybox" in 16.245797ms
  Normal   Pulling    3s (x2 over 4s)  kubelet            Pulling image "registry.minikube/busybox"
  Normal   Created    3s (x2 over 4s)  kubelet            Created container busybox
  Normal   Started    3s (x2 over 3s)  kubelet            Started container busybox
  Normal   Pulled     3s               kubelet            Successfully pulled image "registry.minikube/busybox" in 18.747139ms
  Warning  BackOff    1s (x2 over 2s)  kubelet            Back-off restarting failed container

You might want to double-check minikube ssh docker info:

 Insecure Registries:
  10.96.0.0/12
  10.0.0.0/24
  127.0.0.0/8

[kameshsampath]

that sound weird ! was it on macOS ?

Let me try again on a fresh instance, do you see any issue with minikube start command ??

[kameshsampath]

after a bit of debug @afbjorklund , this happens only with container-runtime=cri-o, if I use container-runtime=docker then all works well. Wondering something to do with internal work on cri-o?

[medyagh]

@kameshsampath interesting do we still have this issue on latest minikube ? I wonder if this issue happens if we specify a different CNI ?

[kameshsampath]

still get the same issue with crio, not sure what CNI driver I can use on macOS.

…

________________________________ From: Medya Ghazizadeh <notifications@github.com> Sent: Thursday, March 4, 2021 1:03 AM To: kubernetes/minikube <minikube@noreply.github.com> Cc: Kamesh Sampath <kamesh.sampath@hotmail.com>; Mention <mention@noreply.github.com> Subject: Re: [kubernetes/minikube] crio: Unable to pull images form internal registry (#10171) @kameshsampath<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkameshsampath&data=04%7C01%7C%7C00d057535cea4f79d76208d8de7b3c39%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637503968168281687%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=VTpOI4Z98MweIKCzJepHxP1nzePl%2BSF0aZTki7FhKmE%3D&reserved=0> interesting do we still have this issue on latest minikube ? I wonder if this issue happens if we specify a different CNI ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkubernetes%2Fminikube%2Fissues%2F10171%23issuecomment-789999613&data=04%7C01%7C%7C00d057535cea4f79d76208d8de7b3c39%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637503968168291684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=nPweTXdxmB6yEC15VbgvkkoExj9gfuV0G7mGH%2B4Zgu0%3D&reserved=0>, or unsubscribe<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAHHMILTS6VCW3ABYI2ORALTB2FI3ANCNFSM4WIK3JNA&data=04%7C01%7C%7C00d057535cea4f79d76208d8de7b3c39%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637503968168301691%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=r13cQSiqkXiJOP2%2BFJNBCqWctcAQST4M55tCVf%2BtNsE%3D&reserved=0>.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[k8s-triage-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ricardozanini]

Just for your information, I got in this issue today with the latest Minikube version. Same setup MacOS, podman, and cri-o. I also started the cluster with the flag --insecure-registry=10.0.0.0/24. The kubelet controller insists on using HTTPS instead of HTTP.

The problem is that the registry service is created exposing the ports 80:5000 and 443:443. But the registry pod, only exposes 5000. So obviously, kubelet or any client won't be able to pull from 443.

I think this is a bug @kameshsampath @medyagh

Cheers!

[ricardozanini]

/reopen

[k8s-ci-robot]

@ricardozanini: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[medyagh]

he latest Minikube version. Same setup MacOS, podman, and cri-o. I also started the cluster with the flag --insecure-registry=10.0.0.0/24. The kubelet controller insists on using HTTPS instead of HTTP.

The problem is that the registry service is created exposing the ports 80:5000 and 443:443. But the registry pod, only exposes 5000. So obviously, kubelet or any client won't be able to pull from 443.

ricardozanini
would you be interested to make a PR to fix this ?

of the issue is Container needs to open a non 443 port you can do that in the Kic Package

[ricardozanini]

@medyagh you bet I would! :)

I'll take a look later today and send a PR.