--insecure-registry does not work with cri-o when using kicbase image

[criztovyl]

when creating a minikube instance using kic driver it looks like the --insecure-registry options are injected to /etc/sysconfig/crio.minikube:

minikube/pkg/provision/provision.go

Lines 241 to 263 in a85e155

	func setCrioOptions(p provision.SSHCommander) error {
	// pass through --insecure-registry
	var (
	crioOptsTmpl = `
	CRIO_MINIKUBE_OPTIONS='{{ range .EngineOptions.InsecureRegistry }}--insecure-registry {{.}} {{ end }}'
	`
	crioOptsPath = "/etc/sysconfig/crio.minikube"
	)
	t, err := template.New("crioOpts").Parse(crioOptsTmpl)
	if err != nil {
	return err
	}
	var crioOptsBuf bytes.Buffer
	if err := t.Execute(&crioOptsBuf, p); err != nil {
	return err
	}
	if _, err = p.SSHCommand(fmt.Sprintf("sudo mkdir -p %s && printf %%s \"%s\" | sudo tee %s && sudo systemctl restart crio", path.Dir(crioOptsPath), crioOptsBuf.String(), crioOptsPath)); err != nil {
	return err
	}
	return nil
	}

But the crio.service in the kcibase image does not include that file:

$ sudo podman ps
CONTAINER ID  IMAGE                                COMMAND  CREATED      STATUS          PORTS                                                                                                                                 NAMES
bdf7c0ce6328  gcr.io/k8s-minikube/kicbase:v0.0.30           5 hours ago  Up 5 hours ago  127.0.0.1:32883->22/tcp, 127.0.0.1:42673->2376/tcp, 127.0.0.1:34131->5000/tcp, 127.0.0.1:44439->8443/tcp, 127.0.0.1:44799->32443/tcp  minikube

$ minikube config view
- container-runtime: cri-o
- driver: podman
- insecure-registry: host.minikube.internal:5000

$ sudo podman exec minikube bash -c "cat /etc/sysconfig/crio.minikube"

CRIO_MINIKUBE_OPTIONS='--insecure-registry 10.96.0.0/12 --insecure-registry host.minikube.internal:5000 '

# https://manpages.ubuntu.com/manpages/bionic/man5/systemd.unit.5.html#unit%20file%20load%20path
$ sudo podman exec minikube bash -c "tail -n+0 {/etc/systemd/system,/run/systemd/system,/lib/systemd/system}/crio.service"
tail: cannot open '/etc/systemd/system/crio.service' for reading: No such file or directory
tail: cannot open '/run/systemd/system/crio.service' for reading: No such file or directory
==> /lib/systemd/system/crio.service <==
[Unit]
Description=Container Runtime Interface for OCI (CRI-O)
Documentation=https://github.com/cri-o/cri-o
Wants=network-online.target
Before=kubelet.service
After=network-online.target

[Service]
Type=notify
EnvironmentFile=-/etc/default/crio
Environment=GOTRACEBACK=crash
ExecStart=/usr/bin/crio \
          $CRIO_CONFIG_OPTIONS \
          $CRIO_RUNTIME_OPTIONS \
          $CRIO_STORAGE_OPTIONS \
          $CRIO_NETWORK_OPTIONS \
          $CRIO_METRICS_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
TasksMax=infinity
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
OOMScoreAdjust=-999
TimeoutStartSec=0
Restart=on-abnormal

[Install]
WantedBy=multi-user.target
Alias=cri-o.service

[criztovyl]

this is similar to #11729, but there crio.service was still reachable via /etc/systemd/system/multi-user.target.wants/crio.service -- which it no longer is. this suggests something changed in the image.
edit: #11729 is --vm-driver=kvm2, I am using --driver=podman.

[afbjorklund]

Different packaging, with the custom /etc/sysconfig/crio.minikube versus the vendor /etc/default/crio

EnvironmentFile=-/etc/sysconfig/crio
EnvironmentFile=-/etc/sysconfig/crio.minikube
EnvironmentFile=/var/run/minikube/env

ExecStart=/usr/bin/crio \
    $CRIO_OPTIONS \
    $CRIO_MINIKUBE_OPTIONS

EnvironmentFile=-/etc/default/crio

ExecStart=/usr/bin/crio \
          $CRIO_CONFIG_OPTIONS \
          $CRIO_RUNTIME_OPTIONS \
          $CRIO_STORAGE_OPTIONS \
          $CRIO_NETWORK_OPTIONS \
          $CRIO_METRICS_OPTIONS

Moving the config to /etc/default and adding to the ISO package, seems like the easiest workaround ?

[afbjorklund]

Need to merge with these files: https://github.com/cri-o/cri-o/tree/main/contrib/systemd/ from the git.

Over to: https://github.com/kubernetes/minikube/tree/master/deploy/iso/minikube-iso/package/crio-bin

[criztovyl]

workaround is

minishift ssh "echo 'CRIO_CONFIG_OPTIONS=--insecure-registry host.minikube.internal:5000' | sudo tee -a /etc/default/crio >/dev/null; sudo systemctl restart crio"

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.