Using with podman

[ccakes]

I'm trying to use this with podman using the Docker API-compat socket. It seems like it'll probably work but I've hit a snag and don't know enough about the Docker ecosystem to know the "right solution" for a PR.

Running ctlptl apply with config to create a registry fails. It creates the container but then can't find it to proceed with the workflow. The problem is this line

ctlptl/pkg/registry/registry.go

Line 106 in f5d2f38

filterArgs.Add("ancestor", "registry:2") // The registry everyone uses.

If I change that to use the fully-qualified path docker.io/library/registry:2 then it works fine. I'm unsure if that would also work on native Docker though, or if instead I should add a check for podman and set the filter depending on Docker v podman. Thoughts?

[nicks]

ooh nice! Ya, would love to accept a PR that gets this working with podman

I just tried that with the docker CLI and it seemed to work OK. We also have pretty good integration tests that will run on CI, so don't worry too much about breaking it.

[nicks]

@ccakes what version of podman are you using? i was playing around with this and couldn't get the podman container ls --filter to work at all 🤔 (on podman 3.0.1)

[nicks]

OK I upgraded to podman 3.4 and now see the problem. Sent out: https://github.com/tilt-dev/ctlptl/pull/new/nicks/issue146

Unfortunately, kind creation still blows up:

failed to create cluster: failed to ensure docker network: command "docker network create -d=bridge -o com.docker.network.bridge.enable_ip_masquerade=true --ipv6 --subnet fc00:f853:ccd:e793::/64 kind" failed with error: exit status 1

:\

[ccakes]

Sorry for the slow reply - I was using latest podman. I got about as far as you, I suspect that making it work for rootful containers is probably not that much work but rootless (how I was trying) would be much more difficult.

For rootless, it feels like the best option would be to create a pod and run both kind and the registry together in that but doing so would require more changes to ctlptl than I really feel comfortable tackling.

[almereyda]

Thanks to

• registry: make registry creation more robust #150

we now have a working registry with Podman. Using Podman pods to create network contexts still appears out of scope.

Meanwhile it happens that a specific networking setup for K3d clusters on Podman would also be required for connecting a K3d registry to a K3d cluster (on Podman):

• cluster: create k3d from a config file. #265 (comment)

[ncdc]

I'm using ctlptl with podman but I had to create a config file in the podman VM to make it treat the registry created by ctlptl as insecure (otherwise I couldn't use it):

$ cat /etc/containers/registries.conf.d/100-kind.conf
[[registry]]
location = "localhost:61358"
insecure = true

A few questions:

• Should I file a new issue for this discussion?
• Is there any other way to do this that doesn't require manually modifying files on the podman VM?
• If not, would it be useful to do a PR that tries to manage this config for you?

[aripollak]

Besides the registries.conf addition (I put mine in ~/.config/containers/registries.conf), for running native rootless podman under Ubuntu 23.10 (upgraded from 22.04) I also had to install the golang-github-containernetworking-plugin-dnsname package so the kind container could resolve the ctlptl-registry hostname. Since I already had the kind network created in podman, I had to run podman network rm kind and recreate the cluster/registry with ctlptl after installing the -plugin-dnsname package.

I don't think ctlptl could/should install the plugin automatically, so that might just be something that just goes in the instructions, and it might also not be needed with a fresh Ubuntu install where you've only ever used podman 4.0+, since it then uses netavark by default, which has DNS support built in. Based on https://github.com/containers/podman/blob/main/docs/tutorials/basic_networking.md#bridge, the DNS lookup issue is also be fixable by destroying all podman resources under podman 4.0+ which should switch to netavark.

[aaomidi]

Creating local k8s clusters
Creating registry "ctlptl-registry"...
registry.ctlptl.dev/ctlptl-registry created
reading docker-desktop settings: Get "http://localhost/app/settings": dial unix /Users/amir/Library/Containers/com.docker.docker/Data/backend.sock: connect: no such file or directory
make: *** [Makefile:120: dev-env-cluster] Error 1

Another fun little issue with podman. ctlptl assumes its docker-desktop and tries to do docker-desktop things to it.

[nicks]

@aaomidi ??? what command are you running? we have a lot of integration tests. the machines they run on don't even have docker-desktop installed...so that seems like an unrelated issue...

[jzhn]

Creating local k8s clusters
Creating registry "ctlptl-registry"...
registry.ctlptl.dev/ctlptl-registry created
reading docker-desktop settings: Get "http://localhost/app/settings": dial unix /Users/amir/Library/Containers/com.docker.docker/Data/backend.sock: connect: no such file or directory
make: *** [Makefile:120: dev-env-cluster] Error 1

Another fun little issue with podman. ctlptl assumes its docker-desktop and tries to do docker-desktop things to it.

I can reproduce the same issue...

Environment Information

• ctlptl: latest main branch (5282f72)
• podman: 5.2.5
• docker cli: 27.3.1

Steps to reproduce

ctlptl apply -f below yaml.

apiVersion: ctlptl.dev/v1alpha1
kind: Registry
name: registry
port: 5005
---
apiVersion: ctlptl.dev/v1alpha1
kind: Cluster
name: kind-foo
product: kind
registry: registry
kindV1Alpha4Cluster:
  networking:
    apiServerAddress: 0.0.0.0
  nodes:
    - role: control-plane
    - role: worker
    - role: worker
    - role: worker

Observations

It seems that ctlptl always assumes docker desktop machine when product is kind:

ctlptl/pkg/cluster/cluster.go

Lines 191 to 200 in 5282f72

	case clusterid.ProductDockerDesktop, clusterid.ProductKIND, clusterid.ProductK3D:
	if c.dmachine == nil {
	machine, err := NewDockerMachine(ctx, dockerCLI.Client(), c.iostreams)
	if err != nil {
	return nil, err
	}
	c.dmachine = machine
	}
	return c.dmachine, nil

And the error is returned from the attempt of reading the settings from docker desktop.

cluster.DockerDesktopClient.settings (docker_desktop.go:313) github.com/tilt-dev/ctlptl/pkg/cluster
<autogenerated>:2
cluster.(*dockerMachine).Restart (machine.go:142) github.com/tilt-dev/ctlptl/pkg/cluster
cluster.(*Controller).Apply (cluster.go:745) github.com/tilt-dev/ctlptl/pkg/cluster
cmd.(*ApplyOptions).run (apply.go:124) github.com/tilt-dev/ctlptl/pkg/cmd
cmd.(*ApplyOptions).Run (apply.go:58) github.com/tilt-dev/ctlptl/pkg/cmd
<autogenerated>:2
cobra.(*Command).execute (command.go:989) github.com/spf13/cobra
cobra.(*Command).ExecuteC (command.go:1117) github.com/spf13/cobra
cobra.(*Command).Execute (command.go:1041) github.com/spf13/cobra
main.main (main.go:34) main
runtime.main (proc.go:272) runtime
runtime.goexit (asm_arm64.s:1223) runtime
 - Async Stack Trace
<autogenerated>:2

[jzhn]

@aaomidi @nicks : I found a workaround for above issue with ctlptl trying to modify docker-desktop settings during ctlptl apply on podman environment.

From

ctlptl/pkg/docker/docker.go

Lines 34 to 38 in 5282f72

	return strings.Contains(dockerHost, "/var/run/docker.sock") ||
	// Docker Desktop for Linux - socket is in ~/.docker/desktop/docker.sock
	strings.HasSuffix(dockerHost, "/.docker/desktop/docker.sock") ||
	// Docker Desktop for Mac 4.13+ - socket is in ~/.docker/run/docker.sock
	strings.HasSuffix(dockerHost, "/.docker/run/docker.sock")

ctlplt detects if it's "local" docker by checking the docker daemon socket at some typical default locations. Only when it's local docker desktop, it'll try to call its APIs (which will fail for podman).

ctlptl/pkg/cluster/machine.go

Line 142 in 5282f72

settings, err := m.d4m.settings(ctx)

Usually for a podman installation with docker compatibility, we'll synlink the podman socket to the docker socket location, like I did here:

lrwxr-xr-x  1 root  daemon  64 20 Nov 11:32 /var/run/docker.sock -> /Users/foo/.local/share/containers/podman/machine/podman.sock

A quick workaround is to export DOCKER_HOST=<your podman sock> directly, which will make sure ctlptl to bypass above logic.