Running containers inside of a container environment (with docker-compose.yml) using podman?

[Conan-Kudo]

/kind feature

Description

I'd like to be able to run and test batches of containers defined with docker-compose.yml. As it is now, doing this with actual Docker inside an environment that runs through Docker gets rather risky and leaky in all kinds of bad ways.

For building containers, I'm starting to use buildah for this, but I don't quite yet have an answer for running them. The goal is to be able to build and test in a manner that is consistent with how people can do it on their local machines, and easily transition to OpenShift for production run environments.

Additional environment details (AWS, VirtualBox, physical, etc.):
GitLab CI runners with Docker container (of Fedora with buildah + podman)

[mheon]

I'd love to get something similar to docker compose up and running (I don't know if we'd go for exact compatability in this case, though). The ability to define a pod with a number of containers, sharing various namespaces and starting in a specific order as needed, is already baked into the backend, though work would be needed to make a good user interface to expose all of that.

[giuseppe]

slightly related: we need to ensure that we correctly pass NOTIFY_SOCKET from systemd down to runC. We will not only have startup ordering with systemd dependencies, but containers wouldn't need to poll for another service to be ready (and if they do, it will return immediately) if it configured to use NOTIFY_SOCKET.

[rhatdan]

@giuseppe I worked on that a while ago. But not sure if I got it all working. Would also like to get socket activation working properly. Both would be cool features that don't work in Docker.

[rhatdan]

@edsantiago @TomSweeneyRedHat Could you guys attempt to setup a test to make sure NOTIFY_SOCKET and SD_NOTIFY works with podman?

[edsantiago]

@rhatdan in progress ... but infuriatingly nonworking. And according to my notes, reminiscent of my frustrations in October 2016. It Just Ain't Working. The simplest reproducer I can come up with is:

# NOTIFY_SOCKET=/run/systemd/notify podman run --rm fedora date

This just hangs. No error, also no output. It also hangs in such a way that podman ps also hangs -- maybe the locking issue in #658? /bin/ps reports a long-running, presumably-also-hung runc create process:

# ps auxww --forest |grep -5 runc
...
root     20545  0.0  0.0  86084  1808 ?        Ssl  20:50   0:00 /usr/libexec/crio/conmon -c ced645cf9da7b61613126a938300b796c47ec76ae275dda4d76719af101949b7 -u ced645cf9da7b61613126a938300b796c47ec76ae275dda4d76719af101949b7 -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/ced645cf9da7b61613126a938300b796c47ec76ae275dda4d76719af101949b7/userdata -p /var/run/containers/storage/overlay-containers/ced645cf9da7b61613126a938300b796c47ec76ae275dda4d76719af101949b7/userdata/pidfile -l /var/lib/containers/storage/overlay-containers/ced645cf9da7b61613126a938300b796c47ec76ae275dda4d76719af101949b7/userdata/ctr.log --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket
root     20546  0.0  0.2 392772 10656 ?        Sl   20:50   0:00  \_ /usr/bin/runc create --bundle /var/lib/containers/storage/overlay-containers/ced645cf9da7b61613126a938300b796c47ec76ae275dda4d76719af101949b7/userdata --pid-file /var/run/containers/storage/overlay-containers/ced645cf9da7b61613126a938300b796c47ec76ae275dda4d76719af101949b7/userdata/pidfile ced645cf9da7b61613126a938300b796c47ec76ae275dda4d76719af101949b7
root     20555  0.1  0.2 316848  8836 ?        Ssl  20:50   0:00      \_ /usr/bin/runc init

The init process is not killable, even with -9. The create process can be killed, but only with -9. Attempting to podman rm the container while runc is running results in:

failed to delete container ced645cf9da7b61613126a938300b796c47ec76ae275dda4d76719af101949b7: cgroups: unable to remove paths /sys/fs/cgroup/systemd/libpod_parent/libpod-ced645cf9da7b61613126a938300b796c47ec76ae275dda4d76719af101949b7, /sys/fs/cgroup/freezer/libpod_parent/libpod-ced645cf9da7b61613126a938300b796c47ec76ae275dda4d76719af101949b7, /sys/fs/cgroup/pids/libpod_parent/libpod-ced645cf9da7b61613126a938300b796c47ec76ae275dda4d76719af101949b7, /sys/fs/cgroup/net_cls/libpod_parent/libpod-ced645cf9da7b61613126a938300b796c47ec76ae275dda4d76719af101949b7, /sys/fs/cgroup/net_prio/libpod_parent/libpod-ced645cf9da7b61613126a938300b796c47ec76ae275dda4d76719af101949b7, /sys/fs/cgroup/perf_event/libpod_parent/libpod-ced645cf9da7b61613126a938300b796c47ec76ae275dda4d76719af101949b7, /sys/fs/cgroup/cpuset/libpod_parent/libpod-ced645cf9da7b61613126a938300b796c47ec76ae275dda4d76719af101949b7, /sys/fs/cgroup/cpu/libpod_parent/libpod-ced645cf9da7b61613126a938300b796c47ec76ae275dda4d76719af101949b7, /sys/fs/cgroup/cpuacct/libpod_parent/libpod-ced645cf9da7b61613126a938300b796c47ec76ae275dda4d76719af101949b7, /sys/fs/cgroup/memory/libpod_parent/libpod-ced645cf9da7b61613126a938300b796c47ec76ae275dda4d76719af101949b7, /sys/fs/cgroup/blkio/libpod_parent/libpod-ced645cf9da7b61613126a938300b796c47ec76ae275dda4d76719af101949b7, /sys/fs/cgroup/devices/libpod_parent/libpod-ced645cf9da7b61613126a938300b796c47ec76ae275dda4d76719af101949b7, /sys/fs/cgroup/hugetlb/libpod_parent/libpod-ced645cf9da7b61613126a938300b796c47ec76ae275dda4d76719af101949b7

Same results when running from a systemd init file (without the NOTIFY_SOCKET= declaration, since systemd presumably sets that). Using NOTIFY_SOCKET=/nonexistentfile works perfectly fine.

Am close to giving up for today. This has taken a good chunk of time.

[mheon]

@edsantiago Separate locking issue from #658 - this is us holding the container lock until runc has finished executing, to try and order container operations. The root cause here appears to be the runc init hang.

[rhatdan]

This seems to be close, although I am getting Connection Refused.

#!/bin/sh
export NOTIFY_SOCKET=/run/podman_notify.sock
$(rm -f ${NOTIFY_SOCKET}; nc -U ${NOTIFY_SOCKET} -l) &
sleep 1
podman run -v /usr/bin/nc:/usr/bin/nc fedora /usr/bin/nc -U ${NOTIFY_SOCKET}<<EOF
echo ready
EOF

# sh -x notify_sock.sh
+ export NOTIFY_SOCKET=/run/podman_notify.sock
+ NOTIFY_SOCKET=/run/podman_notify.sock
+ sleep 1
++ rm -f /run/podman_notify.sock
++ nc -U /run/podman_notify.sock -l
+ podman run fedora mount
+ grep podman
tmpfs on /run/podman_notify.sock type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
+ podman run -v /usr/bin/nc:/usr/bin/nc fedora /usr/bin/nc -U /run/podman_notify.sock
Ncat: Connection refused.

The passing of the socket is there and the mounting of the socket is there. I don't know why ncat is refusing the connection.

[edsantiago]

I think systemd uses a DGRAM socket, so you need --udp. In my runs this morning, this hangs consistently:

window1# rm -f /run/mysock;ncat -l -U --udp /run/mysock

window2# NOTIFY_SOCKET=/run/mysock podman run --rm fedora date     (hangs as described above)

It does not hang without --udp, but my sdnotify test container fails with EPROTOTYPE (Protocol wrong type for socket).

[rhatdan]

Date is not doing anything with the socket, so this looks like the integration between podman/runc and the socket file is causing issues. I will see if I can repeat the failure on my machine.

[edsantiago]

Yes - my use of date was simply to try the simplest container that would not be mucking with sd_notify. ISTM that runc is the bottleneck

[rhatdan]

Seems to be working for me now

# NOTIFY_SOCKET=/run/mysock podman run --rm fedora date  
Thu May 24 11:43:32 UTC 2018

With podman in master.

[rhatdan]

Never mind It is hanging.

[rhatdan]

This is where runc is hanging.
openat(AT_FDCWD, "/proc/self/fd/4", O_WRONLY|O_CLOEXEC

[edsantiago]

I'm somewhat leaning toward it being a runc issue, not podman, but have no actual evidence to base that on.