systemd not always producing the %t from systemd file

[ikke-t]

/kind bug

Description

I followed the blog to use conmon-pidfile and cidfile from this blog. However, systemd doesn't fills %t for podman user containers.

ping @vrothberg

Steps to reproduce the issue:

1. Ansible does this systemd file:

cat /etc/systemd/system/ha-container-pod.service 
[Unit]
Description=ha Podman Container
After=network.target

[Service]
Type=simple
TimeoutStartSec=15
ExecStartPre=-/usr/bin/rm -f /%t/%n-pid /%t/%n-cid
User=redhatter

ExecStart=/usr/bin/podman run --name ha \
  --rm -p 8123:8123/tcp -v "/var/lib/containers/storage/homeassistant-config:/config:Z" --hostname="ha.localnet" --memory=1024M \
  --conmon-pidfile  /%t/%n-pid --cidfile /%t/%n-cid \
  homeassistant/home-assistant:stable

ExecStop=/usr/bin/sh -c "/usr/bin/podman stop -t "15" `cat /%t/%n-cid`"
ExecStop=/usr/bin/sh -c "/usr/bin/podman rm -f `cat /%t/%n-cid`"
Restart=on-failure
RestartSec=30
KillMode=none
PIDFile=/%t/%n-pid

[Install]
WantedBy=multi-user.target

2 add user sudo useradd redhatter
3. reload systemd sudo systemctl daemon-reload
4. start container: sudo systemctl start ha-container-pod.service

Describe the results you received:

sudo journalctl -eu ha-container-pod.service
Mar 20 17:12:57 gr8.localdomain systemd[1]: Started ha Podman Container.
Mar 20 17:12:57 gr8.localdomain podman[11501]: Error: error opening cidfile //run/ha-container-pod.service-cid
Mar 20 17:12:57 gr8.localdomain systemd[1]: ha-container-pod.service: Main process exited, code=exited, status=125/n/a
Mar 20 17:12:57 gr8.localdomain systemd[1]: ha-container-pod.service: Failed with result 'exit-code'.
Mar 20 17:13:27 gr8.localdomain systemd[1]: ha-container-pod.service: Service RestartSec=30s expired, scheduling restart.
Mar 20 17:13:27 gr8.localdomain systemd[1]: ha-container-pod.service: Scheduled restart job, restart counter is at 46.
Mar 20 17:13:27 gr8.localdomain systemd[1]: Stopped ha Podman Container.

Describe the results you expected:

start the container with proper path in cidfile

Additional information you deem important (e.g. issue happens only occasionally):

always

Output of podman version:

Version:            1.6.4
RemoteAPI Version:  1
Go Version:         go1.12.12
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.12.12
  podman version: 1.6.4
host:
  BuildahVersion: 1.12.0-dev
  CgroupVersion: v1
  Conmon:
    package: conmon-2.0.6-1.module+el8.1.1+5259+bcdd613a.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.6, commit: 6ffbb2ec70dbe5ba56e4bfde946fb04f19dd8bbf'
  Distribution:
    distribution: '"rhel"'
    version: "8.1"
  MemFree: 526786560
  MemTotal: 16276774912
  OCIRuntime:
    name: runc
    package: runc-1.0.0-64.rc9.module+el8.1.1+5259+bcdd613a.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 8167911424
  SwapTotal: 8292134912
  arch: amd64
  cpus: 8
  eventlogger: journald
  hostname: gr8.localdomain
  kernel: 4.18.0-147.5.1.el8_1.x86_64
  os: linux
  rootless: false
  uptime: 76h 28m 48.87s (Approximately 3.17 days)
registries:
  blocked: null
  insecure: null
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - quay.io
  - docker.io
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 3
  GraphDriverName: overlay
  GraphOptions: {}
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 18
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.6.4-2.module+el8.1.1+5363+bf8ff1af.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):
rhel8.1 laptop

[rhatdan]

@vrothberg PTAL

[ikke-t]

It's related to this ansible role. I changed %t to %T to make it work somehow. https://github.com/ikke-t/podman-container-systemd

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[vrothberg]

Thanks for opening the issue and apologies for my silence! I am currently focusing on Podman v2 but will get back to systemd stuff in May.

[cajus]

Same with the latest version. If I replace %t with i.e. /run/users/1234 podman is segfaulting while it works fine if started as the same user by systemctl --user.

[andrewgdunn]

I'd also suggest, even if it seems weird, that there is a ExecStopPost=/usr/bin/rm -f %T/%N.cid. It seems as if enumerating to systemd via PIDFile automatically cleans up that .pid, but the .cid is left around.

edit: apparently conmon does this, not systemd.

I see that by default the template attempts to clear it upon ExecStartPre, but it feels somewhat disingenuous that systemd would clear the .pid at the end of a unit run and we'd leave that .cid around for the next restart of the unit.

More discussion on this in #6157.

[vrothberg]

To clarify the .pid removal. It's not systemd removing but Podman+conmon, so the inconsistency is on our side.

While it should not cause any harm, I find the inconsistency at least confusing. Unfortunately, I don't think we can change the behavior without (potentially) breaking users.

[vrothberg]

Note to self: make sure that the file removal is properly explained in the man pages.

[andrewgdunn]

To clarify the .pid removal. It's not systemd removing but Podman+conmon, so the inconsistency is on our side.

While it should not cause any harm, I find the inconsistency at least confusing. Unfortunately, I don't think we can change the behavior without (potentially) breaking users.

I'd venture that one of the big strengths of podman is rootless/daemon-less with combination to your existing init system. So any amount of consistency that can be achieved would be welcome as quality of life when combining podman+systemd . Documenting, while helpful, is going to only help comprehensive RTFM momements, when really this idiosyncrasy is likely to catch and confuse people for some time before they discover it documented as an idiosyncrasy in the doc.

[vrothberg]

Thanks for the feedback! I cannot disagree but I need to keep backwards compat in the back of my head.

@mheon @baude @rhatdan , what are your thoughts on the issue? In summary, the --conmon-pidfile is being removed with the container while the --cidfile is not.

[mheon]

--cidfile is a deliberate Docker compatability measure, AFAIK - we made the lifecycle match what Docker expected. They're also entirely outside of Libpod - we don't know that they exist, so once they're made we never remove them. We had no such constraints with pidfiles - they're fully managed by Libpod, and required to interact with init systems (and, since a while ago, used for every container). I think we also need to remove them to ensure that Conmon starts cleanly (it blows up when they still exist).