Quadlet generator should avoid using `%t` if `User=` is set

[Valloric]

Feature request description

The Podman Quadlet unit generator uses --cidfile=%t/%N.cid in the generated ExecStart, ExecStop, ExecStopPost commands.

The %t specifier is the issue. When a system service uses User=, %t resolves to /run and the unprivileged user doesn't have permissions to create a file there.

Suggest potential solution

A possible solution would be for the Podman unit generator to notice that the .container file has User=foo in it and then use something other than %t. Since XDG_RUNTIME_DIR has to be set for the user anyway, perhaps using that value would work better.

At worst, placing this one file in a common temporary directory with foo as owner and 0600 mode (user-only r/w) would suffice. root can read the file just fine, so the system service manager won't care.

I understand there are issues with these solutions. Please don't let "perfect be the enemy of good". User= support is important and a silly thing like the %t specifier for the cid file directory should really not be a blocker.

Have you considered any alternatives?

Various comments on Podman's (and Systemd's) issue tracker say some version of the following:

If you have to start a user session anyway to be able to use User=, why not just run the service with systemctl --user start and be done with it? Just use a proper user service, stop trying to run it as a system service.

There's a bunch of reasons to use a system service, not the least of which is having all system services properly centralized in the system service manager (and all the UX benefits thereof). But these "creature comforts" don't seem to be convincing enough to various people, and I can somewhat understand why.

But then we get to the following: all sorts of useful Systemd features (like sandboxing) do not work with user service managers:

Also note that some sandboxing functionality is generally not available in user services (i.e. services run by the per-user service manager). [systemd.exec manpage]

Not supporting User= well reduces security.

Additional context

@vrothberg mentioned this way back in 2022 in a comment here: #20573 (comment)

Much is different in Systemd and Podman since then, and if I'm not mistaken, this usage of %t is the only blocker for fairly functional usage of User= in quadlet files.

@felixc recently posted a comprehensive report about his experience (thank you!) with using User= with Podman on Debian Trixie. I am running a similar setup to him and can confirm that things pretty much work fine after applying workarounds for Podman issues.

The necessary workarounds:

• Can't use Quadlets because of %t.
• Must work around Podman bug that reads $HOME as a fallback to $XDG_CONFIG_HOME (podman run fails when HOME set to invalid path and unprivileged system user specified as systemd User= #26667)
• (I will file bugs for other issues as I discover them)

The admin must also tell Systemd to start the user session for the user specified in User= with sudo loginctl enable-linger myuser (applies after next boot), but that's fair IMO.

Thank you for working on Podman! It's much nicer than Docker. I love you Podman folks, you are the reason I don't have to run a huge VC-backed codebase as a privileged daemon. 🥲

cc @Luap99 @rhatdan

No AI was used to write any part of this issue report. Every word was written by a human.

[Luap99]

Usage of cidfiles were already removed in 9b2fb40 but there are still a two places where %t is used. For the infra conmon pidfile when using a pod unit and RequiresMountsFor=%t/containers
The fist one is likely hard to get rid of and the second one the second one I am not sure about the history of why we have it, it should be relatively simple to just not set it but it is likely there for a reason.

But I like to repeat myself running the services with User= is not something we support, we even generate warnings now #26552

But then we get to the following: all sorts of useful Systemd features (like sandboxing) do not work with user service managers:

Also note that some sandboxing functionality is generally not available in user services (i.e. services run by the per-user service manager). [systemd.exec manpage]

Not supporting User= well reduces security.

Many of these sanboxing features will break rootless podman in subtle ways. Different user/mount namespaces between units will make it so containers cannot join another namespace for example and will mess with podmans mount tracking and such. I am not saying these can never be used but given most of these restrict some path all involve a mount namespace that will almost certainly cause hard to diagnose problems.

[Valloric]

Oh man, reading that comment was such a roller-coaster of emotions. 😄

Usage of cidfiles were already removed in 9b2fb40

YES, thank you!

but there are still a two places where %t is used.

Oh well, one step at a time...

running the services with User= is not something we support, we even generate warnings now #26552

Nooooo... 😢

Well, I guess the warning is more than fair. Actually it's a good idea; using User= with Podman is not a well-lit path.

running the services with User= is not something we support

I'd like clarification on this point so I can make informed decisions about any such usage in production. Right now, I'm just exploring rootless Podman with system users in my homelab, but my long-term goals for this feature are more ambitious. I have created separate sections for every question.

User-with-Quadlets support?

Does the warning about User= in Quadlets mean that the Podman team does not support User-with-Quadlets now—while desiring to support it in the future—OR does it mean "we have no plans to support User-with-Quadlets at all"?

User-with-plain-unit support?

Does "this is not a supported use-case" also apply to User-with-plain-unit? In other words, does the Podman team take a "we don't support this at all" position for using User= in systemd service units that run rootless Podman? I know it doesn't work great now, but that's why I'm trying to explore the limits of this feature and file bugs for stuff that doesn't work.

And who knows, if I find the time and Podman team wishes to see this feature improve, maybe send some PRs.

I'm asking because I'm confused by the latest messaging. @vrothberg said in 2023:

At the moment, it is unsupported but this issue is meant to find means where it can be supported.

Is this out-of-date? Is the team's position some version of "it doesn't work well now so we can't support it, but we'd sure like to see it improve and then possibly support it one day"?

User= support desirable or "meh"?

I guess what I'd like to know is am I adding any value by filing these issues or am I just barking at the moon and there's little-to-no appetite to see User= support (in Quadlets or plain units) improve? If so, please say so and I'll stop wasting everyone's time.

[Valloric]

(@Luap99 Apologies, I forgot to respond to this part in my previous comment.)

Many of these sanboxing features will break rootless podman in subtle ways. Different user/mount namespaces between units will make it so containers cannot join another namespace for example and will mess with podmans mount tracking and such. I am not saying these can never be used but given most of these restrict some path all involve a mount namespace that will almost certainly cause hard to diagnose problems.

That's entirely understandable, thank you for the heads-up. "Don't come crying back to us if Podman can't mount a volume if you took away its ability to mount anything" is fine by me. :)

I am willing to find the parts that are broken and file issues for stuff that I think shouldn't be broken¹ if that's desirable.

Footnotes

1. This means not filing an issue for "you sandboxed away Podman's ability to do X and now it can't do X." ↩

[Luap99]

running the services with User= is not something we support

I'd like clarification on this point so I can make informed decisions about any such usage in production. Right now, I'm just exploring rootless Podman with system users in my homelab, but my long-term goals for this feature are more ambitious. I have created separate sections for every question.

support is loose term, for starters we know it is broken in some ways as you encountered already. As such it cannot be supported or recommended at the moment. For me support means at least two things. 1) We have some kind of CI tests for the feature to ensure basic things work and 2) There is at least one maintainer (person) actually working on this, i.e. triaging bug reports about it and fixing them, etc...
I at least have no interest in working towards making User= viable, I don't think other maintainers do either but I won't speak for them so I rather defer to other to chime in.

Long term I don't know. It is not like I am against such thing on principle if someone figures out how to make all these things work without introducing a lot of complexity into quadlet/podman then I am certainly not going to object to patches.
While I do think running a full blown user session is the recommended method even then I don't like to tell other how to run things and I certainly get the appeal of running everything in the system session to manage everything as root easily.

For the quadlet via plain unit question I don't see a big difference in terms of support. Arguably by writing your own unit you can get rid of the %t specifier yourself so there is that but besides the actual podman sd_notify, cgroup management and so on still has to work. I think it does make sense for us to have quadlet provide "blessed" and supported (i.e. known to work and tested) systemd units by default and outside of that I would always say you are own your own.
If you write your own systemd unit similar to what quadlet produces then I am certainly not going to refuse to help as long as said modification are not breaking the service.

I guess what I'd like to know is am I adding any value by filing these issues or am I just barking at the moon and there's little-to-no appetite to see User= support (in Quadlets or plain units) improve? If so, please say so and I'll stop wasting everyone's time.

I have no issue with you filling these issues, it is a good thing at the very least to document what doesn't work right now. If you or somebody else likes to fix these things upstream I am fine with that as long as they do not cause regressions or break something else. But don't expect me or another maintainer to work on it.

[Valloric]

But don't expect me or another maintainer to work on it.

This is entirely fair; thank you for a direct answer.

So the gist I'm getting is "it's not a priority for the podman team, but we're not against issues being filed to record what doesn't work, but we're not going to work to implement them at this time."

Depending on time availability, I might send some PRs in the future (no promises).

Thank you for being open about accepting patches for it! I have spent decades maintaining popular OSS projects and I'm well aware that accepting patches still involves significant work (code review, communication overhead etc).

[CutterXYZ]

Another reason for running system containers is the ability set dependencies on other system services. There is currently no way AFAIK to start a user service after a system service.

[Luap99]

Another reason for running system containers is the ability set dependencies on other system services. There is currently no way AFAIK to start a user service after a system service.

Yes that is already causing a fair bit of issues for us as there is no proper way to wait for network-online.target because of this. #24796 systemd/systemd#3312

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[ikruglov]

I want to express +1 to the idea of system-level quadlets to be able to depend on system-level dependencies. In my case, a mount unit.