Podman missinterprets %h symbol in the bind volume source when container created/started from a unit (BTRFS file system)

[PavelSosin-320]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When container that assumes bind volume mounting is created or run from the generated by podman generate systemd command the source path can be expressed by the absolute path or via % builtin symbols, %h for example. All % builtin "path" symbols like %t and %h are expanded into an absolute path, i.e. must be accepted by podman as a valid bind volume source. All subdirectories related to the %h, %v, etc. should be accepted. I hope, local driver and fuse.mount support this scenario.

Steps to reproduce the issue:

1. Create a container from the docker.io/theiaide/theis image with bind volume as /home/user/project:/home/project
2. Generate Systemd unit from the container.
3. Put the unit into ~/.config/systemd/user and enable it
4. Create a "project workspace directory" ~/project
5. Modify unit .service file: add Condition=PathExists=%h/project (to be sure that bind volume source exists and its full path is correctly expanded) and the bind volume source expressed as -v %h/project:/home/project
6. Start the unit using systemctl start
   Describe the results you received:
   "Invalid reference" error message
   Describe the results you expected:
   It must work because systemd unit must be shareable between users

Additional information you deem important (e.g. issue happens only occasionally):
It works OK in WSL because the workspace is located in the host filesystem
Output of podman version:
Podman Version 3.3.1
API Version: 3.3.1

**Output of `podman info --debug`:**

(paste your output here)

**Package info (e.g. output of `rpm -q podman` or `apt list podman`):**

(paste your output here)

**Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)**


Yes

**Additional environment details (AWS, VirtualBox, physical, etc.):**
Fedora 4 Workstation on the laptop
security-opt lable=disable has been added
Standalone podman run  -v /home/pavelsosin/project:/home/project [TheiaIDE](https://hub.docker.com/r/theiaide/theia) is OK.

[PavelSosin-320]

Sorry! It may be unrelated because it happens regardless of the volume type. Simply, podman run and Podman container create return err code 125 when executed by rootless user, i.e. systemd unit on my Fedora WS in the user session, like in ##2197. Then, cid file is not created , ............. stop and rm don't work. It never happens in when podman runs in the terminal, only when from the unit using systemctl --user start, daemon-reexec, etc. Hint: Podman REST API service is running in the background.

[rhatdan]

@vrothberg PTAL

[PavelSosin-320]

@giuseppe I think that it is exactly the ##2172 because the main difference between working case onWSL distro i.e. ext4 filesystem and failing Fedora34 WS i.e. btrfs filesystem. But I expected that in 3.1 the configuration file should be corrected. Please, look at attachement.
podman-info.log
I'm using Fedora WS GUI session, rootless user. Which conf file defines the overlay driver behavior in this case?

[giuseppe]

could you share the systemd service file generated by Podman?

[PavelSosin-320]

The recent working in rootless WSL version for Theia IDE with Podman backend enabled, slightly modified to reference project workspace in the user's home and starting Chrome in the kiosk mode. Can be used as a test scenario.I use a pre-pooled Image ID to avoid pull delay and .interaction with the user that is impossible inside systemd units. Everything works in WSL!!!

…

On Tue, Sep 14, 2021 at 11:34 AM Giuseppe Scrivano ***@***.***> wrote: could you share the systemd service file generated by Podman? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#11547 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM7YVKFB3JL5HUPYRXNQW5DUB4CK3ANCNFSM5D5MP75A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[PavelSosin-320]

The root cause is certainly related to the file system type: Fedora34 uses BTRFS by the default mounted at /home. If user data is located at /home too but not necessary on the btrfs subvolume the cross-fs mount will never work. The generated Unit checks that the Podman tree and container storage exist but it doesn't check or trigger mounting of subvolume or validate that user's data are in the same FS as container storage. The same problem appears in the anonymous volume scenario - nothing enforces that everything is on the same volume. storage.conf allows everything. WSL VM with its single ext4 filesystem works perfectly: no btrfs and Host folders are mounted using MS-specific mechanism. How bind mount should work on BTRFS and subvolumes???

[vrothberg]

could you share the systemd service file generated by Podman?

@PavelSosin-320, please share the systemd service file.

[PavelSosin-320]

From the Podman on WSL instance with some comments and TODOs
PodmanUnit.txt

[vrothberg]

Thanks, can you also the contents of run-r8df511c2cb034c33a1ec70d63b670529.service?

[PavelSosin-320]

Sorry for the long delay due to the holidays. I tried to run theia image via systemd-run as a transient unit and the result is:
[pavelsosin@Dell user]$ journalctl --unit run-u1276.service
-- Journal begins at Mon 2021-09-13 16:33:09 IDT, ends at Thu 2021-09-30 17:28:05 IDT. --
Sep 30 17:26:17 Dell systemd[1]: Started /usr/bin/podman run -d -p 3000:3000 -v /home/pavelsosin/project:/home/project docker.io/theiaide/theia.
Sep 30 17:26:17 Dell podman[384240]: 2021-09-30 17:26:17.896394707 +0300 IDT m=+0.044832762 image pull docker.io/theiaide/theia
Sep 30 17:26:18 Dell podman[384240]: 2021-09-30 17:26:18.338036042 +0300 IDT m=+0.486474147 container create ea48aca6642ccc7e87476b7611950d711efb28569efd8823a749e13a1d2f0c0e (image=docker.io/theiaide/theia:latest, name=determined_hertz)
Sep 30 17:26:19 Dell podman[384240]: 2021-09-30 17:26:19.592840034 +0300 IDT m=+1.741278171 container init ea48aca6642ccc7e87476b7611950d711efb28569efd8823a749e13a1d2f0c0e (image=docker.io/theiaide/theia:latest, name=determined_hertz)
Sep 30 17:26:19 Dell podman[384240]: 2021-09-30 17:26:19.937165559 +0300 IDT m=+2.085603734 container start ea48aca6642ccc7e87476b7611950d711efb28569efd8823a749e13a1d2f0c0e (image=docker.io/theiaide/theia:latest, name=determined_hertz)
Sep 30 17:26:19 Dell podman[384240]: ea48aca6642ccc7e87476b7611950d711efb28569efd8823a749e13a1d2f0c0e
Sep 30 17:26:20 Dell podman[384445]: 2021-09-30 17:26:20.229830119 +0300 IDT m=+0.072758624 container died ea48aca6642ccc7e87476b7611950d711efb28569efd8823a749e13a1d2f0c0e (image=docker.io/theiaide/theia:latest, name=determined_hertz)
Sep 30 17:26:22 Dell podman[384445]: 2021-09-30 17:26:22.572881081 +0300 IDT m=+2.415809589 container cleanup ea48aca6642ccc7e87476b7611950d711efb28569efd8823a749e13a1d2f0c0e (image=docker.io/theiaide/theia:latest, name=determined_hertz)
Sep 30 17:26:22 Dell systemd[1]: run-u1276.service: Deactivated successfully.
Sep 30 17:26:22 Dell systemd[1]: run-u1276.service: Consumed 1.134s CPU time
In other words, bind mount succeeded. This is good news. The bad news that possibly, either podman or storage driver search something in the unit environment that doesn't exist -"empty".
I also checked that no hidden dependency from systemd fs* target effects the bind because AssertMount /home is passed. BTRFS /home userspace FS is mounted, i.e. /home works as a BTRFS subvolume.

[PavelSosin-320]

Since all possible scenarios that I tested including Image volumes and anonymous volumes worked OK I suppose the root cause is that Podman parses -v option value exactly as described in the Documentation:
Any source that does not begin with a . or / will be treated as the name of a named volume. If a volume with that name does not exist, it will be created .. in a blind way. Systemd units placeholders that start from % don't conform with this pattern and are not substituted as Env variables. Unit, unlike systemd-run, ignores session environment - it runs without a shell. For example the equivalent of $(PWD) is WorkingDirectory=.
And indeed it worked for me fine:

1. I added WorkingDirectory=%h directive
2. And addressed the source directory of the bind volume via (.).. I suppose, the same pattern to designate "project folder" with all necessary files that can be passed to the container started as a service in combination with --new.

[PavelSosin-320]

@vwbusguy Unfortunately, now I can say definitely that the issue is related to the btrfs filesystem because exactly the same syntax works correctly on the ext4 filesystem of the WSL-Fedora VM instance but doesn't work in the Fedora 34 Desktop with its default btrfs FS. Since BTRFS itself is a userspace FS and has its own Kernel module and mount utilities it can conflict with the FUSE-mount. In the Docker's documentation, this issue is addressed explicitly in Docker doc BTRFS storage driver. Although Podman info on Fedora reports that the backing FS is btrfs all other configurations look like the same as on ext4 FS.
Maybe it is the same as ##4764 but in the case of creation container from unit generated with the --new option only "invalid reference message" bubbles on the surface, produces err 125 and stops unit execution.

[PavelSosin-320]

After eliminating BTRF-related issues via Podman with BTRFS I found very simple thing: When Podman tries to create a container from the systemd-unit that ran by rootless user it can't find storage configuration and it causes the "invalid reference" error. The testing using systemd-run results in
Oct 12 12:47:23 Dell systemd[1]: Started /usr/bin/podman container create --conmon-pidfile %t/container-theiaUnit.pid --cidfile %t/container-theiaUnit.ctr-id cgroups=no-conmon --replace -p >
Oct 12 12:47:23 Dell podman[21664]: time="2021-10-12T12:47:23+03:00" level=warning msg="Storage configuration is unset - using hardcoded default graph root "/var/lib/containers/storage""
Oct 12 12:47:23 Dell podman[21664]: time="2021-10-12T12:47:23+03:00" level=warning msg="Storage configuration is unset - using hardcoded default graph root "/var/lib/containers/storage""
Oct 12 12:47:23 Dell podman[21664]: time="2021-10-12T12:47:23+03:00" level=warning msg="Storage configuration is unset - using hardcoded default graph root "/var/lib/containers/storage""
Oct 12 12:47:23 Dell podman[21664]: Error: invalid reference format
Oct 12 12:47:23 Dell systemd[1]: run-u236.service: Main process exited, code=exited, status=125/n/a
Oct 12 12:47:23 Dell systemd[1]: run-u236.service: Failed with result 'exit-code'.
Even after importing HOME and XDG_RUNTIME_DIR into User's systemd manager environment the issue is not resolved.
Intesive usage environment variables in the configuration, like in sthe storage.conf is not safe because Session environment doesn't exist in the systemd runtime - nothing is inherited automatically.