using mount program /usr/local/bin/fuse-overlayfs: fuse: device not found while running on Kubernetes

[DanielVenturini]

Hey, nice project, I appreciate podman-static.

Description

I'm trying to build an image inside a Kubernetes Pod on a Jenkins Pipeline.

My Pod yaml definition is the following:

apiVersion: v1
kind: Pod
spec:
  containers:
  - name: podman-builder
    image: mgoltzsche/podman:5.6.1
    command:
    - sleep
    args:
    - infinity
    env:
      - name: "REPO"
        value: "harbor.localhost.com"
      - name: "ORG"
        value: "gitops-project"
      - name: "APP"
        value: "restapi-flask"
      - name: "TAG"
        value: "latest"

and on container I simple run the following:

podman build --tag ${REPO}/${ORG}/${APP}:${TAG} .

but I get the following issue:

+ podman build --tag harbor.localhost.com/gitops-project/restapi-flask:latest .
STEP 1/8: FROM python:3.9.12-alpine3.15
Resolving "python" using unqualified-search registries (/etc/containers/registries.conf)
Trying to pull docker.io/library/python:3.9.12-alpine3.15...
Getting image source signatures
Copying blob sha256:7b5fd79052445f1c2eefc0f34823883f539778a286721816037c4ab44749a4e2
Copying blob sha256:a1ef3e6b7a02c0ce1d7d3ee96f273c2c5074b3f127082b4cd3f629f82097e1a8
Copying blob sha256:df9b9388f04ad6279a7410b85cedfdcb2208c0a003da7ab5613af71079148139
Copying blob sha256:365abad9bce0fa20bba4e2209f134a68e24436596ff74923130ed5bea00ba7e1
Copying blob sha256:92eb277ed3c67180e9693a2bc59d33bb9519da9b44c6b2a49ec630cde33b5cff
Copying config sha256:9f14ea10b14641c5e7e7b4ebadb6ccb60e586d1cae4b9f506fdb47ba85f70af2
Writing manifest to image destination
Error: mounting new container: mounting build container "a1fe5e8ab875abe5318884d9d0cd0f544efdc94ad8bc47a4c011dbb303384b7e": creating overlay mount to /var/lib/containers/storage/overlay/cd879eb2efe0d730144e8d7886a4ea0ff50834544eb871aeb0840da2254b633f/merged, mount_data="lowerdir=/var/lib/containers/storage/overlay/l/BEA7TKGE6FY7B7G2RAK2PR4OI3:/var/lib/containers/storage/overlay/l/7RF4VQ56JF3LTSFEQOPNJENR47:/var/lib/containers/storage/overlay/l/QWK4TGCVB2DPZQY52UWDSF5VWK:/var/lib/containers/storage/overlay/l/LVEIK3OLNPLWLVSCLLWTJPDQV2:/var/lib/containers/storage/overlay/l/M6E4IMLBWKTOFTYNZCSE4NASTP,upperdir=/var/lib/containers/storage/overlay/cd879eb2efe0d730144e8d7886a4ea0ff50834544eb871aeb0840da2254b633f/diff,workdir=/var/lib/containers/storage/overlay/cd879eb2efe0d730144e8d7886a4ea0ff50834544eb871aeb0840da2254b633f/work,nodev,fsync=0,volatile": using mount program /usr/local/bin/fuse-overlayfs: fuse: device not found, try 'modprobe fuse' first
fuse-overlayfs: cannot mount: No such file or directory
: exit status 1

Solution (Workaround?)

I've found a related issue (containers/buildah#5456) which I noticed you already observed this, where I was able to get rid of this issue using the following command to use VFS driver instead of overlay in storage.conf:

sed -i 's/^driver = .*/driver = "vfs"/' /etc/containers/storage.conf

[mgoltzsche]

Hi @DanielVenturini,
I think you have to mount /dev/fuse into your podman-builder container in order be able to use the fuse-overlayfs storage driver.
Though, that's not the last issue you'll stumble over. The container actually needs to be privileged (and once that's the case you may not have to mount /dev/fuse explicitly).

Regarding the workaround: while the vfs storage driver doesn't require advanced file system features, it is super slow.