Issue: Excessive Creation of Temporary Directories in Root with Sysbox

[xabru]

Description:

We have observed that many temporary directories are being created in the root directory with names following the pattern .sysbox-sysfs-<number>. This is causing clutter and potential confusion.

System Environment:

• Sysbox version: v0.6.4
• Dockerfile Template: Systemd in Docker

Additional Information:

This issue might have arisen because I masked the following services using systemctl:

systemctl mask \
	systemd-udevd.service \
	systemd-udevd-kernel.socket \
	systemd-udevd-control.socket \
	systemd-modules-load.service \
	sys-kernel-config.mount \
	sys-kernel-debug.mount \
	sys-kernel-tracing.mount \
	e2scrub_all.timer \
	e2scrub_reap \
	accounts-daemon.service \
	rtkit-daemon.service \
	systemd-hostnamed.service

$_ ls -hasl /

└> ll /
total 284
drwxr-xr-x   1 root   root    4096 Aug 24 17:51 ./
drwxr-xr-x   1 root   root    4096 Aug 24 17:51 ../
lrwxrwxrwx   1 root   root       7 Apr 22 08:08 bin -> usr/bin/
drwxr-xr-x   2 root   root    4096 Mar 31 04:00 bin.usr-is-merged/
drwxr-xr-x   2 root   root    4096 Apr 22 08:08 boot/
drwxr-xr-x   6 root   root     440 Aug 24 17:50 dev/
-rwxr-xr-x   1 root   root       0 Aug 24 17:50 .dockerenv*
drwxr-xr-x   2 root   root    4096 Aug 23 07:48 dockerstartup/
drwxr-xr-x   1 root   root    4096 Aug 24 17:50 etc/
drwxr-xr-x   1 root   root    4096 Aug 23 07:40 home/
lrwxrwxrwx   1 root   root       7 Apr 22 08:08 lib -> usr/lib/
lrwxrwxrwx   1 root   root       9 Apr 22 08:08 lib64 -> usr/lib64/
drwxr-xr-x   2 root   root    4096 Apr  8 09:37 lib.usr-is-merged/
drwxr-xr-x   2 root   root    4096 Aug  1 06:59 media/
drwxr-xr-x   2 root   root    4096 Aug  1 06:59 mnt/
drwxr-xr-x   1 root   root    4096 Aug 24 17:50 opt/
dr-xr-xr-x 283 root   root       0 Aug 24 17:50 proc/
drwx------   1 root   root    4096 Aug 24 18:59 root/
drwxr-xr-x  20 root   root     580 Aug 24 17:51 run/
lrwxrwxrwx   1 root   root       8 Apr 22 08:08 sbin -> usr/sbin/
drwxr-xr-x   2 root   root    4096 Mar 31 04:00 sbin.usr-is-merged/
drwxr-xr-x   2 root   root    4096 Aug  1 06:59 srv/
dr-xr-xr-x  13 nobody nogroup    0 Aug 24 18:10 sys/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-1086082501/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-1098625909/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-1220785409/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-1227003664/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-1270347815/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-1402730274/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-1427450857/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-1431106688/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-148529927/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-149844544/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-1557262215/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-1591870079/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-1692417296/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-1863566724/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-1988865741/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-2052994440/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-2188883916/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-2412479115/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-269341428/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-270781546/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-2722782661/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-273937039/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-2892516956/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-2924257033/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-3224083108/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-3229380096/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-324496473/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-3591021414/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-3622867146/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-3633530951/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-3707117874/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-3742760121/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-3787667621/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-4095605731/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-4136466978/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-4175937595/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-586173192/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-628597558/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-714564999/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-735075657/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-740134491/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-863786270/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-867797088/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-927480688/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-981093558/
drwxrwxrwt   1 root   root    4096 Aug 24 18:59 tmp/
drwxr-xr-x   1 root   root    4096 Aug  1 06:59 usr/
drwxr-xr-x   1 root   root    4096 Aug 24 17:50 var/

[AidanAbd]

We have experiences this as well, and are currently working around it by manually cleaning up these empty directories.

[ctalledo]

Hi @xabru, @AidanAbd , thanks for reporting this.

Yes, I can confirm this is a bug. I have a fix in mind, will work on this ASAP. Sorry for the inconvenience.

[EddieX64]

Hello @ctalledo @rodnymolina

I have tested sysbox v0.6.5 on GKE 1.29 and still see .sysbox-sysfs directories, but now there are also .sysbox-procfs directories in /

When i execute docker run -it alpine:latest sh inside a sysbox pod, additionally i can see .sysbox-procfs directories in the / of the inner container. In some scenarios this directory causing permission denied error as it's owned by root:root. Previously I used image ghcr.io/nestybox/sysbox-deploy-k8s:v0.6.5-dev-0 and it was working fine, but for some reason got deleted after official release of v0.6.5 :(

Now my projects are broken, because i can't rollback to v0.6.4 since i need support of sysbox-fs emulation to allow writes to '/proc/sys/kernel/shm*' paths, and this feature was added in v0.6.5-dev-0, but now in official v0.6.5 these .sysbox-procfs and .sysbox-sysfs are causing .sysbox-procfs permission denied issues.
Could you please have a look?

[ctalledo]

Hi @EddieX64, apologies for the late reply.

Thanks for letting me know, it was my mistake (thought I had fixed this in v0.6.5 but I made a mistake in the fix and did not catch it somehow).

This PR will fix it once and for all: nestybox/sysbox-fs#100

We will generate a v0.6.6 release soon afterwards.

[aldcons]

Hi @EddieX64, apologies for the late reply.

Thanks for letting me know, it was my mistake (thought I had fixed this in v0.6.5 but I made a mistake in the fix and did not catch it somehow).

This PR will fix it once and for all: nestybox/sysbox-fs#100

We will generate a v0.6.6 release soon afterwards.

We are also experiencing the permissions problem with '/proc/sys/kernel/shm*' on certain builds.

Is there a rough estimate when release 0.6.6 will be out?

Or is there a Dev registry we can use in the meantime?

Thanks.

[aldcons]

Hi @EddieX64, apologies for the late reply.
Thanks for letting me know, it was my mistake (thought I had fixed this in v0.6.5 but I made a mistake in the fix and did not catch it somehow).
This PR will fix it once and for all: nestybox/sysbox-fs#100
We will generate a v0.6.6 release soon afterwards.

We are also experiencing the permissions problem with '/proc/sys/kernel/shm*' on certain builds.

Is there a rough estimate when release 0.6.6 will be out?

Or is there a Dev registry we can use in the meantime?

Thanks.

After all that, we seem to now be getting caught with this one - nestybox/sysbox-fs#101

Is that also in the next release @ctalledo ?

[ctalledo]

Hi @aldcons,

Is there a rough estimate when release 0.6.6 will be out?

We are targeting end of this week.

It will fix both nestybox/sysbox-fs#100 and nestybox/sysbox-fs#101. Plus some other fixes.

Or is there a Dev registry we can use in the meantime?

There is no dev registry, but you can always build the sysbox package from source by cloning the sysbox repo.

$ git clone --recurse-submodules git@github.com:nestybox/sysbox.git
$ cd sysbox
$ Edit the VERSION file (e.g., `0.6.6-dev`)
$ cd sysbox-pkgr
$ make sysbox-ce-repo <path-to-where-you-cloned-sysbox>
$ make sysbox-ce-deb

This will generate the Sysbox Debian package under directory sysbox-pkgr/deb/build/amd64/ubuntu-jammy/.

You can then install that on your host with sudo apt-get install .... But first uninstall any current sysbox installation with sudo apt-get purge sysbox-ce.

If all is rignt, the systemctl status sysbox should show Sysbox has the version you put in the VERSION file in the steps above.

[chloe6888]

Hi @ctalledo
After upgrading to Sysbox v0.6.5, we have also encountered the same issue. Could you please provide an updated timeline or any news regarding the release of v0.6.6?

[ctalledo]

Hi @chloe6888 , Sysbox v0.6.6 is now released, please give it a try and let me know if the problem persists (should be resolved).