File system change events (inotify) support for mounted directories

[charleskorn]

Description

From abiosoft/colima#151:

We're using Colima to run Docker in a VM on our macOS laptops. Colima uses Lima under the hood to run this VM.

Are there any plans to support file system change events for mounted directories through inotify?

This is a key feature for us as we use containers in scenarios like auto-reloading dev servers and re-running unit tests on changes.

If there are no plans to support it at the moment, what kind of effort would be involved in adding this? Depending on the effort required, we might be open to putting together a PR to add support for this.

[jandubois]

what kind of effort would be involved in adding this

As I already commented in the colima issue, I'm not sure this is a possibility at all (generating inotify events). Even if you capture them on the host (fsevents), I think you would need to implement a custom notification mechanism in the guest and can't expect to use inotify, which is a kernel mechanism.

[AkihiroSuda]

In far future we may support inotify with virtiofs (not virtio-9p-pci).
https://lwn.net/Articles/874000/

[afbjorklund]

There were some earlier series, about the various limits of inotify.
https://lwn.net/Articles/605128/

Inotify reports only events that a user-space program triggers through the filesystem API. This constitutes a fairly serious limitation of the API. For example, it means that inotify does not inform us of events on monitored objects via a remote filesystem (e.g. NFS) operation.

Previously we "solved" it by keeping the files locally on the VM.

[atipugin]

Looks like virtiofs is already supported (although marked as experimental), but I still can't get FS events working. Tools like entr (which relies on inotify and kqueue) and webpack (in watch mode) don't see file changes. Interesting that FS notifications work fine with Docker Desktop (with same settings - VZ + virtiofs).

[lobshunter]

Looks like Apple's virtio-fs document doesn't mention anything about inotify. Inotify in docker desktop container does work(although it would drop DELETE event, and inject additional WRITE event). docker.log shows messages like level=info msg="injected MODIFY event (ts: 1677314215061760000) on /host_mnt/private/xxx, seems like it's doing some kind of inotify event injection 🤔.

[abiosoft]

docker.log shows messages like level=info msg="injected MODIFY event (ts: 1677314215061760000) on /host_mnt/private/xxx

That shows that Docker is manually handling the notify events i.e. the feature is missing in virtiofs.

[lobshunter]

i.e. the feature is missing in virtiofs.

Looks like so. I just tried qemu-virtiofsd on a Linux host, host to guest inotify doesn't work either(so it's not Apple's issue). There was some work trying to add inotify support for FUSE, but seems like it wasn't merged in mainline.

[lobshunter]

I bet docker desktop watches files in host, and forwards events into guest(perhaps by invoking something like chmod/write).
Start a docker destop container with a volume, then execute lskq(1) in host, it would show docker is watching files.

It might not be a great solution to do the same in lima.

[henrik242]

I'm trying to get node/next.js development working with colima, and next.js doesn't pick up changes I do on the host. However, if I touch the changed file from within the container, next.js picks up the change and my web page refreshes.

So, as a (silly) workaround, maybe one can use a script or daemon in the container to continuously scan the files and touch any changed files?

[lobshunter]

However, if I touch the changed file from within the container, next.js picks up the change and my web page refreshes.

Yes, guest to guest inotify works as expected, but host to guest inotify doesn't(most likely because FUSE protocol doesn't support it).

maybe one can use a script or daemon in the container to continuously scan the files and touch any changed files?

As a personal workaround, that would work(just less performant, performance degrades linearly with the number of files being added).