Mounts stop receiving inotify change events from Windows host

[8ctopus]

[x] I have tried with the latest version of my channel (Stable or Edge): tested under edge 2.3.6 47622 and stable 2.3.0.4 46911
[x] I have uploaded Diagnostics
Diagnostics ID: F6FACDFC-D90A-40E8-8A1A-FC5A6D749036/20200915092839

Windows Version: Microsoft Windows 10 Pro 2004 19041.508
Docker Desktop Version: edge 2.3.6 47622 and stable 2.3.0.4 46911
Are you running inside a virtualized Windows e.g. on a cloud server or on a mac VM: No

Expected behavior
File change events from editing in the host should propagate to the container to enable tools like modd, nodemon or webpack to re-compile files in-container.

Actual behavior
Event triggered when changes happen within the container but not when on host. The problem existed in the past #5701 but seems to have reappeared in the latest Docker for Windows versions.

Steps to reproduce the behavior

docker-compose.yml

version: '3.7'

services:
  test:
    image: alpine:latest
    container_name: test
    volumes:
      - ./tmp:/tmp
    command: sh -c "sleep 1000"

Using git-bash shell on Windows

$ docker-compose up --detach
Creating network "inotify-wait_default" with the default driver
Creating test ... done

$ winpty docker exec -it test sh

/ # apk add inotify-tools
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
(1/1) Installing inotify-tools (3.20.2.2-r0)
Executing busybox-1.31.1-r16.trigger
OK: 6 MiB in 15 packages

/ # inotifywait --event modify,create,delete --recursive /tmp/ &
/ # Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.

# event properly captured when file is created inside docker container
/ # touch /tmp/test
/tmp/ CREATE test
[1]+  Done                       inotifywait --event modify,create,delete --recursive /tmp/

/ # inotifywait --event modify,create,delete --recursive /tmp/ &
/ # Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.

/ # exit

# event not captured when file is created on host
$ touch tmp/test2

[stephen-turner]

Thanks for the report, @8ctopus. Are you running Docker in WSL 2, but keeping your files in the Windows file system instead of the Linux file system? If so, then this is expected (and documented, and out of our control because we don't manage the file sharing in that scenario).

[8ctopus]

@stephen-turner Thank you so much for your reply, I've been trying to figure out where the problem comes from for 4-5 hours. I've just disabled WSL 2 and everything works again as it should. It's unfortunate that the better WSL 2 breaks that functionality and I had to revert to Hyper-V.

[stephen-turner]

The alternative is to use WSL 2 but keep your files on the Linux side. That's what we recommend for best performance.

[8ctopus]

@stephen-turner Thank you again, I'll try that. I'm completely new to WSL 2 so don't have any experience with it yet.

[YouveGotMeowxy]

The alternative is to use WSL 2 but keep your files on the Linux side. That's what we recommend for best performance.

@stephen-turner What exactly do you mean by "keep files on the linux side?" (I'm to much of a Linux person). To physically move the files into the wsl2 Ubuntu VM?

I thought that if you use for example -v /mnt/d/some/folder:/folder that you're using the Linux file system, even if the files are hosted in Windows (in the case of previously using the Hyper-V version of docker, and not physically moving any files)?

I'm having issues getting my "old" Hyper-V mounts to work as well after upgrading to wsl2 and I'm not sure why? When looking at the paths it all seems OK:

That's from a compose that looks like this:

  dnscrypt-proxy:
    hostname: dnscrypt-proxy
    image: klutchell/dnscrypt-proxy:latest
    environment:
      TZ: ${TZ}
    volumes:
      - '/mnt/f/Sites/DNSCrypt/dnscrypt-proxy.toml:/config/dnscrypt-proxy.toml'
    ports:
      - '5252:5053/udp'
    networks:
      - ocdns
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.labels.MainDaemon == true
      restart_policy:
        condition: any

[stephen-turner]

Yes keep your files in the WSL 2 filesystem. Don't share them across the VM boundary because it's unperformant and you lose inotify.

[YouveGotMeowxy]

@stephen-turner I have no problem doing that in cases where the files may be small, such as simple config files, but what about in cases like for example, Nextcloud, where the binded mounts can be terabytes? (I use an external 2tb drive bind mounted to that container).

I don't understand why using /mnt/ isn't working at all? Do bind mounted paths still need to be "shared" in linux, the way they did using the Docker GUI for Hyper-V? if so, how does one "share" paths in Linux?

Update:
I copied the file into the Ubuntu WSL vm at: \\wsl$\Ubuntu\home\me\docker\DNSCrypt\dnscrypt-proxy.toml and it still won't mount when I use either:

 volumes:
  - ~/docker/DNSCrypt/dnscrypt-proxy.toml:/config/dnscrypt-proxy.toml

or

volumes:
  - ${HOME}/docker/DNSCrypt/dnscrypt-proxy.toml:/config/dnscrypt-proxy.toml

it shows the error:

type "bind": bind source path does not exist: /docker/DNSCrypt/dnscrypt-proxy.toml"

[stephen-turner]

1. Sure, I get that some files are difficult to move out of the Windows filesystem. But unfortunately we do not control the file sharing across the WSL boundary, it's wholly managed by Windows, so we can't do anything to optimise it. Your alternative is to stick to Hyper-V instead of WSL 2.

2. No, you do not need to share files in the UI like you do in Hyper-V. I'm not sure why your example didn't work, but I did find this on stack overflow: https://stackoverflow.com/questions/48971065/getting-invalid-mount-config-for-type-bind-bind-source-path-does-not-exist-in. Does it work if you do a simple docker run -v instead of using a compose file?

[YouveGotMeowxy]

Thanks @stephen-turner

1. Sure, I get that some files are difficult to move out of the Windows filesystem. But unfortunately we do not control the file sharing across the WSL boundary, it's wholly managed by Windows, so we can't do anything to optimise it. Your alternative is to stick to Hyper-V instead of WSL 2.

So in order to use Nextcloud, or other containers that need lots of storage space, like torrent clients, NZB clients, etc. we have to stick with the Hyper-V version and lose out on all the WSL2 benefits? :/

2. Does it work if you do a simple docker run -v instead of using a compose file?

Hmmm, I think you're onto something, but I don't have enough experience to know what, lol.

When I do:

docker run -v ~/.docker/configs/DNSCrypt/dnscrypt-proxy.toml:/config/dnscrypt-proxy.toml -h dnscrypt-proxy --network dns -p 5252:5053/udp -e TZ=America/Chicago klutchell/dnscrypt-proxy:latest

It works, but using this in compose (I use Portainer), I get the error shown above:

  dnscrypt-proxy:
    hostname: dnscrypt-proxy
    image: klutchell/dnscrypt-proxy:latest
    environment:
      TZ: 'America/Chicago'
    volumes:
      - ~/.docker/configs/DNSCrypt/dnscrypt-proxy.toml:/config/dnscrypt-proxy.toml
    ports:
      - '5252:5053/udp'
    networks:
      - ocdns
    deploy:
      mode: global
      restart_policy:
        condition: any

Any ideas? I'm left scratching my head here, lol.

[stephen-turner]

1. Well, I guess so. I mean, the only options are (1) Hyper-V (2) WSL 2 but files in Windows, which has performance problems (3) WSL 2 with files in WSL 2.

2. I'm not sure either, but I notice something else that might be relevant. The error message referred to /docker/DNSCrypt/dnscrypt-proxy.toml as the file that doesn't exist. Is $HOME perhaps blank in this context?

[YouveGotMeowxy]

@stephen-turner Question: when using WSL2 but keeping the files in Windows. is the performance worse than using Hyper-V with the files in Windows?

[stephen-turner]

In our experiments it is worse, yes. Exactly how much worse depends on the nature of your workload: it can be anything from a few percent to multiples. The biggest penalty is when you have frequent accesses to lots of small files.

Living wholly within the WSL 2 world is always fastest, if you can achieve that.

[airtonix]

btw inotify events have never worked for files stored on filesystems that come from a VMs "shared folder strategy" (it's essentially coming into your vm via Samba).

You've always had to enable some kind of polling solution like CHOKIDAR_ENABLE_POLLING

Does anyone have a release version where this was working? It worked for us for a long while. You could save me sometime :).

Turns out I was able to answer my own question: 2.2.0.5 (in case someone was interested)

[8ctopus]

I'm attaching the WSL2 issue to the thread [WSL2] File changes made by Windows apps on Windows filesystem don't trigger notifications for Linux apps