Issue building packages on windows: rmdir, Directory not empty

[hhugo]

I'm seeing issues on windows using dune 3.9.1.

I've witnessed this behavior in jsoo and zarith CI so far.

• https://github.com/ocaml/Zarith/actions/runs/5589049596/jobs/10216750778?pr=143
• https://github.com/ocsigen/js_of_ocaml/actions/runs/5588550201/jobs/10215512355?pr=1490

The error looks like this:

#=== ERROR while compiling sexplib0.v0.16.0 ===================================#
# context     2.0.10 | win32/x86_64 | ocaml.5.0.0 ocaml-option-mingw.1 | git+https://github.com/ocaml/opam-repository.git
# path        D:/a/js_of_ocaml/js_of_ocaml/_opam/.opam-switch/build/sexplib0.v0.16.0
# command     D:\a\js_of_ocaml\js_of_ocaml\_opam\bin\dune.exe build -p sexplib0 -j 2
# exit-code   1
# env-file    D:/.opam/log/sexplib0-5660-51cb9c.env
# output-file D:/.opam/log/sexplib0-5660-51cb9c.out
### output ###
# Error:
# rmdir(C:\Users\runneradmin\AppData\Local\Microsoft\Windows\INetCache\dune\db\temp\dune_80f647_artifacts): Directory not empty
# -> required by
#    _build/default/src/.sexplib0.objs/sexplib0__Sexp_conv_grammar.impl.d
# -> required by
#    _build/default/src/.sexplib0.objs/sexplib0__Sexp_conv_grammar.impl.all-deps
# -> required by
#    _build/default/src/.sexplib0.objs/byte/sexplib0__Sexp_conv_grammar.cmt
# -> required by
#    _build/install/default/lib/sexplib0/sexplib0__Sexp_conv_grammar.cmt
# -> required by _build/default/sexplib0.install
# -> required by alias install

Specifications

• Version of dune (output of dune --version): 3.9.1
• Version of ocaml (output of ocamlc --version): 4.14 and 5.0
• Operating system (distribution and version): windows

[hhugo]

This issue doesn't seem to exist in dune 3.8.3

[emillon]

#7371 looks like a promising candidate (given the different semantics of unlinking on Windows), but it seems to be disabled by default.

[emillon]

can you try setting the following environment variable? (in case it's the background sandboxes)

DUNE_CONFIG__BACKGROUND_SANDBOXES=disabled

[hhugo]

Setting the env variable doesn't fix the issue.
see https://github.com/ocsigen/js_of_ocaml/actions/runs/5600013207/jobs/10241780134?pr=1490

[emillon]

Ok, thanks! I'll try to reproduce that locally.

[hhugo]

issue also exists with dune 3.9.0

[emillon]

I've not managed to repro locally yet with dkml but I'll try with a mingw environment.

[emillon]

OK, I managed to reproduce this locally. It's an issue with the shared cache. You can disable it on Windows as a workaround.

[emillon]

I'm struggling to bisect this precisely, but I think there's a race condition in Fpath.rm_rf (used here):

dune/otherlibs/stdune/src/fpath.ml

Lines 135 to 138 in 92d8848

	match clear_dir path with
	| Directory_does_not_exist -> ()
	| Cleared -> (
	match Unix.rmdir path with

If something has created a file in the directory between clear_dir (which reads the directory) and rmdir, there is going to be an error in rmdir. But that would be an issue for all OSes though.

[nojb]

If you can reproduce easily, perhaps it is a good idea to list the contents of the directory in question before calling `rmdir`; it may help figuring out the root cause of the issue. Le ven. 21 juil. 2023 à 15:49, Etienne Millon ***@***.***> a écrit :

…

If something has created a file in the directory between clear_dir (which reads the directory) and rmdir, there is going to be an error in rmdir. But that would be an issue for all OSes though.

[emillon]

Good idea. These are ocaml source files. For example when running make dev, they are bits of dune source code. They are marked readonly.
Either we're creating files we can't remove on Windows (unlikely since I don't think we changed anything related to that recently), or there's a race going on. I verified that disabling background sandboxes does not change this, so I'm still confused. I'll continue investigating.