Non-deterministic Windows CI error: `rmdir: Directory not empty`

[DilumAluthge]

For example, see this CI log.

The error usually looks something like this:

Error During Test at D:\a\PackageCompiler.jl\PackageCompiler.jl\test\runtests.jl:96
  Got exception outside of a @test
  SystemError (with C:\Users\RUNNER~1\AppData\Local\Temp\jl_kFRDyU\artifacts\fdff308295487f361ef6e8dc2d27f5abe8a6eee9\mingw64\libexec\gcc\x86_64-w64-mingw32\8.1.0): rmdir: Directory not empty
  Stacktrace:

It seems to be non-deterministic.

Retrying the failed job usually makes the error go away.