rm: move open DLLs to temp dir to allow dir to be deleted
#53456
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vtjnash
reviewed
Feb 24, 2024
base/file.jl
Outdated
| if err.code==Base.UV_EACCES && endswith(path, ".dll") | ||
| # Loaded DLLs cannot be deleted on Windows, even with posix delete mode | ||
| # but they can be moved. So move out to allow the dir to be deleted | ||
| # and DLL deleted via the temp dir cleanup on next reboot |
There was a problem hiding this comment.
FYI: tempdir is never cleaned up on Windows
IanButterworth
force-pushed
the
ib/win_dll_cleanup
branch
from
58202e5 to
c79f4b8
Compare
IanButterworth
force-pushed
the
ib/win_dll_cleanup
branch
from
82fb204 to
6b37c5a
Compare
staticfloat
reviewed
Feb 25, 2024
base/file.jl
Outdated
Comment on lines
284
to
285
| # TODO: Add a mechanism to delete these moved files after dlclose or process exit | ||
| dir = mkpath(joinpath(tempdir(), "julia_delayed_deletes")) |
There was a problem hiding this comment.
I think adding something to Pkg.gc() that checks for this folder and attempts to delete everything inside (silently failing if something is still opened) should work well. Moving it outside of the depot is good in once sense, as it allows the depot to be deleted, but it's bad in another sense in that it's more likely to be on a different disk, which will probably fail the mv() as I doubt you can move a mmap'ed file across device boundaries. That's probably quite an edge case however.
There was a problem hiding this comment.
Yeah sounds good JuliaLang/Pkg.jl#3812
|
This is ready to go from my perspective. Windows CI looks clean. |
staticfloat
reviewed
Feb 26, 2024
IanButterworth
force-pushed
the
ib/win_dll_cleanup
branch
from
83f908a to
1cc1cb8
Compare
tecosaur
pushed a commit
to tecosaur/julia
that referenced
this pull request
Mar 4, 2024
mkitti
pushed a commit
to mkitti/julia
that referenced
this pull request
Mar 7, 2024
adienes
pushed a commit
that referenced
this pull request
Oct 1, 2025
… in-use DLL (#59635) On Windows, during precompilation of package `A`, a DLL file is generated and may replace the existing one. If `A` is already loaded in the julia session however, the corresponding soon-to-be-obsolete DLL cannot be removed while julia is running. Currently, this problem is solved by moving the old DLL in a special "julia_delayed_deletes" folder, which will be cleaned in a later julia session by `Pkg.gc()`. However, moving an in-use DLL is only permitted on the same drive, and the "julia_delayed_deletes" is located in the `tempdir`, a.k.a. on a fixed drive, say `C:`. Thus, if the `DEPOT_PATH` points to a ".julia" in another drive, say `D:`, any precompilation occuring on an already loaded package will fail. This is what happens in #59589, actually resulting in an infinite loop that bloats up both memory and disk. @staticfloat had actually predicted that such an issue could occur in #53456 (comment). This PR fixes #59589 by changing the "delayed deleting" mechanism: instead of moving the old DLLs to a fixed folder, they are renamed in their initial folder and recorded in a list stored at a fixed location. Upon `Pkg.gc()`, the listed obsolete files will be deleted (JuliaLang/Pkg.jl#4392). It also prevents any similar infinite loops by cutting the `rm -> mv -> rm` cycle into `rm -> rename`. ~I also removed the private argument `allow_delayed_delete` from `rm`, which is only called in by [Pkg](https://github.com/JuliaLang/Pkg.jl/blob/7344e2656475261a83a6bd37d9d4cc1e7dcf5f0d/src/API.jl#L1127) but does not appear to be useful.~ EDIT: current state is #59635 (comment) --------- Co-authored-by: Jameson Nash <vtjnash@gmail.com> Co-authored-by: Elliot Saba <staticfloat@gmail.com>
KristofferC
pushed a commit
that referenced
this pull request
Oct 10, 2025
… in-use DLL (#59635) On Windows, during precompilation of package `A`, a DLL file is generated and may replace the existing one. If `A` is already loaded in the julia session however, the corresponding soon-to-be-obsolete DLL cannot be removed while julia is running. Currently, this problem is solved by moving the old DLL in a special "julia_delayed_deletes" folder, which will be cleaned in a later julia session by `Pkg.gc()`. However, moving an in-use DLL is only permitted on the same drive, and the "julia_delayed_deletes" is located in the `tempdir`, a.k.a. on a fixed drive, say `C:`. Thus, if the `DEPOT_PATH` points to a ".julia" in another drive, say `D:`, any precompilation occuring on an already loaded package will fail. This is what happens in #59589, actually resulting in an infinite loop that bloats up both memory and disk. @staticfloat had actually predicted that such an issue could occur in #53456 (comment). This PR fixes #59589 by changing the "delayed deleting" mechanism: instead of moving the old DLLs to a fixed folder, they are renamed in their initial folder and recorded in a list stored at a fixed location. Upon `Pkg.gc()`, the listed obsolete files will be deleted (JuliaLang/Pkg.jl#4392). It also prevents any similar infinite loops by cutting the `rm -> mv -> rm` cycle into `rm -> rename`. ~I also removed the private argument `allow_delayed_delete` from `rm`, which is only called in by [Pkg](https://github.com/JuliaLang/Pkg.jl/blob/7344e2656475261a83a6bd37d9d4cc1e7dcf5f0d/src/API.jl#L1127) but does not appear to be useful.~ EDIT: current state is #59635 (comment) --------- Co-authored-by: Jameson Nash <vtjnash@gmail.com> Co-authored-by: Elliot Saba <staticfloat@gmail.com> (cherry picked from commit 3e2a4ed)
KristofferC
pushed a commit
that referenced
this pull request
Oct 12, 2025
… in-use DLL (#59635) On Windows, during precompilation of package `A`, a DLL file is generated and may replace the existing one. If `A` is already loaded in the julia session however, the corresponding soon-to-be-obsolete DLL cannot be removed while julia is running. Currently, this problem is solved by moving the old DLL in a special "julia_delayed_deletes" folder, which will be cleaned in a later julia session by `Pkg.gc()`. However, moving an in-use DLL is only permitted on the same drive, and the "julia_delayed_deletes" is located in the `tempdir`, a.k.a. on a fixed drive, say `C:`. Thus, if the `DEPOT_PATH` points to a ".julia" in another drive, say `D:`, any precompilation occuring on an already loaded package will fail. This is what happens in #59589, actually resulting in an infinite loop that bloats up both memory and disk. @staticfloat had actually predicted that such an issue could occur in #53456 (comment). This PR fixes #59589 by changing the "delayed deleting" mechanism: instead of moving the old DLLs to a fixed folder, they are renamed in their initial folder and recorded in a list stored at a fixed location. Upon `Pkg.gc()`, the listed obsolete files will be deleted (JuliaLang/Pkg.jl#4392). It also prevents any similar infinite loops by cutting the `rm -> mv -> rm` cycle into `rm -> rename`. ~I also removed the private argument `allow_delayed_delete` from `rm`, which is only called in by [Pkg](https://github.com/JuliaLang/Pkg.jl/blob/7344e2656475261a83a6bd37d9d4cc1e7dcf5f0d/src/API.jl#L1127) but does not appear to be useful.~ EDIT: current state is #59635 (comment) --------- Co-authored-by: Jameson Nash <vtjnash@gmail.com> Co-authored-by: Elliot Saba <staticfloat@gmail.com> (cherry picked from commit 3e2a4ed)
KristofferC
pushed a commit
that referenced
this pull request
Oct 14, 2025
… in-use DLL (#59635) On Windows, during precompilation of package `A`, a DLL file is generated and may replace the existing one. If `A` is already loaded in the julia session however, the corresponding soon-to-be-obsolete DLL cannot be removed while julia is running. Currently, this problem is solved by moving the old DLL in a special "julia_delayed_deletes" folder, which will be cleaned in a later julia session by `Pkg.gc()`. However, moving an in-use DLL is only permitted on the same drive, and the "julia_delayed_deletes" is located in the `tempdir`, a.k.a. on a fixed drive, say `C:`. Thus, if the `DEPOT_PATH` points to a ".julia" in another drive, say `D:`, any precompilation occuring on an already loaded package will fail. This is what happens in #59589, actually resulting in an infinite loop that bloats up both memory and disk. @staticfloat had actually predicted that such an issue could occur in #53456 (comment). This PR fixes #59589 by changing the "delayed deleting" mechanism: instead of moving the old DLLs to a fixed folder, they are renamed in their initial folder and recorded in a list stored at a fixed location. Upon `Pkg.gc()`, the listed obsolete files will be deleted (JuliaLang/Pkg.jl#4392). It also prevents any similar infinite loops by cutting the `rm -> mv -> rm` cycle into `rm -> rename`. ~I also removed the private argument `allow_delayed_delete` from `rm`, which is only called in by [Pkg](https://github.com/JuliaLang/Pkg.jl/blob/7344e2656475261a83a6bd37d9d4cc1e7dcf5f0d/src/API.jl#L1127) but does not appear to be useful.~ EDIT: current state is #59635 (comment) --------- Co-authored-by: Jameson Nash <vtjnash@gmail.com> Co-authored-by: Elliot Saba <staticfloat@gmail.com> (cherry picked from commit 3e2a4ed)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Based on findings in #53394, loaded (memory mapped) DLLs cannot be deleted on Windows, even with posix delete mode, but they can be moved without breaking the loaded library. So move out to the temp dir to allow the dir to be deleted
and the DLL will be deleted via the temp dir cleanup on next reboot.Turns out Windows doesn't tidy up the temp dir automatically.. So this saves them to a fixed temp dir that Pkg.gc will tidy up JuliaLang/Pkg.jl#3812