Command hangs when autoMemoryReclaim=gradual with distro systemd enabled

[AzureZeng]

Windows Version

Microsoft Windows [Version 10.0.22621.2428]

WSL Version

2.0.5.0

Are you using WSL 1 or WSL 2?

• WSL 2
• WSL 1

Kernel Version

5.15.133.1-1

Distro Version

Debian 12 + Arch Linux

Other Software

Visual Studio Code (Latest version) with C# extension
.NET Core SDK 6.0/7.0
Docker Desktop 4.24.2 (124339)

Repro Steps

1. Update to latest WSL pre-release version by running wsl --update --pre-release
2. Enable systemd support in wsl.conf
3. Set autoMemoryReclaim=gradual in .wslconfig
4. Restart WSL to apply settings
5. Do something such as updating packages or writing apps in WSL, or open Docker Desktop, then WSL will allocate a lot of buff/cache memory
6. After that, place WSL distro command line windows aside for several minutes, to avoid WSL terminating distro automatically, and to allow WSL reclaiming memory
7. Then, execute arbitrary of commands, such as sudo pacman -Syu sudo apt update systemctl status.

Expected Behavior

This command will be executed instantly.

Actual Behavior

You will find that this command may hangs for a very long time. After kernel re-allocating buff/cache memory, everything works again.

Disabling systemd support also works.

I think it is maybe a mechanism in Linux kernel.

Screenshots:

Diagnostic Logs

WslLogs-2023-10-25_16-30-50.zip

[OneBlue]

Thank you for reporting this @AzureZeng. Could you capture logs of the entire sequence, and then capture /dumps when the command hangs so we can see what's happening ?

[AzureZeng]

Updated more details about this issue.

What specifically do you mean by ". After kernel allocating buff/cache memory, everything works again." ?

[AzureZeng]

What specifically do you mean by ". After kernel allocating buff/cache memory, everything works again." ?

The buff/cache memory allocated when starting distro is gradually dropped by WSL mechanism (autoMemoryReclaim=gradual). When buff/cache memory is fully dropped, execute any commands in WSL may cause hanging. When hanging, I found that kernel is trying to re-allocate buff/cache memory, and after the full kernel buff/cache memory re-allocation, everything works again.

I think dropping buff/cache memory related to systemd by this memory reclaim mechanism, which is not presented in autoMemoryReclaim=dropcache or manually echo 3 > /proc/sys/vm/drop_caches, is the root cause of this problem.

[gallois]

I was wondering why running apt update was hanging on my laptop and this seems to make sense, as the steps for reproducing aligned with what I experienced.

WSL version: 2.0.14.0
Kernel version: 5.15.133.1-1
Ubuntu 22.04.3

running systemd and autoMemoryReclaim=gradual

[tl24]

I see similar issues on my system as well. Whenever autoMemoryReclaim=gradual, then wsl just hangs and becomes unresponsive after a short period of time. I have systemd enabled as well.

[Artoria2e5]

Analysis (guesses)

The gradual reclaim works by writing to /proc/cgroup/fs/memory.reclaim. Unlike the drop_cache thing, this knob allows you to specify how much you want to reclaim.

The write blocks until the reclaim is done (or a certain number of iterations is reached), which could be a reason for the hang. It also throttles a bunch of things while it is running, so that could also affects things other than the thread that calls reclaim.

https://lkml.org/lkml/2020/10/5/145 seems to be an early version of the reclaim patch. It suggests sending a signal to the writing process to stop the reclaim. The actual merged version torvalds/linux@9496838 does not have a similar documentation sentence, but it has a signal_pending(current) check so it likely does the same. WSL should use it to do a time limit.

The connection to systemd is less obvious. The most likely explanation is that enabling a unified hierarchy in systemd causes /sys/fs/cgroup/memory.reclaim to exist in the first place, as it would otherwise be only be available to a different cgroup-v1 hierarchy, not the root node (a controller can only be available in one hierarchy; systemd on a v1-compatible mode would have memory.reclaim in /sys/fs/cgroup/memory/memory.reclaim.) In other words we have a bug (reclaim being prone to hang to do not time limit) being hidden by another bug (reclaim path is written in a way that depends on the unified hierarchy, which is hidden behind the systemd switch). This hypothesis can be verified or disproven by checking the WSL log on a non-systemd "enabled" distro and looking for the warning

access(/sys/fs/cgroup/memory/memory.reclaim, W_OK) failed 2, falling back to autoMemoryReclaim = dropcache

(2 is a guess based on ENOENT)

There is also a very strong recommendation from systemd that you do not fight systemd over any cgroup. The strongest wording is that if systemd writes it, you should not. systemd writes the root cgroup, but does not write memory.reclaim specifically, so honestly it's not really fighting so I recommend disregarding it.

Recommended fix

WSL should make the write to /proc/cgroup/fs/memory.reclaim time limited; I would say 125ms is long enough for normal reclaims, so make that the limit.

Unfortunately there just isn't a time-limited write utility anywhere yet. src/linux/init/util.cpp only uses the basic syscall write(). This is fixable since we already have a thread system: just spawn a new thread for the write then send a signal to it when the time is up.

Actual async io like io_uring isn't in this codebase and would not work anyways because the interrupt-check specifically looks for a signal.

The other recommended fix

We can also just forget about writing to memory.reclaim ourselves. Like, why are we even doing that? The kernel has a DAMON_RECLAIM already:

• It has a very sensible time limit of 10 ms default. This limit is also applied over a window (default 1 s) for a maximum CPU load of 0.1% default.
• It works based on the LRU age of the region and can guarantee to not remove regions younger than a limit (default 120 s). That reduces thrashing. Being non-userspace has its perks.
• It too has a limit for reclaim throughput using the quota mechanism (default 128 MB per 1-s window, the same window is used for time limit).
• You don't even need to manage your own thread, at all!