tokio task spawn memory "leak"

[flumm]

Version
tokio v1.15.0

Platform
Linux rust-dev 5.13.19-3-pve #1 SMP PVE 5.13.19-6 (Tue, 11 Jan 2022 16:44:47 +0100) x86_64 GNU/Linux

Description
it seems that some part (i guess it has something to do with spawned tasks) does not return memory as it should

I tried this code:

use std::time::Duration;
use tokio::task;
use std::io;

async fn wait(i: u32) {
    let delay_in_seconds = Duration::new(5, 0);
    tokio::time::sleep(delay_in_seconds).await;
    println!("{}", i);
}

fn main() {
    let rt = tokio::runtime::Runtime::new().unwrap();
    rt.block_on(async move {
        let mut handles = Vec::new();
        for i in 0..1000000 {
            handles.push(task::spawn(async move {
                wait(i).await;
            }));
        }

        for handle in handles.into_iter() {
            let _ = handle.await;
        }

        let mut buffer = String::new();
        io::stdin().read_line(&mut buffer).expect("error");
    });
}

which allocates up to 1.4GiB! (checking htops RES column) here (16core/32thread epyc) and frees it again when waiting on the input
but if i comment out the line

            let _ = handle.await;                          

though, it still consumes the 1.4GiB and never frees it
i expected the resources of a spawned task to be freed in any case, not when i immediately await the spawned tasks

release or debug mode does not seem to make a difference

[Darksonn]

The JoinHandle type holds a reference count to the task, and the task's memory is not freed until the JoinHandle is dropped. It does not matter whether it is awaited or not, dropping it without awaiting it will also release the memory.

[flumm]

Hi, yeah, sorry for the noise.

I researched a bit more, and it seems that the default allocator does not properly free the memory in those situations (I don't completely understand why, though).
I can "fix" the behavior by either using jemalloc, or by tuning the malloc parameters according to mallopt(3).
So tokio does the right thing, but the example (and seemingly our code), seems to trigger some bad cases for the default rust allocator under Linux.

Thanks

[sfackler]

Allocators hold onto deallocated memory regions to improve allocation throughput.

[Darksonn]

I researched a bit more, and it seems that the default allocator does not properly free the memory in those situations (I don't completely understand why, though).

This is actually the cause behind most of the leak reports we get for Tokio. It happens for performance reasons — it is faster to hold on to the memory than to release it and then ask for it again later.

In your case, I actually guessed that you had commented out the entire for loop (as opposed to just the line with the .await), in which case the memory really wouldn't have been freed because the JoinHandles still existed in the vector during the read from stdin.

[flumm]

Allocators hold onto deallocated memory regions to improve allocation throughput.

i knew that, but my understanding was/is that there will be memory returned to the host 'under certain conditions' (e.g. see mallopt(3) comment about M_TRIM_THRESHOLD) but in case of glibc it seems not to be that simple..

a program that basically has gigabytes allocated but not used seems rather pointless. in our case, customers ran into oom situations because of the combined used ram of various things, including our main daemon (that "used" >5GiB)

I researched a bit more, and it seems that the default allocator does not properly free the memory in those situations (I don't completely understand why, though).

This is actually the cause behind most of the leak reports we get for Tokio. It happens for performance reasons — it is faster to hold on to the memory than to release it and then ask for it again later.

In your case, I actually guessed that you had commented out the entire for loop (as opposed to just the line with the .await), in which case the memory really wouldn't have been freed because the JoinHandles still existed in the vector during the read from stdin.

no i deliberately left the 'into_iter()' so that the joinhandles were consumed. i could have phrased it better though, yes.