Question about releasing memory

[elgatito]

Hello,

I'm trying to use mimalloc with my binary. And have questions about memory release to OS.
I have a memory storage, it allocates 300MB of memory.
Then I destroy it.
And expect to return memory back to OS, but instead I see memory is not returned.
Then I init another storage and memory grows by another 300MB.

Compiling mimalloc:

cmake -B cmake-build-dir/release -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_STANDARD=17

Compiling my binary with:

/usr/x86_64-linux-musl-cross/lib/mimalloc-1.7/libmimalloc.a

Added to main.cxx:

#include <mimalloc.h>
#include <mimalloc-override.h>

Compiling everything with Release build type.

What I see in process stats:

Started:
elgato    2429  2.0  0.1 272984  8808 pts/3    Sl+  17:57   0:00 build/linux_x64/lt2http

Storage init:
elgato    2429  1.4  4.0 797568 322636 pts/3   Sl+  17:57   0:00 build/linux_x64/lt2http

Storage destroyed:
elgato    2429  1.0  4.0 797712 322900 pts/3   Sl+  17:57   0:00 build/linux_x64/lt2http

Storage re-init:
elgato    2429  1.0  4.7 797712 381244 pts/3   Sl+  17:57   0:00 build/linux_x64/lt2http

Storage re-destroyed:
elgato    2429  0.8  4.7 797712 381244 pts/3   Sl+  17:57   0:01 build/linux_x64/lt2http

Stats after closing:

heap stats:     peak      total      freed    current       unit      count  
  reserved:   768.0 mb   768.0 mb   512.0 mb   256.0 mb       1 b              not all freed!
 committed:   745.9 mb   745.9 mb   512.0 mb   233.9 mb       1 b              not all freed!
     reset:     4.2 mb     5.4 mb     1.1 mb     4.2 mb       1 b              not all freed!
   touched:       0 b        0 b    606.3 mb  -606.3 mb       1 b              ok
  segments:      16        318        314          4                           not all freed!
-abandoned:       5          5          5          0                           ok
   -cached:       0          0          0          0                           ok
     pages:     300        353        325         28                           not all freed!
-abandoned:      30         30         30          0                           ok
 -extended:       0   
 -noretire:       0   
     mmaps:       0   
   commits:     198   
   threads:       7          7         12         -5                           ok
  searches:     0.0 avg
numa nodes:       1
   elapsed:     144.618 s
   process: user: 0.610 s, system: 0.552 s, faults: 0, rss: 372.6 mb, commit: 745.9 mb

I understand that mimalloc is probably keeping memory for re-use in future.
But is there any configuration, that can be changed to force memory release?

[daanx]

Hmm, as I understand it, every time you re-init your "storage" the memory usage grows by 300MiB ? That is not good. Does this happen with the default allocator as well ? (ie. did you confirm it is not a leak in your program?).

Is it possible to share the program so I can test it locally? If you cnannot share the source, perhaps even the binary may work (best on Linux so I can easily switch allocators using LD_PRELOAD).

[elgatito]

@daanx with default allocator memory is released properly. I will try to prepare a use case for you.

[daanx]

@daanx with default allocator memory is released properly. I will try to prepare a use case for you.

Ouch! I am very interested to be able to test with this. Thanks and let me know if I can help. You can use my work email (daan at microsoft.com) if you want the tests not be public.

[daanx]

Hi @elgatito; just following up on this -- if you have a test that reproduces this I would like to look at it :-) Just let me know

[elgatito]

@daanx Sorry for late response. Wanted to make a clean project to reproduce the issue. But when I just create vector of memory buffers and destroy them - memory is not increasing.

Code is open source, but needs adding a torrent file for downloading, to create a memory storage.
https://github.com/ElementumOrg/lt2http/blob/master/src/bittorrent/memory_storage.cpp

Basically, there is a vector of memory buffers (vector), they are created when torrent is added, then torrent is removed, storage is destroyed, when new torrent is added and new memory storage constructed - I have increased RES memory.

[elgatito]

@daanx huh.. After writing the comment I tried to change memory storage initialization parameters and I could reproduce the increasing memory.

https://github.com/elgatito/mimalloc-test

mkdir -p build && cd build
cmake .. && make 

./test

When you run test it will make a memory storage, then sleep, destroy it, then another one.

[daanx]

Thanks for the test! I was able to reproduce the problem on v1.7.1 but it was good for v2.0.1 (dev-slice branch). Can you see if in your full program v2.0.1 works fine as well? I will continue to investigate the issue for v1.7.1 as well. Thanks