Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Another apparent memory leak #634

Open
tavmem opened this issue Oct 22, 2022 · 9 comments
Open

Another apparent memory leak #634

tavmem opened this issue Oct 22, 2022 · 9 comments
Labels
bug

Comments

@tavmem
Copy link
Collaborator

tavmem commented Oct 22, 2022 •
edited
Loading

Make this change

$ git diff
diff --git a/src/0.c b/src/0.c
index 24a2753..5be0011 100644
--- a/src/0.c
+++ b/src/0.c
@@ -509,6 +509,7 @@ K _1m(K x) {    //Keeps binary files mapped
   K z = _1m_r(f,v,v,v+s,&b);
   r=close(f); if(r)R FE;
   r=munmap(v,s); if(r)R UE;
+  O("mUsed: %f\n",mUsed);
   R z;
 }
$

then

$ rlwrap -n ./k
kona      \ for help. \\ to exit.

  "file" 1: ("a"; 4 5); 1: "file";
mUsed: 3472.000000
  "file" 1: ("a"; 4 5); 1: "file";
mUsed: 3552.000000
  "file" 1: ("a"; 4 5); 1: "file";
mUsed: 3757.000000
  "file" 1: ("a"; 4 5); 1: "file";
Segmentation fault (core dumped)
$

(The segfault is being addressed in issue #633.)

Note:
3552-3472
80
3757-3552
205

The operating system here was problably Linux.
@tavmem tavmem added the bug label Oct 22, 2022
@tavmem tavmem mentioned this issue Oct 25, 2022
Closed
@tavmem
Copy link
Collaborator Author

tavmem commented Jan 21, 2023 •
edited
Loading

Since this issue was opened, on Oct 22, 2022 there were 9 commits made resolving issues 633, 635, 637, 638, 628 and 640.
The other 3 commits resolved miscellaneous problems. Trying the above now (in Windows), we get

$ rlwrap -n ./k
kona      \ for help. \\ to exit.

  "file" 1: ("ab"; "cd"); 1: "file";
mUsed: 3512.000000
  "file" 1: ("ab"; "cd"); 1: "file";
mUsed: 3756.000000
  "file" 1: ("ab"; "cd"); 1: "file";
mUsed: 4000.000000
  "file" 1: ("ab"; "cd"); 1: "file";
mUsed: 4244.000000
  3756-3512
244
  4000-3756
244
  4244-4000
244

The memory leak (in Windows) is now consistent.

@tavmem
Copy link
Collaborator Author

tavmem commented Jan 21, 2023 •
edited
Loading

In Linux

$ rlwrap -n ./k
kona      \ for help. \\ to exit.

  "file" 1: ("a"; 4 5); 1: "file";
mUsed: 3472.000000
  "file" 1: ("a"; 4 5); 1: "file";
mUsed: 3552.000000
  "file" 1: ("a"; 4 5); 1: "file";
mUsed: 3632.000000
  "file" 1: ("a"; 4 5); 1: "file";
mUsed: 3712.000000
  3552-3472
80
  3632-3552
80
  3712-3632
80
  \\
$

The memory leak is also consistent, but smaller.
The results in IOS are the same as in Linux.

@tavmem
Copy link
Collaborator Author

tavmem commented May 24, 2023 •
edited
Loading

So ... I've been away since January working on some other projects ...
Anyway ... the 1st task (in Linux & IOS) is to identify the source of the 80 extra bytes in each execution.
Making the following code changes:

$ git diff
diff --git a/src/0.c b/src/0.c
index 9f5672f..efb16ae 100644
--- a/src/0.c
+++ b/src/0.c
@@ -546,7 +546,7 @@ Z K _1m_r(I f,V fixed, V v,V aft,I*b) {   //File descriptor, moving * into mmap,
     if(MAP_FAILED==(u=mmap(0,size,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_NORESERVE,f,offset))){R SE;}
 
     mMap+=length;
-    mUsed+=length;if(mUsed>mMax)mMax=mUsed;
+    O("          0.c mUsed0: %0.0f  ",mUsed); mTmp=mUsed; mUsed+=length;
+    O("mUsed1: %0.0f  %0.0f   ****************************************\n",mUsed,mUsed-mTmp);
+    if(mUsed>mMax)mMax=mUsed;
 
     z=(K)(((V)u+mod)-3*sizeof(I)); //3*sizeof(I) for c,t,n
 
diff --git a/src/0.h b/src/0.h
index 2dc3b73..919c979 100644
--- a/src/0.h
+++ b/src/0.h
@@ -53,4 +53,5 @@ V membswpF(V d,V s,I n,I x);
 K read_tape(I i,I j,I type);
 extern F mMap;
 extern F mUsed;
+extern F mTmp;
 extern F mMax;
diff --git a/src/km.c b/src/km.c
index fd92020..1f2a5b9 100644
--- a/src/km.c
+++ b/src/km.c
@@ -28,7 +28,7 @@
 #define KP_MAX 26 //2^x, 26->64MB  //TODO: base on available memory at startup (fixed percent? is 64M/2G a good percent?)
 V KP[KP_MAX+1]; //KPOOL
 I PG; //pagesize:  size_t page_size = (size_t) sysconf (_SC_PAGESIZE);
-F mUsed=0.0, mAlloc=0.0, mMap=0.0, mMax=0.0;
+F mUsed=0.0, mAlloc=0.0, mMap=0.0, mMax=0.0, mTmp=0.0;
 
 #if UINTPTR_MAX >= 0xffffffffffffffff //64 bit
 #define MAX_OBJECT_LENGTH (((unsigned long long)1) << 45) //for catching obviously incorrect allocations
@@ -188,7 +188,7 @@ Z V unpool(I r)
     *L=z;
   }
   z=*L;*L=*z;*z=0;
-  mUsed+=k; if(mUsed>mMax)mMax=mUsed;
+  O("r: %lld   unpool-mUsed0: %0.0f  ",r,mUsed); mTmp=mUsed; mUsed+=k; 
+  O("mUsed1: %0.0f  %0.0f\n",mUsed,mUsed-mTmp); if(mUsed>mMax)mMax=mUsed;
   R z;
 }
 
@@ -217,7 +217,7 @@ I repool(V v,I r)//assert r < KP_MAX
   memset(v,0,k);
   *(V*)v=KP[r];
   KP[r]=v;
-  mUsed -= k;
+  O("r: %lld   repool-mUsed0: %0.0f  ",r,mUsed); mTmp=mUsed; mUsed -= k;
+  O("mUsed1: %0.0f  %0.0f\n",mUsed,mUsed-mTmp);
   R 0;

we get

$ rlwrap -n ./k
r: 6   unpool-mUsed0: 0  mUsed1: 64  64
r: 6   unpool-mUsed0: 64  mUsed1: 128  64
r: 6   unpool-mUsed0: 128  mUsed1: 192  64
r: 6   unpool-mUsed0: 192  mUsed1: 256  64
r: 6   unpool-mUsed0: 256  mUsed1: 320  64
r: 6   unpool-mUsed0: 320  mUsed1: 384  64
r: 6   unpool-mUsed0: 384  mUsed1: 448  64
r: 6   unpool-mUsed0: 448  mUsed1: 512  64
r: 6   unpool-mUsed0: 512  mUsed1: 576  64
r: 6   unpool-mUsed0: 576  mUsed1: 640  64
r: 6   unpool-mUsed0: 640  mUsed1: 704  64
r: 6   unpool-mUsed0: 704  mUsed1: 768  64
r: 6   unpool-mUsed0: 768  mUsed1: 832  64
r: 6   unpool-mUsed0: 832  mUsed1: 896  64
r: 6   unpool-mUsed0: 896  mUsed1: 960  64
r: 6   unpool-mUsed0: 960  mUsed1: 1024  64
r: 6   unpool-mUsed0: 1024  mUsed1: 1088  64
r: 6   unpool-mUsed0: 1088  mUsed1: 1152  64
r: 6   unpool-mUsed0: 1152  mUsed1: 1216  64
r: 6   unpool-mUsed0: 1216  mUsed1: 1280  64
r: 6   unpool-mUsed0: 1280  mUsed1: 1344  64
r: 6   unpool-mUsed0: 1344  mUsed1: 1408  64
r: 6   unpool-mUsed0: 1408  mUsed1: 1472  64
kona      \ for help. \\ to exit.

  "file" 1: ("a"; 4 5); 1: "file";
r: 6   unpool-mUsed0: 1472  mUsed1: 1536  64
r: 9   unpool-mUsed0: 1536  mUsed1: 2048  512
r: 6   unpool-mUsed0: 2048  mUsed1: 2112  64
r: 6   repool-mUsed0: 2112  mUsed1: 2048  -64
r: 7   unpool-mUsed0: 2048  mUsed1: 2176  128
r: 6   unpool-mUsed0: 2176  mUsed1: 2240  64
r: 6   unpool-mUsed0: 2240  mUsed1: 2304  64
r: 6   unpool-mUsed0: 2304  mUsed1: 2368  64
r: 7   unpool-mUsed0: 2368  mUsed1: 2496  128
r: 6   unpool-mUsed0: 2496  mUsed1: 2560  64
r: 6   unpool-mUsed0: 2560  mUsed1: 2624  64
r: 6   unpool-mUsed0: 2624  mUsed1: 2688  64
r: 7   unpool-mUsed0: 2688  mUsed1: 2816  128
r: 6   unpool-mUsed0: 2816  mUsed1: 2880  64
r: 6   repool-mUsed0: 2880  mUsed1: 2816  -64
r: 7   unpool-mUsed0: 2816  mUsed1: 2944  128
r: 6   unpool-mUsed0: 2944  mUsed1: 3008  64
r: 6   unpool-mUsed0: 3008  mUsed1: 3072  64
r: 6   unpool-mUsed0: 3072  mUsed1: 3136  64
r: 6   unpool-mUsed0: 3136  mUsed1: 3200  64
r: 6   unpool-mUsed0: 3200  mUsed1: 3264  64
r: 6   unpool-mUsed0: 3264  mUsed1: 3328  64
r: 6   unpool-mUsed0: 3328  mUsed1: 3392  64
r: 6   unpool-mUsed0: 3392  mUsed1: 3456  64
r: 6   unpool-mUsed0: 3456  mUsed1: 3520  64
r: 6   repool-mUsed0: 3520  mUsed1: 3456  -64
r: 6   unpool-mUsed0: 3456  mUsed1: 3520  64
r: 6   repool-mUsed0: 3520  mUsed1: 3456  -64
r: 6   unpool-mUsed0: 3456  mUsed1: 3520  64
r: 6   unpool-mUsed0: 3520  mUsed1: 3584  64
r: 7   repool-mUsed0: 3584  mUsed1: 3456  -128
r: 6   repool-mUsed0: 3456  mUsed1: 3392  -64
r: 6   unpool-mUsed0: 3392  mUsed1: 3456  64
r: 6   unpool-mUsed0: 3456  mUsed1: 3520  64
r: 6   unpool-mUsed0: 3520  mUsed1: 3584  64
r: 6   unpool-mUsed0: 3584  mUsed1: 3648  64
r: 6   unpool-mUsed0: 3648  mUsed1: 3712  64
r: 9   repool-mUsed0: 3712  mUsed1: 3200  -512
r: 6   repool-mUsed0: 3200  mUsed1: 3136  -64
r: 6   unpool-mUsed0: 3136  mUsed1: 3200  64
r: 6   unpool-mUsed0: 3200  mUsed1: 3264  64
r: 6   repool-mUsed0: 3264  mUsed1: 3200  -64
r: 6   repool-mUsed0: 3200  mUsed1: 3136  -64
r: 6   unpool-mUsed0: 3136  mUsed1: 3200  64
r: 6   unpool-mUsed0: 3200  mUsed1: 3264  64
r: 6   unpool-mUsed0: 3264  mUsed1: 3328  64
          0.c mUsed0: 3328  mUsed1: 3472  144   ***********************************************************
r: 6   repool-mUsed0: 3472  mUsed1: 3408  -64
r: 6   repool-mUsed0: 3408  mUsed1: 3344  -64
r: 6   repool-mUsed0: 3344  mUsed1: 3280  -64
r: 6   repool-mUsed0: 3280  mUsed1: 3216  -64
r: 7   repool-mUsed0: 3216  mUsed1: 3088  -128
r: 6   repool-mUsed0: 3088  mUsed1: 3024  -64
r: 6   repool-mUsed0: 3024  mUsed1: 2960  -64
r: 6   repool-mUsed0: 2960  mUsed1: 2896  -64
r: 6   repool-mUsed0: 2896  mUsed1: 2832  -64
r: 6   repool-mUsed0: 2832  mUsed1: 2768  -64
r: 6   repool-mUsed0: 2768  mUsed1: 2704  -64
r: 6   repool-mUsed0: 2704  mUsed1: 2640  -64
r: 6   repool-mUsed0: 2640  mUsed1: 2576  -64
r: 6   repool-mUsed0: 2576  mUsed1: 2512  -64
r: 6   repool-mUsed0: 2512  mUsed1: 2448  -64
r: 6   repool-mUsed0: 2448  mUsed1: 2384  -64
r: 6   repool-mUsed0: 2384  mUsed1: 2320  -64
r: 7   repool-mUsed0: 2320  mUsed1: 2192  -128
r: 6   repool-mUsed0: 2192  mUsed1: 2128  -64
r: 6   repool-mUsed0: 2128  mUsed1: 2064  -64
r: 6   repool-mUsed0: 2064  mUsed1: 2000  -64
r: 6   repool-mUsed0: 2000  mUsed1: 1936  -64
r: 6   repool-mUsed0: 1936  mUsed1: 1872  -64
r: 6   repool-mUsed0: 1872  mUsed1: 1808  -64
r: 6   repool-mUsed0: 1808  mUsed1: 1744  -64
r: 7   repool-mUsed0: 1744  mUsed1: 1616  -128
r: 6   repool-mUsed0: 1616  mUsed1: 1552  -64

You can check that nothing is missed in that the final mUsed1 becomes the initial mUsed0 in the next step.
The problem occurs in the step with the "*******************" where 144 bytes are added.
This is 80 bytes more than the typical release of 64 bytes.

Note that the final result of 1552 is exactly the 80 bytes higher than before the executed command.

@tavmem
Copy link
Collaborator Author

tavmem commented May 24, 2023

The next task is to compare the results above to the results we get by using valgrind:

$ valgrind --leak-check=full ./k
==42301== Memcheck, a memory error detector
==42301== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==42301== Using Valgrind-3.21.0 and LibVEX; rerun with -h for copyright info
==42301== Command: ./k
==42301== 
kona      \ for help. \\ to exit.

  "file" 1: ("a"; 4 5); 1: "file";
  \\
==42301== 
==42301== HEAP SUMMARY:
==42301==     in use at exit: 180 bytes in 12 blocks
==42301==   total heap usage: 64 allocs, 52 frees, 28,606 bytes allocated
==42301== 
==42301== 7 bytes in 1 blocks are definitely lost in loss record 4 of 10
==42301==    at 0x484182F: malloc (vg_replace_malloc.c:431)
==42301==    by 0x425BD8: alloc (km.c:45)
==42301==    by 0x42A803: glueSS (v.c:27)
==42301==    by 0x406F7D: _1d (0.c:568)
==42301==    by 0x41E63F: vf_ex (kx.c:455)
==42301==    by 0x4202D8: dv_ex (kx.c:381)
==42301==    by 0x41DA92: ex2.part.0 (kx.c:911)
==42301==    by 0x422D87: ex0 (kx.c:645)
==42301==    by 0x4237C5: ex_ (kx.c:610)
==42301==    by 0x4237C5: ex_ (kx.c:599)
==42301==    by 0x4237C5: ex (kx.c:632)
==42301==    by 0x41A3C7: line (kc.c:274)
==42301==    by 0x41C5B0: attend (kc.c:433)
==42301==    by 0x4037C2: main (main.c:8)
==42301== 
==42301== 17 bytes in 1 blocks are possibly lost in loss record 5 of 10
==42301==    at 0x484182F: malloc (vg_replace_malloc.c:431)
==42301==    by 0x425BD8: alloc (km.c:45)
==42301==    by 0x429C8A: sdupI (ks.c:10)
==42301==    by 0x429C8A: sp (ks.c:27)
==42301==    by 0x419FBB: kinit (kc.c:165)
==42301==    by 0x4037B1: main (main.c:6)
==42301== 
==42301== 18 bytes in 1 blocks are possibly lost in loss record 6 of 10
==42301==    at 0x484182F: malloc (vg_replace_malloc.c:431)
==42301==    by 0x425BD8: alloc (km.c:45)
==42301==    by 0x429C8A: sdupI (ks.c:10)
==42301==    by 0x429C8A: sp (ks.c:27)
==42301==    by 0x419FF9: kinit (kc.c:172)
==42301==    by 0x4037B1: main (main.c:6)
==42301== 
==42301== 18 bytes in 1 blocks are possibly lost in loss record 7 of 10
==42301==    at 0x484182F: malloc (vg_replace_malloc.c:431)
==42301==    by 0x425BD8: alloc (km.c:45)
==42301==    by 0x429C8A: sdupI (ks.c:10)
==42301==    by 0x429C8A: sp (ks.c:27)
==42301==    by 0x41A033: kinit (kc.c:174)
==42301==    by 0x4037B1: main (main.c:6)
==42301== 
==42301== 19 bytes in 1 blocks are possibly lost in loss record 8 of 10
==42301==    at 0x484182F: malloc (vg_replace_malloc.c:431)
==42301==    by 0x425BD8: alloc (km.c:45)
==42301==    by 0x429DE7: sdupI (ks.c:10)
==42301==    by 0x429DE7: sp (ks.c:24)
==42301==    by 0x419FAA: kinit (kc.c:165)
==42301==    by 0x4037B1: main (main.c:6)
==42301== 
==42301== 38 bytes in 1 blocks are possibly lost in loss record 9 of 10
==42301==    at 0x484182F: malloc (vg_replace_malloc.c:431)
==42301==    by 0x425BD8: alloc (km.c:45)
==42301==    by 0x429C8A: sdupI (ks.c:10)
==42301==    by 0x429C8A: sp (ks.c:27)
==42301==    by 0x40FBBF: _h (r.c:1030)
==42301==    by 0x41A0BB: kinit (kc.c:180)
==42301==    by 0x4037B1: main (main.c:6)
==42301== 
==42301== 54 bytes in 3 blocks are possibly lost in loss record 10 of 10
==42301==    at 0x484182F: malloc (vg_replace_malloc.c:431)
==42301==    by 0x425BD8: alloc (km.c:45)
==42301==    by 0x429C8A: sdupI (ks.c:10)
==42301==    by 0x429C8A: sp (ks.c:27)
==42301==    by 0x419FCF: kinit (kc.c:165)
==42301==    by 0x4037B1: main (main.c:6)
==42301== 
==42301== LEAK SUMMARY:
==42301==    definitely lost: 7 bytes in 1 blocks
==42301==    indirectly lost: 0 bytes in 0 blocks
==42301==      possibly lost: 164 bytes in 8 blocks
==42301==    still reachable: 9 bytes in 3 blocks
==42301==         suppressed: 0 bytes in 0 blocks
==42301== Reachable blocks (those to which a pointer was found) are not shown.
==42301== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==42301== 
==42301== For lists of detected and suppressed errors, rerun with: -s
==42301== ERROR SUMMARY: 7 errors from 7 contexts (suppressed: 0 from 0)
$

@tavmem
Copy link
Collaborator Author

tavmem commented May 26, 2023

In Windows, running the same test described in the Jan 20, 2024 using the current code base, we get

$ rlwrap -n ./k
kona      \ for help. \\ to exit.

    "file" 1: ("a"; 4 5); 1: "file";
mUsed: 3472.000000
    "file" 1: ("a"; 4 5); 1: "file";
mUsed: 3614.000000
    "file" 1: ("a"; 4 5); 1: "file";
mUsed: 3756.000000

  3614-3472
142
  3756-3614
142

Making the code changes described above in Windows we can see where the problem occurs

$ rlwrap -n ./k
r: 6   unpool-mUsed0: 0  mUsed1: 64  64
r: 6   unpool-mUsed0: 64  mUsed1: 128  64
r: 6   unpool-mUsed0: 128  mUsed1: 192  64
r: 6   unpool-mUsed0: 192  mUsed1: 256  64
r: 6   unpool-mUsed0: 256  mUsed1: 320  64
r: 6   unpool-mUsed0: 320  mUsed1: 384  64
r: 6   unpool-mUsed0: 384  mUsed1: 448  64
r: 6   unpool-mUsed0: 448  mUsed1: 512  64
r: 6   unpool-mUsed0: 512  mUsed1: 576  64
r: 6   unpool-mUsed0: 576  mUsed1: 640  64
r: 6   unpool-mUsed0: 640  mUsed1: 704  64
r: 6   unpool-mUsed0: 704  mUsed1: 768  64
r: 6   unpool-mUsed0: 768  mUsed1: 832  64
r: 6   unpool-mUsed0: 832  mUsed1: 896  64
r: 6   unpool-mUsed0: 896  mUsed1: 960  64
r: 6   unpool-mUsed0: 960  mUsed1: 1024  64
r: 6   unpool-mUsed0: 1024  mUsed1: 1088  64
r: 6   unpool-mUsed0: 1088  mUsed1: 1152  64
r: 6   unpool-mUsed0: 1152  mUsed1: 1216  64
r: 6   unpool-mUsed0: 1216  mUsed1: 1280  64
r: 6   unpool-mUsed0: 1280  mUsed1: 1344  64
r: 6   unpool-mUsed0: 1344  mUsed1: 1408  64
r: 6   unpool-mUsed0: 1408  mUsed1: 1472  64
kona      \ for help. \\ to exit.

    "file" 1: ("a"; 4 5); 1: "file";
r: 6   unpool-mUsed0: 1472  mUsed1: 1536  64
r: 9   unpool-mUsed0: 1536  mUsed1: 2048  512
r: 6   unpool-mUsed0: 2048  mUsed1: 2112  64
r: 6   repool-mUsed0: 2112  mUsed1: 2048  -64
r: 7   unpool-mUsed0: 2048  mUsed1: 2176  128
r: 6   unpool-mUsed0: 2176  mUsed1: 2240  64
r: 6   unpool-mUsed0: 2240  mUsed1: 2304  64
r: 6   unpool-mUsed0: 2304  mUsed1: 2368  64
r: 7   unpool-mUsed0: 2368  mUsed1: 2496  128
r: 6   unpool-mUsed0: 2496  mUsed1: 2560  64
r: 6   unpool-mUsed0: 2560  mUsed1: 2624  64
r: 6   unpool-mUsed0: 2624  mUsed1: 2688  64
r: 7   unpool-mUsed0: 2688  mUsed1: 2816  128
r: 6   unpool-mUsed0: 2816  mUsed1: 2880  64
r: 6   repool-mUsed0: 2880  mUsed1: 2816  -64
r: 7   unpool-mUsed0: 2816  mUsed1: 2944  128
r: 6   unpool-mUsed0: 2944  mUsed1: 3008  64
r: 6   unpool-mUsed0: 3008  mUsed1: 3072  64
r: 6   unpool-mUsed0: 3072  mUsed1: 3136  64
r: 6   unpool-mUsed0: 3136  mUsed1: 3200  64
r: 6   unpool-mUsed0: 3200  mUsed1: 3264  64
r: 6   unpool-mUsed0: 3264  mUsed1: 3328  64
r: 6   unpool-mUsed0: 3328  mUsed1: 3392  64
r: 6   unpool-mUsed0: 3392  mUsed1: 3456  64
r: 6   unpool-mUsed0: 3456  mUsed1: 3520  64
r: 6   repool-mUsed0: 3520  mUsed1: 3456  -64
r: 6   unpool-mUsed0: 3456  mUsed1: 3520  64
r: 6   repool-mUsed0: 3520  mUsed1: 3456  -64
r: 6   unpool-mUsed0: 3456  mUsed1: 3520  64
r: 6   unpool-mUsed0: 3520  mUsed1: 3584  64
r: 7   repool-mUsed0: 3584  mUsed1: 3456  -128
r: 6   repool-mUsed0: 3456  mUsed1: 3392  -64
r: 6   unpool-mUsed0: 3392  mUsed1: 3456  64
r: 6   unpool-mUsed0: 3456  mUsed1: 3520  64
r: 6   unpool-mUsed0: 3520  mUsed1: 3584  64
r: 6   unpool-mUsed0: 3584  mUsed1: 3648  64
r: 6   unpool-mUsed0: 3648  mUsed1: 3712  64
r: 9   repool-mUsed0: 3712  mUsed1: 3200  -512
r: 6   repool-mUsed0: 3200  mUsed1: 3136  -64
r: 6   unpool-mUsed0: 3136  mUsed1: 3200  64
r: 6   unpool-mUsed0: 3200  mUsed1: 3264  64
r: 6   repool-mUsed0: 3264  mUsed1: 3200  -64
r: 6   repool-mUsed0: 3200  mUsed1: 3136  -64
r: 6   unpool-mUsed0: 3136  mUsed1: 3200  64
r: 6   unpool-mUsed0: 3200  mUsed1: 3264  64
r: 6   unpool-mUsed0: 3264  mUsed1: 3328  64
          0.c mUsed0: 3328  mUsed1: 3472  144   ****************************************
r: 6   repool-mUsed0: 3472  mUsed1: 3408  -64
r: 1   repool-mUsed0: 3408  mUsed1: 3406  -2
r: 6   repool-mUsed0: 3406  mUsed1: 3342  -64
r: 6   repool-mUsed0: 3342  mUsed1: 3278  -64
r: 7   repool-mUsed0: 3278  mUsed1: 3150  -128
r: 6   repool-mUsed0: 3150  mUsed1: 3086  -64
r: 6   repool-mUsed0: 3086  mUsed1: 3022  -64
r: 6   repool-mUsed0: 3022  mUsed1: 2958  -64
r: 6   repool-mUsed0: 2958  mUsed1: 2894  -64
r: 6   repool-mUsed0: 2894  mUsed1: 2830  -64
r: 6   repool-mUsed0: 2830  mUsed1: 2766  -64
r: 6   repool-mUsed0: 2766  mUsed1: 2702  -64
r: 6   repool-mUsed0: 2702  mUsed1: 2638  -64
r: 6   repool-mUsed0: 2638  mUsed1: 2574  -64
r: 6   repool-mUsed0: 2574  mUsed1: 2510  -64
r: 6   repool-mUsed0: 2510  mUsed1: 2446  -64
r: 6   repool-mUsed0: 2446  mUsed1: 2382  -64
r: 7   repool-mUsed0: 2382  mUsed1: 2254  -128
r: 6   repool-mUsed0: 2254  mUsed1: 2190  -64
r: 6   repool-mUsed0: 2190  mUsed1: 2126  -64
r: 6   repool-mUsed0: 2126  mUsed1: 2062  -64
r: 6   repool-mUsed0: 2062  mUsed1: 1998  -64
r: 6   repool-mUsed0: 1998  mUsed1: 1934  -64
r: 6   repool-mUsed0: 1934  mUsed1: 1870  -64
r: 6   repool-mUsed0: 1870  mUsed1: 1806  -64
r: 7   repool-mUsed0: 1806  mUsed1: 1678  -128
r: 6   repool-mUsed0: 1678  mUsed1: 1614  -64

In the line with the asterisks, 144 bytes are added.
2 lines down from there, only 2 bytes are released.
Hence, we gain 142 bytes with each execution.

Note that in Linux, 64 bytes were released at 2 lines down from the asterisks.
Hence we gained 80 bytes with each execution.

@tavmem
Copy link
Collaborator Author

tavmem commented May 26, 2023

In src/km.c the functions unpool and repool always use a length that is a power of 2 (see value of r).
However, in src/0.c length is set at 144, which is not a power of 2.
mUsed is incremented by 144.
The corresponding repool length in Linux is 64 (causing a diff of 80) and in Windows it is 2 (a diff of 142).

Interesting that valgrind only reports 7 bytes definitely lost, using Linux.
The next step is to examine the valgrind results further.

@tavmem
Copy link
Collaborator Author

tavmem commented May 30, 2023

If we make the following 3 simple changes to the source code:

$ git diff
diff --git a/src/0.c b/src/0.c
index 9f5672f..e504f83 100644
--- a/src/0.c
+++ b/src/0.c
@@ -562,6 +562,7 @@ Z K _1m_r(I f,V fixed, V v,V aft,I*b) {   //File descriptor, moving * into mmap,
 }
 
 K _1d(K x,K y) {
+  O("_1d  ");
   I t=x->t;
   if(4==t || -3==t){
     S m=CSK(x); I sm = strlen(m);
diff --git a/src/km.c b/src/km.c
index fd92020..c3d8367 100644
--- a/src/km.c
+++ b/src/km.c
@@ -42,6 +42,7 @@ Z V kalloc(I k,I*r);
 Z V unpool(I r);
 
 V alloc(size_t sz) {
+  if (sz==7)O("sz: %ld\n",sz);
   V r=malloc(sz);if(!r){fputs("out of memory\n",stderr);exit(1);}
   R r; }
 
diff --git a/src/v.c b/src/v.c
index c47fd19..1a733d2 100644
--- a/src/v.c
+++ b/src/v.c
@@ -23,6 +23,7 @@ K itemAtIndex(K a, I i) {   // Return i-th item from any type as K - TODO: oom w
 
 //Glue will be useful when it comes time to implement \d ?
 S glueSS(S c, S d) {
+  O("glueSS  ");
   I x=strlen(c),y=strlen(d);
   S m = alloc(x+y+2); //oom
   sprintf(m,"%s.%s",c,d);
$

we get

$ rlwrap -n ./k
kona      \ for help. \\ to exit.

  "file" 1: ("a"; 4 5); 1: "file";
_1d  glueSS  sz: 7
glueSS  sz: 7

and see that the problematic sequence only happens once ...
but is followed by second call of glueSS when sz=7.

@tavmem
Copy link
Collaborator Author

tavmem commented Jun 1, 2023

The loss of 7 bytes identified by valgrind is due to this code

K _1d(K x,K y) {
  I t=x->t;
  if(4==t || -3==t){
    S m=CSK(x); I sm = strlen(m);
    S e= sm > 1 && '.'==m[sm-2] && *KFX==m[sm-1] ? strdupn(m,sm) : glueSS(m,KFX);
    //TODO: lfop (lower-case l on Windows -- differs from 'L' in manual)
    U(e)
    remove(e);
    R _1d_write(x,y,0,e); //char-vector but not char-atom
    free(e); }

Note that the command free(e); can never executed because the prior line returns the result of _1d_write.

@tavmem tavmem closed this as completed in ac4e4c5 Jun 2, 2023
@tavmem
Copy link
Collaborator Author

tavmem commented Jun 2, 2023 •
edited
Loading

After the commit of June 1, 2023 the valgrind problem (where 7 bytes are definitely lost) is fixed:

$ valgrind --leak-check=full ./k
==56723== Memcheck, a memory error detector
==56723== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==56723== Using Valgrind-3.21.0 and LibVEX; rerun with -h for copyright info
==56723== Command: ./k
==56723== 
kona      \ for help. \\ to exit.

  "file" 1: ("a"; 4 5); 1: "file";
  \\
==56723== 
==56723== HEAP SUMMARY:
==56723==     in use at exit: 173 bytes in 11 blocks
==56723==   total heap usage: 64 allocs, 53 frees, 28,610 bytes allocated
==56723== 
==56723== 17 bytes in 1 blocks are possibly lost in loss record 4 of 9
==56723==    at 0x484182F: malloc (vg_replace_malloc.c:431)
==56723==    by 0x4256F8: alloc (km.c:45)
==56723==    by 0x42965A: sdupI (ks.c:10)
==56723==    by 0x42965A: sp (ks.c:27)
==56723==    by 0x419BBB: kinit (kc.c:165)
==56723==    by 0x4037B1: main (main.c:6)
==56723== 
==56723== 18 bytes in 1 blocks are possibly lost in loss record 5 of 9
==56723==    at 0x484182F: malloc (vg_replace_malloc.c:431)
==56723==    by 0x4256F8: alloc (km.c:45)
==56723==    by 0x42965A: sdupI (ks.c:10)
==56723==    by 0x42965A: sp (ks.c:27)
==56723==    by 0x419BF9: kinit (kc.c:172)
==56723==    by 0x4037B1: main (main.c:6)
==56723== 
==56723== 18 bytes in 1 blocks are possibly lost in loss record 6 of 9
==56723==    at 0x484182F: malloc (vg_replace_malloc.c:431)
==56723==    by 0x4256F8: alloc (km.c:45)
==56723==    by 0x42965A: sdupI (ks.c:10)
==56723==    by 0x42965A: sp (ks.c:27)
==56723==    by 0x419C33: kinit (kc.c:174)
==56723==    by 0x4037B1: main (main.c:6)
==56723== 
==56723== 19 bytes in 1 blocks are possibly lost in loss record 7 of 9
==56723==    at 0x484182F: malloc (vg_replace_malloc.c:431)
==56723==    by 0x4256F8: alloc (km.c:45)
==56723==    by 0x4297B7: sdupI (ks.c:10)
==56723==    by 0x4297B7: sp (ks.c:24)
==56723==    by 0x419BAA: kinit (kc.c:165)
==56723==    by 0x4037B1: main (main.c:6)
==56723== 
==56723== 38 bytes in 1 blocks are possibly lost in loss record 8 of 9
==56723==    at 0x484182F: malloc (vg_replace_malloc.c:431)
==56723==    by 0x4256F8: alloc (km.c:45)
==56723==    by 0x42965A: sdupI (ks.c:10)
==56723==    by 0x42965A: sp (ks.c:27)
==56723==    by 0x40FE9F: _h (r.c:1030)
==56723==    by 0x419CBB: kinit (kc.c:180)
==56723==    by 0x4037B1: main (main.c:6)
==56723== 
==56723== 54 bytes in 3 blocks are possibly lost in loss record 9 of 9
==56723==    at 0x484182F: malloc (vg_replace_malloc.c:431)
==56723==    by 0x4256F8: alloc (km.c:45)
==56723==    by 0x42965A: sdupI (ks.c:10)
==56723==    by 0x42965A: sp (ks.c:27)
==56723==    by 0x419BCF: kinit (kc.c:165)
==56723==    by 0x4037B1: main (main.c:6)
==56723== 
==56723== LEAK SUMMARY:
==56723==    definitely lost: 0 bytes in 0 blocks
==56723==    indirectly lost: 0 bytes in 0 blocks
==56723==      possibly lost: 164 bytes in 8 blocks
==56723==    still reachable: 9 bytes in 3 blocks
==56723==         suppressed: 0 bytes in 0 blocks
==56723== Reachable blocks (those to which a pointer was found) are n$ valgrind --leak-check=full ./k
==56723== Memcheck, a memory error detector
==56723== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==56723== Using Valgrind-3.21.0 and LibVEX; rerun with -h for copyright info
==56723== Command: ./k
==56723==

However, the original statment of this issue also identified problems with how Kona tracks Memory Used.
This has not been fully addressed.
I will reopen this issue to address the Memory Used tracking.

@tavmem tavmem reopened this Jun 2, 2023
@tavmem tavmem mentioned this issue Jul 28, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@tavmem