Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

memory leak in octodash v2.2.0 #2026

Closed
mithkr opened this issue Jul 6, 2021 · 19 comments · Fixed by #2334
Closed

memory leak in octodash v2.2.0 #2026

mithkr opened this issue Jul 6, 2021 · 19 comments · Fixed by #2334
Assignees
Labels
bug Something isn't working

Comments

@mithkr
Copy link

mithkr commented Jul 6, 2021

Describe the bug
System runs out of memory after a while also if idle (no print jobs running).
OOM killer kills octodash, as it has leaked the memory.
[289804.365120] Out of memory: Killed process 509 (octodash) total-vm:817456kB, anon-rss:249080kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:632kB oom_score_adj:0

To Reproduce
Boot Raspberry Pi with auto stared octodash and wait until system runs out of memory.
Monitoring memory usage with top shows memory usage increase of octodash.

Expected behavior
System should not run out of memory.

Screenshots

top - 09:44:16 up 11:16,  2 users,  load average: 0,18, 0,21, 0,15
Tasks: 120 total,   1 running, 119 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1,4 us,  0,8 sy,  0,0 ni, 97,7 id,  0,0 wa,  0,0 hi,  0,1 si,  0,0 st
MiB Mem :    460,7 total,     40,5 free,    203,7 used,    216,6 buff/cache
MiB Swap:    100,0 total,     18,5 free,     81,5 used.    197,8 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                                                     
  509 pi        20   0  608560  80112  31016 S   1,0  17,0   8:06.60 octodash                                                                                                                                                                                    
  401 pi        20   0  410112  75380   7972 S   8,6  16,0  59:32.91 octoprint                                                                                                                                                                                   
  575 pi        20   0  335044  74144  41980 S   0,3  15,7   3:19.44 octodash                                                                                                                                                                                    
  556 pi        20   0  209476  31196  28804 S   0,0   6,6   1:04.19 octodash                                                                                                                                                                                    
  591 pi        20   0  215552  22668  16420 S   0,0   4,8   0:03.32 octodash

top - 13:17:04 up 2 days, 14:48,  2 users,  load average: 0,09, 0,07, 0,07
Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1,6 us,  1,6 sy,  0,0 ni, 96,6 id,  0,2 wa,  0,0 hi,  0,0 si,  0,0 st
MiB Mem :    460,7 total,     25,5 free,    355,8 used,     79,4 buff/cache
MiB Swap:    100,0 total,      0,0 free,    100,0 used.     48,8 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                                                     
  509 pi        20   0  768304 220252  19344 S   1,7  46,7  45:16.04 octodash                                                                                                                                                                                    
  575 pi        20   0  355008  72464  26512 S   0,3  15,4  17:26.14 octodash                                                                                                                                                                                    
  401 pi        20   0  410276  68804   3940 S   8,3  14,6 329:22.30 octoprint                                                                                                                                                                                   
  556 pi        20   0  210500  16100  13344 S   0,0   3,4   6:00.81 octodash                                                                                                                                                                                    
  591 pi        20   0  215040  10256   9280 S   0,0   2,2   0:07.32 octodash                                                                                                                                                                                    
  512 pi        20   0  156208   5556   5428 S   0,0   1,2   0:00.14 octodash                

[289804.363927] python invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[289804.363944] CPU: 2 PID: 491 Comm: python Tainted: G         C        5.10.17-v7+ #1421
[289804.363947] Hardware name: BCM2835
[289804.363951] Backtrace: 
[289804.363975] [<809e477c>] (dump_backtrace) from [<809e4b0c>] (show_stack+0x20/0x24)
[289804.363982]  r7:ffffffff r6:00000000 r5:60000113 r4:80ee5d54
[289804.363990] [<809e4aec>] (show_stack) from [<809e8cb8>] (dump_stack+0xcc/0xf8)
[289804.363997] [<809e8bec>] (dump_stack) from [<809e6fa8>] (dump_header+0x64/0x208)
[289804.364014]  r10:80e05144 r9:00100cca r8:00000000 r7:80c13384 r6:81f6f600 r5:82c22e80
[289804.364019]  r4:89579d50 r3:df8c3ffb
[289804.364029] [<809e6f44>] (dump_header) from [<802b18b0>] (oom_kill_process+0x1b4/0x1c0)
[289804.364035]  r7:80c13384 r6:89579d50 r5:82c23400 r4:82c22e80
[289804.364043] [<802b16fc>] (oom_kill_process) from [<802b2428>] (out_of_memory+0x2b8/0x390)
[289804.364049]  r7:80e083c0 r6:80e05008 r5:82c22e80 r4:89579d50
[289804.364058] [<802b2170>] (out_of_memory) from [<803018dc>] (__alloc_pages_nodemask+0x7ec/0x1184)
[289804.364064]  r7:80e05784 r6:00000000 r5:ffffe000 r4:00400040
[289804.364072] [<803010f0>] (__alloc_pages_nodemask) from [<802abde0>] (pagecache_get_page+0x10c/0x34c)
[289804.364079]  r10:80f316db r9:00000006 r8:80e059e4 r7:898314e8 r6:00100cca r5:00000044
[289804.364083]  r4:17240003
[289804.364090] [<802abcd4>] (pagecache_get_page) from [<802add40>] (filemap_fault+0x638/0xb20)
[289804.364097]  r10:80e05008 r9:89579ec8 r8:00000040 r7:834d76c0 r6:89579ec8 r5:834d76c0
[289804.364101]  r4:898313d8
[289804.364110] [<802ad708>] (filemap_fault) from [<80427a9c>] (ext4_filemap_fault+0x38/0x4c)
[289804.364117]  r10:89579fb0 r9:89579ec8 r8:00000040 r7:00000000 r6:75774000 r5:89579ec8
[289804.364120]  r4:898313c0
[289804.364130] [<80427a64>] (ext4_filemap_fault) from [<802e4f04>] (__do_fault+0x48/0x14c)
[289804.364134]  r5:894200c0 r4:89579ec8
[289804.364142] [<802e4ebc>] (__do_fault) from [<802e9fc0>] (handle_mm_fault+0xae8/0xe30)
[289804.364148]  r7:00000000 r6:75774000 r5:80e05008 r4:00000254
[289804.364156] [<802e94d8>] (handle_mm_fault) from [<809f4258>] (do_page_fault+0x144/0x314)
[289804.364163]  r10:00000254 r9:00000000 r8:827f2e48 r7:827f2e00 r6:80000007 r5:757743b8
[289804.364167]  r4:89579fb0
[289804.364174] [<809f4114>] (do_page_fault) from [<80114080>] (do_PrefetchAbort+0x48/0x9c)
[289804.364181]  r10:0092c308 r9:00000000 r8:89579fb0 r7:757743b8 r6:809f4114 r5:00000007
[289804.364186]  r4:80e0adf8
[289804.364193] [<80114038>] (do_PrefetchAbort) from [<80101024>] (ret_from_exception+0x0/0x1c)
[289804.364198] Exception stack(0x89579fb0 to 0x89579ff8)
[289804.364204] 9fa0:                                     000471bd 00000000 0d0409d9 41119c28
[289804.364210] 9fc0: 740bb098 00000000 00000001 0092d308 00047267 00000000 0092c308 0000000e
[289804.364216] 9fe0: 75786090 75545d90 75774b04 757743b8 20000010 ffffffff
[289804.364222]  r8:10c5387d r7:10c5383d r6:ffffffff r5:20000010 r4:757743b8
[289804.364226] Mem-Info:
[289804.364239] active_anon:46277 inactive_anon:55427 isolated_anon:0
                 active_file:48 inactive_file:373 isolated_file:0
                 unevictable:4 dirty:0 writeback:0
                 slab_reclaimable:2789 slab_unreclaimable:4097
                 mapped:332 shmem:1861 pagetables:1468 bounce:0
                 free:4097 free_pcp:55 free_cma:24
[289804.364250] Node 0 active_anon:185108kB inactive_anon:221708kB active_file:192kB inactive_file:1492kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:1328kB dirty:0kB writeback:0kB shmem:7444kB writeback_tmp:0kB kernel_stack:1920kB all_unreclaimable? yes
[289804.364263] DMA free:16388kB min:16384kB low:20480kB high:24576kB reserved_highatomic:0KB active_anon:185108kB inactive_anon:221708kB active_file:0kB inactive_file:1404kB unevictable:16kB writepending:0kB present:491520kB managed:471756kB mlocked:16kB pagetables:5872kB bounce:0kB free_pcp:220kB local_pcp:0kB free_cma:96kB
[289804.364268] lowmem_reserve[]: 0 0 0
[289804.364290] DMA: 598*4kB (UMEC) 513*8kB (UME) 231*16kB (UME) 25*32kB (UE) 36*64kB (UME) 4*128kB (UM) 2*256kB (UM) 1*512kB (M) 0*1024kB 1*2048kB (M) 0*4096kB = 16880kB
[289804.364389] 2418 total pagecache pages
[289804.364396] 72 pages in swap cache
[289804.364402] Swap cache stats: add 32148, delete 32076, find 48186/50004
[289804.364408] Free swap  = 0kB
[289804.364413] Total swap = 102396kB
[289804.364419] 122880 pages RAM
[289804.364424] 0 pages HighMem/MovableOnly
[289804.364429] 4941 pages reserved
[289804.364434] 16384 pages cma reserved
[289804.364439] Tasks state (memory values in pages):
[289804.364444] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[289804.364466] [    118]     0   118     6790      344    67584      121             0 systemd-journal
[289804.364476] [    144]     0   144     4578       11    20480      203         -1000 systemd-udevd
[289804.364487] [    285]   100   285     5604       37    26624      127             0 systemd-timesyn
[289804.364496] [    336]     0   336     6378      121    20480       92             0 rsyslogd
[289804.364504] [    342]   104   342     1639       71    16384       49          -900 dbus-daemon
[289804.364514] [    346]     0   346     7170       23    20480       12             0 rngd
[289804.364522] [    354]   108   354     1475       76    14336       29             0 avahi-daemon
[289804.364531] [    355] 65534   355     1080        7    14336       37             0 thd
[289804.364540] [    356]     0   356     2686       12    20480       89             0 wpa_supplicant
[289804.364569] [    361]     0   361     1987       14    18432       31             0 cron
[289804.364585] [    367]     0   367     3261       45    20480      122             0 systemd-logind
[289804.364598] [    376]   108   376     1442       13    12288       50             0 avahi-daemon
[289804.364607] [    401]  1000   401   102569    16506   210944     1581             0 octoprint
[289804.364616] [    404]     0   404      708       74    10240       16             0 dhcpcd
[289804.364625] [    430]  1000   430    13379     2115    38912     1188             0 python
[289804.364635] [    437]     0   437     1405        1    12288      111             0 login
[289804.364644] [    438]     0   438     2682       29    18432      117         -1000 sshd
[289804.364653] [    441]     0   441     2777       69    20480      101             0 wpa_supplicant
[289804.364663] [    442]     0   442     3681        6    20480     1174             0 haproxy
[289804.364672] [    449]   109   449     3813       83    20480     1234             0 haproxy
[289804.364681] [    458]  1000   458     3651       53    24576      187             0 systemd
[289804.364691] [    459]  1000   459     4271       56    26624      428             0 (sd-pam)
[289804.364700] [    471]  1000   471     2123        2    18432      228             0 bash
[289804.364710] [    489]  1000   489      968        0    14336       53             0 xinit
[289804.364719] [    490]  1000   490    49541     1829   139264     2261             0 Xorg
[289804.364731] [    504]  1000   504      485        0    12288       18             0 .xinitrc
[289804.364743] [    508]  1000   508     3134       23    22528      150             0 ratpoison
[289804.364753] [    509]  1000   509   204364    62270   647168     8303             0 octodash
[289804.364762] [    512]  1000   512    39052       31   131072     1150             0 octodash
[289804.364771] [    515]  1000   515    39052        0   133120     1180             0 octodash
[289804.364781] [    517]  1000   517    39052        0    73728     1179             0 octodash
[289804.364793] [    556]  1000   556    52625      764   167936      881             0 octodash
[289804.364803] [    572]  1000   572     1604        0    16384       72             0 dbus-daemon
[289804.364813] [    575]  1000   575    89123    11589   372736     1426             0 octodash
[289804.364822] [    591]  1000   591    53824      254   165888     3141           200 octodash
[289804.364831] [    593]  1000   593    42771       29   108544     1375           200 octodash
[289804.364840] [    915]     0   915     3060      178    20480       19             0 sshd
[289804.364852] [    921]  1000   921     3216      354    20480       23             0 sshd
[289804.364861] [    922]  1000   922     2122      166    16384       66             0 bash
[289804.364870] [    941]     0   941     2561       83    18432       41             0 sudo
[289804.364879] [    945]     0   945     2156      278    16384        4             0 bash
[289804.364889] [  27015]     0 27015     1966      156    14336        9             0 man
[289804.364897] [  27025]     0 27025     1689       35    14336        5             0 pager
[289804.364909] [  23390]     0 23390      485       25    10240        0             0 apt.systemd.dai
[289804.364918] [  23400]     0 23400      485       25     8192        0             0 apt.systemd.dai
[289804.364927] [  23411]     0 23411    13027     2026    61440        0             0 apt-get
[289804.364936] [  23429]     0 23429     2367       77    20480        5             0 cron
[289804.364944] [  23434]     0 23434      485       17     8192        0             0 sh
[289804.364953] [  23436]     0 23436      465       20    12288        0             0 run-parts
[289804.364962] [  23439]     0 23439      485       18    12288        0             0 dpkg
[289804.364971] [  23444]     0 23444      485       24    10240        0             0 savelog
[289804.364982] [  23497]     0 23497      747       26    10240        0             0 mv
[289804.364990] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=octodash,pid=509,uid=1000
[289804.365120] Out of memory: Killed process 509 (octodash) total-vm:817456kB, anon-rss:249080kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:632kB oom_score_adj:0
[289804.469221] oom_reaper: reaped process 509 (octodash), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

General Information:

  • Hardware: Raspberry Pi 3 A+ and DFR0550 (5'' 800x480 TFT Raspberry Pi DSI Touchscreen)
  • OS Info: OctoPi 0.17.0
  • OctoDash Version v2.2.0
  • OctoPrint Version 1.6.1

Additional context

@mithkr mithkr added the bug Something isn't working label Jul 6, 2021
@UnchartedBull
Copy link
Owner

Oh damn. I'm going to investigate this as soon as possible and release a fixed version. I'll let you know once I was able to replicate this.

@UnchartedBull
Copy link
Owner

Ok I just had a look at this and did a bit of testing. I wasn't able to reproduce this on the dev system. The JS heap size always stayed in the 5-8MB area and also the memory profile doesn't show any obvious signs of a memory leak:

Screenshot 2021-07-08 at 18 36 07

Could yo maybe elaborate on what you did to cause OctoDash to fail? Was it just idleing? According to your logs OctoDash is filling up around half of the memory in about 45 minutes, so it would crash after around 70-80 minutes?

@mithkr
Copy link
Author

mithkr commented Jul 9, 2021

The problem is on my system seen immediately after reboot. System is fresh rebooted and octoprint and octodash is started. Octoprint is connected to 3d printer and only temperature sensor values are exchange between printer, octoprint and octodash. Nothing else is started. I made new top measurement after reboot and there are two processes (pid: 510, 576) of octodash are continuous increasing in the RES memory.
I also add at the end the list of the octodash processes with their args and attached strace files of this two processes (strace -o octodash-510 -tt -ff -p 510; strace -o octodash-576 -tt -ff -p 576)
octodash-strace.zip

top - 19:06:10 up 3 min,  2 users,  load average: 1,92, 1,47, 0,63
Tasks: 131 total,   1 running, 130 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1,6 us,  1,2 sy,  0,0 ni, 97,2 id,  0,1 wa,  0,0 hi,  0,0 si,  0,0 st
MiB Mem :    460,7 total,     38,3 free,    197,6 used,    224,7 buff/cache
MiB Swap:    100,0 total,     50,5 free,     49,5 used.    206,7 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                                                     
  431 pi        20   0  410264  76412   8008 S   8,3  16,2   1:18.95 octoprint                                                                                                                                                                                   
  510 pi        20   0  580912  43900   7692 S   0,0   9,3   0:05.87 octodash                                                                                                                                                                                    
  576 pi        20   0  309048  42876  18456 S   0,7   9,1   0:11.68 octodash                                                                                                                                                                                    
  592 pi        20   0  215040  18748   8544 S   0,0   4,0   0:02.19 octodash                                                                                                                                                                                    
  557 pi        20   0  209476  10924   5056 S   0,3   2,3   0:00.59 octodash 

  top - 19:08:34 up 6 min,  2 users,  load average: 0,27, 0,93, 0,54
Tasks: 125 total,   1 running, 124 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2,2 us,  0,7 sy,  0,0 ni, 97,0 id,  0,1 wa,  0,0 hi,  0,0 si,  0,0 st
MiB Mem :    460,7 total,     33,2 free,    198,8 used,    228,7 buff/cache
MiB Swap:    100,0 total,     50,5 free,     49,5 used.    205,4 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                                                     
  431 pi        20   0  410264  76412   8008 S   8,3  16,2   1:31.23 octoprint                                                                                                                                                                                   
  510 pi        20   0  580912  52180  15808 S   0,3  11,1   0:06.36 octodash                                                                                                                                                                                    
  576 pi        20   0  309048  43180  18840 S   0,3   9,2   0:12.53 octodash                                                                                                                                                                                    
  592 pi        20   0  215040  18748   8544 S   0,0   4,0   0:02.19 octodash                                                                                                                                                                                    
  557 pi        20   0  209476  10924   5056 S   0,3   2,3   0:00.78 octodash

  top - 19:11:36 up 9 min,  2 users,  load average: 0,08, 0,53, 0,45
Tasks: 124 total,   1 running, 123 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2,0 us,  0,7 sy,  0,0 ni, 97,3 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
MiB Mem :    460,7 total,     31,3 free,    199,2 used,    230,2 buff/cache
MiB Swap:    100,0 total,     50,5 free,     49,5 used.    205,0 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                                                     
  431 pi        20   0  410264  76412   8008 S   8,6  16,2   1:46.65 octoprint                                                                                                                                                                                   
  510 pi        20   0  580912  52180  15808 S   0,3  11,1   0:06.84 octodash                                                                                                                                                                                    
  576 pi        20   0  311096  44420  19012 S   0,7   9,4   0:13.63 octodash                                                                                                                                                                                    
  592 pi        20   0  215040  18748   8544 S   0,0   4,0   0:02.19 octodash                                                                                                                                                                                    
  557 pi        20   0  209476  10924   5056 S   0,3   2,3   0:01.03 octodash

  top - 19:24:26 up 22 min,  2 users,  load average: 0,26, 0,20, 0,27
Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3,7 us,  1,1 sy,  0,0 ni, 95,2 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
MiB Mem :    460,7 total,     30,1 free,    192,6 used,    238,0 buff/cache
MiB Swap:    100,0 total,     50,5 free,     49,5 used.    211,6 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                                                     
  431 pi        20   0  410264  76416   8008 S   9,3  16,2   2:51.60 octoprint                                                                                                                                                                                   
  510 pi        20   0  580912  53892  17468 S   0,3  11,4   0:08.72 octodash                                                                                                                                                                                    
  576 pi        20   0  311096  52464  28260 S   0,7  11,1   0:18.08 octodash                                                                                                                                                                                    
  592 pi        20   0  215040  18748   8544 S   0,0   4,0   0:02.19 octodash                                                                                                                                                                                    
  557 pi        20   0  209476  11380   5512 S   0,0   2,4   0:02.07 octodash

root@octopi:~#  ps aux | grep octodash
pi         510  0.6 11.4 580912 53892 tty1     Sl   19:02   0:08 /opt/OctoDash/octodash
pi         515  0.0  0.9 156208  4316 tty1     S    19:02   0:00 /opt/OctoDash/octodash --type=zygote --no-zygote-sandbox
pi         516  0.0  0.7 156208  3332 tty1     S    19:02   0:00 /opt/OctoDash/octodash --type=zygote
pi         518  0.0  0.0 156208   336 tty1     S    19:02   0:00 /opt/OctoDash/octodash --type=zygote
pi         557  0.1  2.4 209476 11380 tty1     Sl   19:02   0:02 /opt/OctoDash/octodash --type=utility --utility-sub-type=network.mojom.NetworkService --field-trial-handle=8698878411638238832,2576688415711229123,131072 --enable-features=WebComponentsV0Enabled --disable-features=CertVerifierService,CookiesWithoutSameSiteMustBeSecure,SameSiteByDefaultCookies,SpareRendererForSitePerProcess --lang=de --service-sandbox-type=network --standard-schemes=app --secure-schemes --bypasscsp-schemes --cors-schemes --fetch-schemes --service-worker-schemes --streaming-schemes --shared-files=v8_context_snapshot_data:100
pi         576  1.3 11.1 311096 52728 tty1     Sl   19:02   0:18 /opt/OctoDash/octodash --type=renderer --touch-events=enabled --field-trial-handle=8698878411638238832,2576688415711229123,131072 --enable-features=WebComponentsV0Enabled --disable-features=CertVerifierService,CookiesWithoutSameSiteMustBeSecure,SameSiteByDefaultCookies,SpareRendererForSitePerProcess --lang=de --standard-schemes=app --secure-schemes --bypasscsp-schemes --cors-schemes --fetch-schemes --service-worker-schemes --streaming-schemes --app-path=/opt/OctoDash/resources/app.asar --no-sandbox --no-zygote --num-raster-threads=2 --enable-main-frame-before-activation --renderer-client-id=4 --no-v8-untrusted-code-mitigations --shared-files=v8_context_snapshot_data:100
pi         592  0.1  3.9 215040 18748 tty1     Sl   19:02   0:02 /opt/OctoDash/octodash --type=gpu-process --field-trial-handle=8698878411638238832,2576688415711229123,131072 --enable-features=WebComponentsV0Enabled --disable-features=CertVerifierService,CookiesWithoutSameSiteMustBeSecure,SameSiteByDefaultCookies,SpareRendererForSitePerProcess --gpu-preferences=UAAAAAAAAAAgAAAQAAAAAAAAAAAAAAAAAABgAAAAAAAwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgAAAAAAAAACAAAAAAAAAAAAAAAAAAAAAIAAAAAAAAAAgAAAAAAAAACAAAAAAAAAA= --use-gl=swiftshader-webgl --shared-files
pi         594  0.0  0.6 171084  3228 tty1     S    19:02   0:00 /opt/OctoDash/octodash --type=broker
root      1631  0.0  0.1   7360   504 pts/1    S+   19:24   0:00 grep octodash

@UnchartedBull
Copy link
Owner

Thanks for the additional info! I'll do some further investigation and probably provide a dev build here to test.

@mithkr
Copy link
Author

mithkr commented Jul 22, 2021

I have made a further investigation and looked how octodash behave if printer is disconnected from octoprint.
If printer is disconnected, than octodash show sleep screen and memory leak is not seen. So memory leak maybe somehow related to the printer status update messages (e.g. temp sensors) between octodash and octoprint.
Here the top measurement with disconnected printer:

top - 22:43:06 up 1 day, 12:27,  2 users,  load average: 0,04, 0,06, 0,05
Tasks:   8 total,   0 running,   8 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1,4 us,  0,6 sy,  0,0 ni, 97,9 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
MiB Mem :    460,7 total,     44,6 free,    210,3 used,    205,8 buff/cache
MiB Swap:    100,0 total,     66,5 free,     33,5 used.    191,0 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  580 pi        20   0  339864  76724  36328 S   0,3  16,3   6:59.93 /opt/OctoDash/octodash --type=renderer --touch-events=enabled --field-trial-handle=12013433723393411160,11906847453383645420,131072 --disable-features=CookiesWithoutSameSiteMustBeSecure,+
  510 pi        20   0  573872  61488  31660 S   0,2  13,0   4:22.77 /opt/OctoDash/octodash
  591 pi        20   0  217524  28016  14388 S   0,0   5,9   0:02.39 /opt/OctoDash/octodash --type=gpu-process --field-trial-handle=12013433723393411160,11906847453383645420,131072 --disable-features=CookiesWithoutSameSiteMustBeSecure,SameSiteByDefaultCoo+
  558 pi        20   0  209580  17900  12248 S   0,1   3,8   2:15.92 /opt/OctoDash/octodash --type=utility --utility-sub-type=network.mojom.NetworkService --field-trial-handle=12013433723393411160,11906847453383645420,131072 --disable-features=CookiesWith+
  515 pi        20   0  158712   9484   4640 S   0,0   2,0   0:00.13 /opt/OctoDash/octodash --type=zygote
  514 pi        20   0  158712   9392   4576 S   0,0   2,0   0:00.14 /opt/OctoDash/octodash --type=zygote --no-zygote-sandbox
  598 pi        20   0  173568   6992   1756 S   0,0   1,5   0:00.00 /opt/OctoDash/octodash --type=broker
  517 pi        20   0  158712   4984    144 S   0,0   1,1   0:00.00 /opt/OctoDash/octodash --type=zygote

top - 01:43:08 up 2 days, 15:27,  2 users,  load average: 0,00, 0,03, 0,05
Tasks:   8 total,   0 running,   8 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1,4 us,  0,6 sy,  0,0 ni, 98,0 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
MiB Mem :    460,7 total,     49,7 free,    212,6 used,    198,3 buff/cache
MiB Swap:    100,0 total,     63,5 free,     36,5 used.    185,5 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  580 pi        20   0  339864  79320  36344 S   0,3  16,8  11:59.53 /opt/OctoDash/octodash --type=renderer --touch-events=enabled --field-trial-handle=12013433723393411160,11906847453383645420,131072 --disable-features=CookiesWithoutSameSiteMustBeSecure,+
  510 pi        20   0  573872  59028  31652 S   0,2  12,5   7:33.04 /opt/OctoDash/octodash
  591 pi        20   0  217524  28012  14384 S   0,0   5,9   0:02.67 /opt/OctoDash/octodash --type=gpu-process --field-trial-handle=12013433723393411160,11906847453383645420,131072 --disable-features=CookiesWithoutSameSiteMustBeSecure,SameSiteByDefaultCoo+
  558 pi        20   0  225972  18856  13016 S   0,1   4,0   3:56.48 /opt/OctoDash/octodash --type=utility --utility-sub-type=network.mojom.NetworkService --field-trial-handle=12013433723393411160,11906847453383645420,131072 --disable-features=CookiesWith+
  515 pi        20   0  158712   9480   4636 S   0,0   2,0   0:00.13 /opt/OctoDash/octodash --type=zygote
  514 pi        20   0  158712   9388   4572 S   0,0   2,0   0:00.14 /opt/OctoDash/octodash --type=zygote --no-zygote-sandbox
  598 pi        20   0  173568   6992   1756 S   0,0   1,5   0:00.00 /opt/OctoDash/octodash --type=broker
  517 pi        20   0  158712   4984    144 S   0,0   1,1   0:00.00 /opt/OctoDash/octodash --type=zygote

top - 15:43:09 up 3 days,  5:27,  2 users,  load average: 0,02, 0,08, 0,03
Tasks:   8 total,   0 running,   8 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1,4 us,  0,6 sy,  0,0 ni, 98,0 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
MiB Mem :    460,7 total,     49,5 free,    212,1 used,    199,1 buff/cache
MiB Swap:    100,0 total,     63,5 free,     36,5 used.    186,0 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  580 pi        20   0  339864  79616  36340 S   0,3  16,9  14:35.27 /opt/OctoDash/octodash --type=renderer --touch-events=enabled --field-trial-handle=12013433723393411160,11906847453383645420,131072 --disable-features=CookiesWithoutSameSiteMustBeSecure,+
  510 pi        20   0  573872  58928  31612 S   0,2  12,5   9:11.75 /opt/OctoDash/octodash
  591 pi        20   0  217524  28012  14384 S   0,0   5,9   0:02.82 /opt/OctoDash/octodash --type=gpu-process --field-trial-handle=12013433723393411160,11906847453383645420,131072 --disable-features=CookiesWithoutSameSiteMustBeSecure,SameSiteByDefaultCoo+
  558 pi        20   0  225972  18972  13016 S   0,1   4,0   4:48.81 /opt/OctoDash/octodash --type=utility --utility-sub-type=network.mojom.NetworkService --field-trial-handle=12013433723393411160,11906847453383645420,131072 --disable-features=CookiesWith+
  515 pi        20   0  158712   9480   4636 S   0,0   2,0   0:00.13 /opt/OctoDash/octodash --type=zygote
  514 pi        20   0  158712   9388   4572 S   0,0   2,0   0:00.14 /opt/OctoDash/octodash --type=zygote --no-zygote-sandbox
  598 pi        20   0  173568   6992   1756 S   0,0   1,5   0:00.00 /opt/OctoDash/octodash --type=broker
  517 pi        20   0  158712   4984    144 S   0,0   1,1   0:00.00 /opt/OctoDash/octodash --type=zygote

@UnchartedBull
Copy link
Owner

Interesting find. I'm going to post a dev build here sometime tomorrow, please try that one as well and report whether the memory leak is gone. Thanks!

I also don't know what you changed with your top command in your latest comment, but having the arguments for each process actually is super useful, so please include them for further tests if possible! :)

@UnchartedBull
Copy link
Owner

https://drive.google.com/file/d/1M1_Ksvj49Naxz-Yijxj2pmalApb07Bl0/view?usp=sharing please try this version and report whether this fixes the issue.

Also i noticed, that your raspberry only has 512MB of memory, which model are you running this on?

@mithkr
Copy link
Author

mithkr commented Jul 26, 2021

Thanks, I will test it.

I have a raspberry pi 3A+

@mithkr
Copy link
Author

mithkr commented Jul 28, 2021

I have run your test version for 40 hour and capture memory usage every 10 minutes with following command:
top -o RES -p $(pidof -f %d, octodash)0 -b -c -d 600 | tee top.log
unluckily mem leak is still there. See attached top.log and generated graphs from that log file. top_plot.txt contains the script I used to generate the graphs from the top.log file.
top.log
Avail_mem
RES_mem
VIRT_mem
top_plot.txt

@UnchartedBull
Copy link
Owner

Could you try this version: https://drive.google.com/file/d/1uTu9q4XjojOWRfrwU4-win52v4AuwVY0/view?usp=sharing and report whether that fixed the issue?

@mithkr
Copy link
Author

mithkr commented Aug 4, 2021

I have tested. I looks like that after some time memory increase stops now, see charts. I will further test after my vacation in 3 weeks.
VIRT_mem
RES_mem
Avail_mem
top.log

@UnchartedBull
Copy link
Owner

Ok that looks like we're finally making progress! Thanks for the all graphs. It's really weird that it's allocating memory for a while and then just stops. Might need to do some further digging.

Enjoy your vacation!

@meganssmith
Copy link

I have also been sealing with this. Any idea when the fix will go main stream?

@UnchartedBull
Copy link
Owner

Once @mithkr has confirmed that this has been fixed for him :) But for now he should enjoy his vacation :) Will definitely be included in the next release though.

@mithkr
Copy link
Author

mithkr commented Sep 27, 2021

I have now started a long time recording of top, to see memory consumption for 1 week. I will then upload the result.

@UnchartedBull
Copy link
Owner

Thanks!

@mithkr
Copy link
Author

mithkr commented Oct 5, 2021

I have now started a long time recording of top, to see memory consumption for 1 week. I will then upload the result.

@mithkr
Copy link
Author

mithkr commented Oct 5, 2021

Here are the test results. It looks good. Measurement period is about 9 days. No OOM happened. I only see a very little increase in the virtual memory chart (light blue). Due to the jumps in the RES memory chart, due to enabled swap, it is not possible to proof this trend there. I will repeat measurement with disabled swap. But it can be concluded, that the fixes you already provided helped a lot.
top.log
RES_mem
VIRT_mem
Avail_mem

@UnchartedBull
Copy link
Owner

Ok great thanks for the update! If anything weird shows up again feel free to re-open this issue :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants