[x64 arm32] the test smalloom terminated with signal SIGKILL

[sergign60]

the test coreclr/tests/src/GC/Coverage/smalloom
used platforms
Linux signatov 4.4.0-112-generic #3921-Ubuntu SMP Fri Jan 19 11:48:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Linux localhost 3.10.65 dotnet/coreclr#1-Tizen SMP PREEMPT Fri Feb 2 22:29:33 UTC 2018 armv7l GNU/Linux

% ./corerun smalloom.exe
[1] 8137 killed ./corerun smalloom.exe

% gdb -q --args ./corerun smalloom.exe
Reading symbols from ./corerun...done.
gdb>run
Starting program: /share/WORK/overlay.x64/corerun smalloom.exe
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff4d7a700 (LWP 8347)]
[New Thread 0x7ffff436e700 (LWP 8348)]
[New Thread 0x7ffff3b6d700 (LWP 8349)]
[New Thread 0x7ffff2ff3700 (LWP 8350)]
[New Thread 0x7ffff200b700 (LWP 8351)]
[Thread 0x7ffff200b700 (LWP 8351) exited]
[New Thread 0x7ffff200b700 (LWP 8398)]
[Thread 0x7ffff200b700 (LWP 8398) exited]
[New Thread 0x7ffff200b700 (LWP 8399)]
[Thread 0x7ffff200b700 (LWP 8399) exited]
[Thread 0x7ffff2ff3700 (LWP 8350) exited]
[Thread 0x7ffff3b6d700 (LWP 8349) exited]
[Thread 0x7ffff436e700 (LWP 8348) exited]
[Thread 0x7ffff4d7a700 (LWP 8347) exited]

Program terminated with signal SIGKILL, Killed.
The program no longer exists.

the source:

// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

//Regression test for Dev 10 bug 479239: GC hangs on x86 rather than throwing OOM
 
using System;
using System.Runtime;

class TestClass
{
  public static int Main()
  {
        ByteArrayList list = new ByteArrayList();


        try
        {
            while (true)
            {
                list.AddByteArray(84500);
            }
        }
        catch (OutOfMemoryException)
        {
        }

        Console.Write("NodesAllocated: ");
        Console.WriteLine(list.NodeCount);
        return 100;
   }

    class ByteArrayList
    {
        class Node
        {
            byte[] data = null;
            int size = 0;
            public Node next = null;

            public Node(int Size)
            {
              data = new byte[Size];
              size = Size;
            }

        }
       
        Node head;

        public int NodeCount = 0;
        public ByteArrayList()
        {
            head = null;
        }

        public void AddByteArray(int size)
        {
            Node newNode = new Node(size);

            if (head == null)
                head = newNode;
            else
            {
                newNode.next = head;
                head = newNode;
            }
            NodeCount++;
        }
    }

}

No any additional environment variables used

[sergign60]

@dotnet/arm32-contrib

[sergign60]

https://github.com/dotnet/coreclr/issues/15309

[sergign60]

@dotnet/arm32-contrib
The reason for which the process is killed is found. It is the OOM Killer system utility

The OOM Killer represents a Linux kernel component, the protective mechanism designed to solve problems with the shortage of physical memory (OOM — Out of memory). At detection of a lack of memory the OOM Killer "kills" the least important process, sending KILL signal (besides of init and kernel threads). The message about about it appears in /var/log/syslog:

Out of memory: Kill process **** (****) score **** or sacrifice child

In my case it was the following message, (I run corerun from gdb):

Mar 7 11:54:10 signatov kernel: [259563.476003] Out of memory: Kill process 31490 (corerun) score 897 or sacrifice child

I have inserted the debugging print into the function gc:: Alloc in which I output the number of calls of this function, the address of the allocated block of memory and total size of the allocated memory, and I have got the following output:

In gc::Alloc 355982 00007FF84286B0B0 3815693130
In gc::Alloc 355983 00007FF84287FB08 3815777654
In gc::Alloc 355984 00007FF842894560 3815862178
In gc::Alloc 355985 00007FF8428A8FB8 3815946702
[Thread 0x7ffff2ff2700 (LWP 31497) exited]
[Thread 0x7ffff3b6c700 (LWP 31496) exited]
[Thread 0x7ffff436d700 (LWP 31495) exited]
[Thread 0x7ffff7fc2740 (LWP 31490) exited]
 of corerun
Program terminated with signal SIGKILL, Killed.

One of the way of prevention killing the program by the utility of OOM Killer - assigning the value OOM score = -17 to the oom_adj in the corresponding catalog (/proc/PID/oom_adj)

At the following run of corerun in gdb I have stated this value for the corresponding corerun process

From the following output it is visible that not the corerun process, but the gdb process has been killed

In gc::Alloc 371121 00007FF80E5AC860 779889542
In gc::Alloc 371122 00007FF80E5C12B8 779974066
In gc::Alloc 371123 00007FF80E5D5D10 780058590
In gc::Alloc 371124 00007FF80E5EA768 780143114
In gc::Alloc 371125 00007FF80E5FF1C0 780227638
In gc::Alloc 371126 00007FF80E613C18 780312162
In gc::Alloc 371127 00007FF80E628670 780396686
[Thread 0x7ffff1806700 (LWP 31799) exited]
[1]    31488 killed     gdb -q --args ./corerun smalloom.exe

Also some other user processes have been killed:

Mar  7 12:04:17 signatov kernel: [260169.380482] Out of memory: Kill process 22086 (chrome) score 304 or sacrifice child
Mar  7 12:04:18 signatov kernel: [260171.562143] Out of memory: Kill process 19728 (chrome) score 302 or sacrifice child
Mar  7 12:04:19 signatov kernel: [260172.490017] Out of memory: Kill process 18117 (chrome) score 301 or sacrifice child
Mar  7 12:04:20 signatov kernel: [260173.068198] Out of memory: Kill process 18166 (chrome) score 301 or sacrifice child
Mar  7 12:04:22 signatov kernel: [260174.937280] Out of memory: Kill process 1``` gdb```8149 (chrome) score 301 or sacrifice child
Mar  7 12:04:23 signatov kernel: [260175.812934] Out of memory: Kill process 18187 (chrome) score 301 or sacrifice child
Mar  7 12:04:24 signatov kernel: [260176.989037] Out of memory: Kill process 17790 (chrome) score 201 or sacrifice child
Mar  7 12:04:25 signatov kernel: [260177.570681] Out of memory: Kill process 17790 (chrome) score 201 or sacrifice child
Mar  7 12:04:26 signatov kernel: [260178.974076] Out of memory: Kill process 21031 (emacs) score 6 or sacrifice child
Mar  7 12:04:27 signatov kernel: [260179.842953] Out of memory: Kill process 21031 (emacs) score 6 or sacrifice child
Mar  7 12:04:29 signatov kernel: [260182.524123] Out of memory: Kill process 1972 (java) score 6 or sacrifice child
Mar  7 12:04:31 signatov kernel: [260184.3394maintenance68] Out of memory: Kill process 17714 (chrome) score 4 or sacrifice child
Mar  7 12:04:32 signatov kernel: [260184.685242] Out of memory: Kill process 17714 (chrome) score 4 or sacrifice child
Mar  7 12:04:32 signatov kernel: [260184.732448] Out of memory: Kill process 17714 (chrome) score 4 or sacrifice child
Mar  7 12:04:32 signatov kernel: [260185.231718] Out of memory: Kill process 17714 (chrome) score 4 or sacrifice child
Mar  7 12:04:34 signatov kernel: [260187.232724] Out of memory: Kill process 1242 (mysqld) score 4 or sacrifice child
Mar  7 12:04:34 signatov kernel: [260187.269499] Out of memory: Kill process 4194 (gnome-software) score 3 or sacrifice child
Mar  7 12:04:44 signatov kernel: [260197.222841] Out of memory: Kill process 31818 (mysqld) score 3 or sacrifice child
Mar  7 12:04:44 signatov kernel: [260197.224670] Out of memory: Kill process 31488 (gdb) score 2 or sacrifice child

I don't think that such behavior is acceptable

I see a solution to understand system parameters which are used by the OOM Killer for assessment of expediency of maintenance of processes, to read these parameters in corerun during the next memory allocation and issuing Out Of Memory when their critical values are achieved before the OOM Killer kills the corerun process.

Any other opinions are welcomed.

[alpencolt]

It's old Linux issue related with oom. So it's why Android has own LMK.
I've found some approaches to solve it, but they aren't easy:
https://elinux.org/Memory_Management#Enhanced_Out-Of-Memory_handling

[RussKeldorph]

I'm incline to call this a test bug. I believe OOM killer is a known problem.

@jkotas @janvorli feel free to fix triage if you disagree.

[sergign60]

@RussKeldorph I guess that the problem with this test can be fixed by adding coreclr option MaxHeapSize. With this option coreclr allocator'll issue "OutOfMemory" exception when this size is exceeded before the system memory is exhausted and OOM killer terminates the process

[sergign60]

@dotnet/arm32-contrib @RussKeldorph @janvorli Ok, the last attempt :) Let's mark this test as not applicable for Linux and Tizen because of OOM Killer. I quite agree that this test doesn't make big practical sense, performance and simplicity in this case are more important

[janvorli]

@sergign60 that makes sense to me.

[sergign60]

@RussKeldorph How can I do this tests not applicable for Linux? I didn't find any hints in *csprojs files. Thanks in advance.

[RussKeldorph]

@sergign60 If the test is failing but really ought to pass, you add it to: https://github.com/dotnet/coreclr/blob/master/tests/testsFailingOutsideWindows.txt

If the test is fundamentally incompatible with Unix, add it to:
https://github.com/dotnet/coreclr/blob/master/tests/testsUnsupportedOutsideWindows.txt

Sounds like the "Unsupported" is more appropriate here. It's difficult to disable for Linux only, so this will disable for others like macOS as well, but that's probably not a big deal. FYI, I hope we finally get to fix #5808 soon after Core 2.1 goes out so these terrible files go away, but for now it's all we've got.