Default stack size Mono vs CoreCLR

[tmds]

I debugged a crash of vmr which occurred while building fsharp repo on s390x. Looking at the crash, the stack had overflowed.

I had not seen such an overflow on x64/arm64 (using CoreCLR). Looking closer at the implementation, with CoreCLR, the stack size follows the Linux configured limit (ulimit -s, typically 8MB), while for Mono it is hard-coded to 2MB (on 64-bit, and 1MB on 32-bit):

runtime/src/mono/mono/utils/mono-threads-posix.c

Line 77 in f7ccc3c

set_stack_size = (SIZEOF_VOID_P / 4) * 1024 * 1024;

When I changed the hard-coded size to 8MB on Mono, the stack overflow crash no longer occurred.

This issue is for changing the stack size on Mono so it matches with what CoreCLR does (in a similar environment).

cc @jkotas @akoeplinger @omajid

[MihuBot]

I'm a bot. Here is a possible duplicate issue (I may be wrong):

• [mono] Assertion at sgen-stw.c:77, condition not met #76805

[tmds]

I've created a related issue in fsharp repo to better understand the F# compiler stack usage: dotnet/fsharp#19285.

[tmds]

I'm a bot. Here is a possible duplicate issue (I may be wrong):. #76805

This is a different issue. It fixes stack size detection issue for the main thread with musl libc.
It was detected also when compiling fsharp repo on s390x.
The fsharp compiler being a heavy user of the stack with s390x large stack frames make it more susceptible to these issues.

This issue isn't for the main thread. It is for additional threads, like using when using the Thread class and ThreadPool threads.

[hamarb123]

By default, the main thread stack size is whatever the OS gives (8MB usually on unix, but can be configured via ulimit), or in the case of Windows, 4MB (64-bit) / 1MB (32-bit), which are from the exe headers that we use - NAOT uses 1.5MB by default now (not before the PR linked below, where it was 1MB always).

For secondary threads, I believe we updated the handling in #98007 to ensure 1.5MB for all non-mono platforms; mono still uses 1MB (32-bit) / 2MB (64-bit) iirc (this has existed for a long time); this is up from previously where 512kB was being gotten on some unix platforms in some cases (which is particularly bad, since 128kB is reserved for the runtime & 128kB remaining is needed for EnsureSufficientExecutionStack; meaning that only 256kB was usable in the normal way).

If you want to increase stack size, it's probably also important to increase it to at least 1.5MB universally, by ensuring windows apphost-based & dotnet.exe-based get 1.5MB by default on 32-bit (which iirc was considered a breaking change, which is why I didn't touch it), and then ofc up the mono secondary thread amount to 1.5MB. Or if we want to increase it even further than 1.5MB, there would be additional places to update the handling of beyond that too.

[tmds]

On glibc we don't ensure the 1.5MB limit. Probably because the (default limit of) 8MB limit hasn't caused any issues.

The 1.5MB is set on Musl linux. The default for Musl is a very small stack size for pthread_create. Setting it to 1.5MB makes it something usable for .NET. Note that it is fixed to 1.5MB. When a user increases the RLIMIT_STACK, unlike with glibc, threads created by .NET (with a default size) don't follow along.

I think it would make sense to make the Musl Linux build behave like the glibc build: use RLIMIT_STACK for newly created threads.

@jkotas can you give some feedback on the proposal to make Mono follow the RLIMIT_STACK (like CoreCLR does for newly created threads (on glibc))? And to let Musl behave the same like glibc by using RLIMIT_STACK for newly created threads (both with Mono and CoreCLR)?

[tmds]

Based on investigating dotnet/fsharp#19286, the F# compiler (for .NET 11) needs 8MB stacks to prevent overflows on s390x and ppc64le.

[hamarb123]

Based on investigating dotnet/fsharp#19286, the F# compiler (for .NET 11) needs 8MB stacks to prevent overflows on s390x and ppc64le.

Is it just 5.3+x less efficient at using stack than x64 / arm64 on coreclr for example? Anyway, my personal opinion is that if we up the stack size to 8MB, we should do it (by default at least) everywhere (or more than just 2 random archs that only 0.1% of devs are using at least).

[tmds]

Is it just 5.3+x less efficient at using stack than x64 / arm64 on coreclr for example?

Because tail call optimization implementations are limited, additional stack frames are needed.

It affects F# more because code is more likely to use a recursive style.

or more than just 2 random archs that only 0.1% of devs are using at least.

When you run (CoreCLR) .NET on x64/arm64 on (usual glibc) Linux the stacks are 8MB.