valgrind BSD

[irevoire]

Hello,
I’m trying to profile a simple program on FreeBSD 11.2 with rustc 1.39.0 and valgrind 3.10.1.

I always get the following error:

thread '<unnamed>' panicked at 'failed to allocate a guard page', src/libstd/sys/unix/thread.rs:336:17
stack backtrace:
   0:           0x116af8 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hf24f776311cdf529
   1:           0x12eed0 - core::fmt::write::h01edf6dd68a42c9c
   2:           0x115174 - std::io::stdio::_print::hf62f3360234422f1
   3:           0x11870d - rust_oom
   4:           0x1183e3 - rust_oom
   5:           0x118d57 - std::panicking::rust_panic_with_hook::h9a662f58cf3f8ffe
   6:           0x118b36 - std::panicking::begin_panic_fmt::hb07c3a07faf6b5b3
   7:           0x119436 - std::rt::lang_start_internal::h7d3aa0ed326f9560
   8:           0x110dc3 -
   9:           0x110c6b -
  10:           0x110a25 -
  11:          0x4023000 - <unknown>
fatal runtime error: failed to initiate panic, error 94376704
error: valgrind command failed

[Mark-Simulacrum]

This is at least plausibly a valgrind bug? Could you try with a more recent version (I have 3.15 locally, at least)?

In any case I suspect that since it works outside of valgrind (I assume, given you didn't mention otherwise) that this would be a bug inside valgrind.

[irevoire]

Yes it does works outside of valgrind, and valgrind is also working for other C program.

I don’t know how to get a more recent version valgrind, in the pre compiled package and in the port of FreeBSD the latest version of valgrind is 3.10.1.

And when I try to compile valgrind from source it does not work because at some point my cpu is not supported (amd64).

If you are also running under FreeBSD could you send me your valgrind binary?

[Mark-Simulacrum]

I am not running under FreeBSD. I suspect this bug is unlikely to make progress as-is; it sounds like it's probably a valgrind bug (missing emulation or lack of support for guard pages, something along those lines).

[ranma42]

Would it be possible to disable guard pages to check if it makes a difference?
(with a compile-time option or an environment variable?)

[Mark-Simulacrum]

I think no (they're sort of inherently protecting you, i.e., a safety feature). But I'm not sure. You could plausibly comment out the stack guard support in libstd, I guess, and recompile -- https://github.com/rust-lang/rust/blob/master/src/libstd/sys/unix/stack_overflow.rs

[nagisa]

Guard pages are a prerequisite for ensuring soundness in case stack overflow occurs. C works, because it doesn’t do anything along the lines for you.

That being said, there seems to be more stuff going wrong here than just inability to map a page:

fatal runtime error: failed to initiate panic, error 94376704

Either way, this is more attributable to valgrind rather than Rust.

[asomers]

I opened a bug with Valgrind. We'll see what they say.
https://bugs.kde.org/show_bug.cgi?id=432440

[paulfloyd]

FreeBSD Valgrind maintainer here.

I'm not at all familiar with Rust. Is this code creating a stack for the main process or for a secondary thread?

With ktrace I'm seeing a lot of mmap'ing going on - much more than for 'grep'.

From what I understand for C/C++ applications, on startup mmap a MAP_GUARD region above the user stack, similar to what this code is doing but much larger and without needing to use mprotect. Valgrind is leaving space for this mapping, and I don't think that it likes client code mmap'ing into it.

[paulfloyd]

Can anyone comment on my previous question?

Is this happening on the main thread during program startup?
Why isn't rust using a MAP_GUARD region (designed for this purpose, or so it seems to me) rather than mmap'ing then calling mprotect?

[paulfloyd]

More questions:

What is Linux doing differently? Is it also mmap'ing a guard page? And if so, how does it determine the address to use?

My impression on FreeBSD is that the rust runtime is getting the user stack via the "kern.usrstack" sysctl. That's what I see with ktrace, but looking at the rust source I also see calls to pthread_attr_getstack.

The Valgrind code that allocates the user stack is here starting on line 1048. The FreeBSD code does the same, though obviously there are differences in the setup_client_stack function.

I've started doing some tests to see if I can intercept the sysctl for kern.usrstack to return a value that is correct for the guest.

[paulfloyd]

A few tests on Linux. For a C++ executable, when running standalone the user stack is around 0x7ffeffffff. When running under Valgrind 0x7ffeffff00 is where VG's stack is and the guests stack is around 0x1ffeffffff. With strace I see the following 4k mmaps (I'm assuming the stack guard is 4k).

standalone
paulf> strace ./hello 2>&1 | grep mmap | grep 4096
mmap(0x7fed1b45e000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7fed1b45e000
mmap(0x7fed1b45f000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7fed1b45f000

That fits with what I see with the standalone C++ exe.

Under Valgrind

paulf> strace -ff /home/paulf/tools/valgrind/bin/valgrind -q ./hello 2>&1 | grep mmap | grep 4096
mmap(0x4000000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED, 4, 0) = 0x4000000
mmap(0x402e000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, 0, 0) = 0x402e000
mmap(0x482f000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x482f000
mmap(0x4830000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x4830000

mmap(0x48b6000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x48b6000
mmap(0x48b7000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x48b7000

This time I don't see an mmap near where the user stack is.