Mac OS X: Guest hangs when allocating >=2G of RAM

[LekKit]

The issue

• Few people reported RVVM guests hanging on Mac OS (OpenSBI doesn't start, UART is empty), on both M1 and older x86 ones.
• The same setup works on any other system 👀
• It was quickly determined that when they remove some of their arguments, it starts working.
• The culpit is setting -mem to 2G or higher.

Now, there is only one place which is directly affected by -mem and it's riscv_init_ram(), which allocates guest memory using either mmap() or calloc() (depending on the target), and enables KSM/hugepages where available.
The same hang scenario happens with either implementations on OS X, which are also used on Linux, Windows, FreeBSD and the problem doesn't happen there... So, same code, same APIs, different results. This already starts to be confusing.

The RAM size is later passed to the guest inside the FDT, and current debugging efforts show that it starts to work when we lie about RAM size being smaller than it actually is. To be continued...

[LekKit]

Ok, so the culpit is this:

• RVVM loads bootrom by issuing blk_io rvread() on RAM buffer, passing the RAM size as the destination buffer size (Implying, that it should read up to RAM size bytes and not write past over it). Here:
  

  RVVM/src/rvvm.c

  Line 162 in 31ed7f5

  rvread(machine->bootrom_file, machine->mem.data, machine->mem.size, 0);

• The rvread() wrapper is implemented over pread() syscall on POSIX systems (Not that OS X is good at being POSIX... See 357cd04), and the buffer size is passed directly to it

After all, the buffer is a buffer, there is not much about it's size passed - just read the entire file contents without possibility of writing past it. Right?
But knowing that we fail at precisely 2G point, it becomes clear: It's an integer overflow!

So it turns out that OS X may not accept pread() buffer size values higher than INT_MAX, fails silently, and it is not stated anywhere in their documentation.... Ugh dammit, even Windows isn't so lame!

The Linux man doesn't mention that, too, and rightfully so - it just works, what's so hard in 21st century about accepting a 64-bit buffer size and then figuring that it's just an upper limit on the buffer size?

It seems like FreeBSD man is honest at least, and tells us this:
[EINVAL] The value nbytes is greater than INT_MAX.

So it seems like this is the optimal portable way to go, and sadly not mentioned much in most portability guides on the internet. The fix will be there, soon.
Special thanks for everyone involved: @archanox, @pufit, @Corgiek