Use /dev/urandom on macOS

[newpavlov]

See this comment by @RalfJung. It also should help with rust-random/rand#733.

Open question: do we need to read one byte from /dev/random to ensure that entropy pool has initialized?

cc @ebarnard

[RalfJung]

Miri would be sad about this, as we can hook SecRandomCopyBytes much easier than hooking a file. But you probably shouldn't let that affect your decision. This might temporarily regress random testing on macOS on Miri, though.

Cc @oli-obk

[newpavlov]

Note that ring uses SecRandomCopyBytes, see this issue: briansmith/ring#149

[ebarnard]

As is noted in the second comment from briansmith/ring#149 SecRandomCopyBytes is a CSPRNG which periodically reseeds from /dev/random. This CSPRNG is run on its own thread with access synchronised via Grand Central Dispatch. Using SecRandomCopyBytes involves linking with Security.framework and by extension Core Foundation which does a lot of life before main initialisation and is likely the cause of rust-random/rand#733.

Interestingly MacOS supports a getentropy syscall since 10.12 which reads from the same source as /dev/random. Note that /dev/random and /dev/urandom appear to be exactly the same on MacOS.

My suggestion would be to take the same approach on MacOS as Linux. Try the getentropy syscall first and fall back to reading /dev/(u)random on ENOSYS.

For iOS we would try getentropy first and fall back to SecRandomCopyBytes on ENOSYS as /dev/urandom is not directly accessible from the sandbox. Or, if we are not supporting versions before iOS 10, we can just use getentropy with no fallback.

If we agree with this approach I am happy to implement.

@RalfJung I think this would also make Miri happy.

[newpavlov]

At first glance looks good to me and it will be great if you'll implement it! But before doing it I think we should wait for #13 to be closed first.

[RalfJung]

I think this would also make Miri happy.

The dedicated syscall for Linux works great for Miri, so I suppose the same would be true for macOS. :)

[josephlr]

Let me see if I'm understanding this correctly. We don't want to use SecRandomCopyBytes because:

• It makes linking more complex (Use getrandom crate rust-lang/rust#62082 (comment))
• It incurs a notable startup cost (It makes startup time significantly longer on Mac rand#733)

First, iOS:

For iOS we would try getentropy first and fall back to SecRandomCopyBytes on ENOSYS as /dev/urandom is not directly accessible from the sandbox.

Both of the downsides to using SecRandomCopyBytes occur regardless of if we actually use it. So it would seem best to just only use SecRandomCopyBytes until the min Rust supported iOS version is iOS 10. When that happens, getentropy(2) should be added to the libc crate, and we should just call libc::getentropy. I've opened rust-lang/libc#1406 to figure out what the min iOS version should even be.

Now, macOS:

My suggestion would be to take the same approach on MacOS as Linux. Try the getentropy syscall first and fall back to reading /dev/(u)random on ENOSYS.

The main problem with this approach is that it's not supported on macOS. Unlike Linux, which has a stable syscall API, the stable API for macOS is the libc and other system libraries/frameworks. This means that there's not a way to "check" if getentropy is a supported syscall. It's either in the libc or it is not. For example, the libc crate has all the SYS_* numbers on Linux, but does not expose them on macOS, as they are unstable and an implementation detail.

Given this, it seems like the implementation options for macOS are:

1. Stick with using SecRandomCopyBytes
   - Upsides: simple, same implementation as iOS
   - Downsides: hard to link, and a (small) startup cost
2. Do what the solaris implementation does to dynamically see if getentropy is in the libc, and then fall back to /dev/random if it isn't there.
   - Upsides: mitigates file descriptor depletion attack
   - Downsides: very hacky, doesn't work well with miri (@RalfJung is this true?)
3. Just use /dev/random always:
   - Upsides: very simple
   - Downsides: file descriptor depletion attack
4. Just use libc::getentropy unconditionally
   - Infeasible until min macOS version supported by Rust is 10.12

Despite it's weirdness, I think approch (2) is the best, as we already have similar implementations, and it seems to solve most of the problems raised in rust-lang/rust#62082

[ebarnard]

@josephlr

Grand Central Dispatch is not initialised until the first call to SecRandomCopyBytes at which point at least two threads are spawned so there is an advantage to using it only as a fallback. Also, on iOS it is almost guaranteed that you are linking against Core Foundation. I believe it is also possible to dynamically load SecRandomCopyBytes if you really want to guarantee no overhead.

By syscall I mean calling a function provided by libSystem on MacOS which are can be called from the libc crate, and by ENOSYS I mean that dynamically loading that symbol failed. I felt that gave a simpler explanation. Sorry if that caused any confusion.

So yes I agree with your option 2. This pattern is used fairly regularly in Rust's libstd and I think is compatible with miri. There's even a macro for it!

[RalfJung]

• Downsides: very hacky, doesn't work well with miri (@RalfJung is this true?)

dlsym should be easier to support in Miri than file descriptors. It's a hack in Miri as well, but not much worse than some of the things we already do. ;)

[josephlr]

@ebarnard that's awesome info (especially the macro), it's good to know that weak linkage is more reliable than I thought!

So I think we're agreed with what to do here:

1. Use weak linkage to getentropy, and if we have it, call with chunks(256).
2. Fallback to a suboptimal method:

• iOS: SecRandomCopyBytes
• macOS: read from /dev/random

[ebarnard]

I've been looking into the iOS situation a bit more and it seems the header sys/random.h isn't included in the iOS 12 SDK although oddly the syscall number does appear in sys/syscall.h and the symbol is exported from libsystem_kernel.dylib.

Therefore presumably it is a private API so we should probably just stick to using only SecRandomCopyBytes on iOS.

[ebarnard]

1. Use weak linkage to getentropy, and if we have it, call with chunks(256).

Yes, it behaves similarly to the OpenBSD syscall so we would need chunks(256).

[newpavlov]

@ebarnard
The TLS removal PR got merged, so you can start macOS/iOS rework.

[josephlr]

Therefore presumably it is a private API so we should probably just stick to using only SecRandomCopyBytes on iOS.

Sounds very reasonable to me.