Export and document cryptographically secure random numbers

[chethega]

Various applications need a source of cryptographically secure random numbers. We currently do not provide a CSPRNG.

Secure random is a very basic primitive, and imo as widely important as secure hash (which we do provide). Further, we have a very nice AbstractRNG interface. In other words, there are very few API decisions (mainly: whether to define Random.seed! for the CSPRNG). Secure random could live in packages, but this has lots of disadvantages: First, it is inconvenient, which will lead to people misusing the default random generator. Second, a central place for maintaining such basic functionality is preferable to many competing implementations, both security-wise and in terms of simplified audits of packages.

There was some discussion here that inspired me to grep through a lot of packages to gauge how they use randomness. I'll link all of the individual issues here, to make the point that this is important functionality, and that there is confusion about secure random in the package ecosystem. Considerations about the default RNG are separate.

A sensible solution could be to use RandomDevice(). In this case, we would simply update the docs.

[JeffBezanson]

Secure random could live in packages, but this has lots of disadvantages:

On the other hand, putting it in a package makes releasing bug fixes or improvements much faster.

[ViralBShah]

To start with, it should certainly be a package. As it gains more widespread usage, we can figure out if it should live in stdlib. Given that we are more likely to move things out of stdlib rather than into it, having an external package seems better.

[chethega]

I disagree.

Security is still the toxic wastedump of computer science. People are rightfully reluctant of adding external dependencies. Many people are naive with respect to crypto and security. People lack the resources to properly vet packages that provide secure random. We want to support a culture of "when in doubt, then be secure". We absolutely do not want to force people to make ill-informed trade-offs whether they "can get away" with using insecure rand. It must be easy to do the right thing; otherwise people will invariably mess up. Bad random leads to perfectly functional test-passing vulnerable packages.

At the very least, I want the Random stdlib to document a best-practice for secure random. This is basic functionality and there must be an officially endorsed way of getting secure random.

If we do not want to implement a CSPRNG, then we could export a wrapper around one of the mbedtls CSPRNGs. mbedtls is already a binary dependency of julia.

We could also fix up the random wrapper in MbedTLS.jl and be opinionated to point people there. MbedTLS.jl is already a dependency for large parts of the ecosystem, to the point that it is "effectively stdlib".

We could also point people at the RandomDevice, as proposed here. That has thread-safety issues we'd need to resolve. But pointing people there, with the magic words "use this for secure random", relieves them of reading the julia source, then reading the docs of the called windows SystemFunction036 and then searching for cryptoanalysis papers (due dilligence I did for my PR, and that would need to be replicated before officially endorsing RandomDevice for this role).

CC @hustf @essenciary @sbromberger for a user perspective.

[StefanKarpinski]

Question: is this intended to produce truly unpredictable random data like /dev/random or a reproducible stream of pseudo-random data?

[sbromberger]

On the other hand, putting it in a package makes releasing bug fixes or improvements much faster.

I might have more to say later, but I want to address @JeffBezanson 's comment here from a different perspective: while "releasing bug fixes or improvements" might be a valid reason to move non-secure code out to a 3rd-party package, I'm not sure this is a great strategy for things like CSPRNGs. There are a couple of reasons for my belief:

1. Secure primitives should be available in stdlib/base. This is the case for a number of modern languages: Go, Python, Java, Rust.

2. Even if that weren't compelling enough, it is important that bug fixes to crypto libraries be as widely announced and deployed as possible, as quickly as possible. This argues for a centralized model where a new version of the base/stdlib is released in response to a security/crypto problem. There's no better way to advertise "Hey, if you continue to use an older version of this code you're going to be insecure" than saying "Hey, we've just released Julia 1.3.1 with security fixes. All users are encouraged and advised to upgrade as soon as possible". It is wishful thinking to believe that users of third-party libs - especially when they're indirect (transitive) dependencies - will keep their fingers on the pulse of their downstreams.

3. Because crypto libs tend to be static in terms of their development/enhancement, there's little benefit to splitting them out into a fast release process. In fact, I'd argue that stability of crypto libs is a requirement. I don't want to outsource my security to a third-party lib. I want it baked into the language, and vetted/updated by the language maintainers. It's that important, especially if we want to see Julia move from "here's an interesting language that we can play around with interactively" to "here's a service we've built on this language and are making available to some subset of the general public".

Finally, and possibly a distraction: extraordinary claims require extraordinary evidence. I am unconvinced that moving code out to third-party libraries has actually resulted in more frequent releases than what we see in Julia Base/stdlib. Just in 2019, we've had 3 releases of Julia, with two more imminent. I don't know that third-party packages have seen that sort of iteration: picking some popular packages, I see that in the same timeframe,

• DiffEq has had 2 releases
• Flux has had 1
• Gadfly has had 1
• Distributions has had 5 (still 0.x)
• OrderedCollections has had 1

I'm certainly not arguing that these should be moved (back) into Base. (Actually, for a few of them perhaps I am, but that's a different discussion.) To reiterate my point more simply, and to wrap it up:

TL/DR MHO, YMMV, etc.: Crypto functionality should be primitive and stable, and should be important enough to be versioned as part of the language itself.

[StefanKarpinski]

As the person merging the releases, I can assure you that DiffEq* has had a lot more than 2 releases. Doing git log --since=2019-1-1 D/DiffEq* | grep 'New version' | wc -l in the registry repo shows that the DiffEq collection of packages have had 90 releases. Are you talking about DiffEqBase? That alone has had 27 releases. That would have been completely impossible if DiffEq was a standard library.

Suppose that Gadfly were made into a standard library. Does that mean any more work would have been done on it? No. It would get the same amount of maintenance as it does now. Even if it would have technically been included in 3 different Julia releases, if the actual code was unchanged, it's hard to argue that this is really a release.

[StefanKarpinski]

Again, I'd like to ask about the P part of CSPRNG is significant. Because if people just want a good reliable stream of random data, that's a pretty simple, stable API that I'd be happy to commit to. If people don't want reproducibility, then we can change algorithms at any time and not worry about breaking people's programs. If people want reproducibility, then that's a whole different story because then we need to pick and commit to maintaining and supporting a given algorithm. The original suggestion of using RandomDevice() suggests that a CSPRNG is not actually what's wanted, but a good, standard way to get secure true randomness.

[JeffBezanson]

I also think looking at the release schedules of other existing packages is not relevant. The point was just that it's possible to release new versions independent of julia/stdlib. If you want to make a new release, you can at any time. Even if it were true that packages have on average fewer releases than julia, there is no causal relationship. Is it really so crazy to consider a compiler and a crypto library separate projects?

Long term, the structure I'd like to have is everything in stdlib moved to separate repos, but with a linux-distro-like bundling mechanism where we make it easy to get a (possibly custom) set of packages in one download.

But, if we can make the existing RandomDevice do this (and of course make it as easy to use as possible and document it), that would be great since it's within easy reach.

[sbromberger]

Yup, as I feared, that last point became a distraction. However, as mentioned (perhaps ad nauseam) elsewhere,

Long term, the structure I'd like to have is everything in stdlib moved to separate repos

will kill practical use of Julia in many environments. If this is indeed the direction The Project is headed, confirmation now will allow me to start looking at alternatives in my organization. This preference has been repeated often enough by enough people that it is now, in my mind, a Thing To Be Taken Seriously. I'll follow up off github.