Bring Randomness Recipe up to snuff

[JoshOrndorff]

The gen-random pallet has been kicking around the recipes as a vestigial remnant for months. This PR completes it and updates it to alpha.3

Any time randomness comes up, important warnings about complex cryptographic principles come up. To address that I'l lstate some goals for this recipe.

Goals:

• Explain how to use the randomness sources available, should the user want to use them
• Make it clear that the sources are not perfect, and cryptography matters
• Make a reasonable effort to document cryptographic concerns and best practices in an approachable way
• Do not state anything cryptographically incorrect

Non-Goals:

• Write a comprehensive overview of randomness sources
• Explain in full mathematical detail the trade-offs between the available sources

[JoshOrndorff]

Also requesting review from @burdges

[burdges]

We've never encountered any situation in which Randomness Collective Flip provides secure randomness.

Are you answering https://github.com/w3f/polkadot-genesis-audits-issues/issues/13 here? As I wrote there, I've only noticed Randomness Collective Flip being used in impl *::Trait for Test. I presume these are module level test, which sound okay, but maybe system randomness works there too.

It's plausible AURA or PoW users might employ Randomness Collective Flip, but really both should adopt BABE if they need randomness with any security properties. BABE has configurations that emulate either AURA or PoW.

It's likely AURA or PoW speeds early development over BABE however, so maybe that's the use case? Anyone know any use cases for the Randomness Collective Flip pallet?

---

There are several randomness sources worth considering:

1. VRF outputs ala BABE/Sassafras are known well in advance by the block producer, but remain rare and can only be manipulated by sacrificing a block. Almost always use them.

2. Blockhashes are fairly trivial to manipulate. Avoid using blockhashes except when your cryptographer or statistician says so.*

   There do exist operations based on a blocks own hash in which you gain interesting security properties, like Fiat-Shamir transforms, provided adversaries loose except on a negligible fraction of random valus. It's safe doing this with host code like in derandomized batch verification of ed25519 signatures ala Batch signature verification paritytech/substrate#5023 (comment) We shall exploit parachain block hashes inside Polkadot's secondary approval checker assignment procedure too.

   If however runtime code exploits such tricks then one can easily create invalid blocks, but if block producers verify before publishing doing this creates only a DoS vector, not an attack that gets block producers slashed. As an example, imagine our runtime uses hashmaps seeded with the current blockhash, which might or might not give an adversary some trouble creating blocks that DoSed validators. Our block producer must make the block with a different seed since they do not yet know the block hash. It'll almost surely be possible to give block producers transactions such that they occasionally assemble invalid blocks because the hashmaps iterate differently. If this is easy enough then this becomes another DoS vector.

3. There are statistics like median that reduce any single measurements' influence upon the result. We exploit medians in the relative time algorithm for example, but exploiting them remains situation specific. In particular, these measurements occur over a wide domain and obey some non-uniform distribution.

4. There are low-influence boolean functions like collective flip from which folks construct randomness with bitwize "votes" taken from another randomness source like blockhashes or VRF outputs. An adversary biases these by voting some bits into their preferred state, but these "votes" act independently on all other bits, which arguably further weakens individuals. There are however many bits our adversary can shift with only a few votes, which strengthens adversaries who control several voters.

   Aside from https://www.cse.huji.ac.il/~nati/PAPERS/coll_coin_fl.pdf cited in the safemix crate, there is an enormous literature on these boolean functions like https://eccc.weizmann.ac.il/report/2018/140/ We thus expect some become useful somewhere, but so far we've never seen reasonable use cases, and actually majority rule wins under some important models.

[burdges]

Actually all randomness sources fail badly for parachains with few collators, but randomness collective flip fails really badly. We need the parachain to draw randomness from the relay chain, but it'll be months before anything like that can be implemented: paritytech/cumulus#72

[JoshOrndorff]

@burdges Thanks for the review.

Can you say more about configuring babe to work like PoW?

Is it possible or advisable to use Babe only for the randomness beacon, while using something different for actual block authoring?

Are you answering w3f/polkadot-genesis-audits-issues#13 here

That link seems to be dead (or maybe just private). I was not intentionally addressing it.

[burdges]

Can you say more about configuring babe to work like PoW?

BABE has two types of block, round robin blocks similar to AURA and VRF authorized blocks similar to Ouroboros Praos. If you turn off the round robin blocks, then your VRF authorized blocks simulate PoW with PoS. This is the theorem of the Ouroboros Praos paper. It's not a perfect simulation because nodes must lock stake and wait a couple epoch to start, but within the set of already staked nodes that's what happens.

Is it possible or advisable to use Babe only for the randomness beacon, while using something different for actual block authoring?

Why bother?

We'll eventually feed randomness from the relay chains back into parachains, which makes such concerns irrelevant. I think.

[JoshOrndorff]

We'll eventually feed randomness from the relay chains back into parachains, which makes such concerns irrelevant. I think.

The Substrate Recipes are about all kinds of blockchains, not just Polkadot. A stand-alone chain may want a babe-like randomness beacon with PoW

[burdges]

I doubt it. You need PoS for BABE so why do any PoW then?

You might extend BABE with PoW authorized blocks in addition to the AURA and Praos authorized blocks, but not sure why.

I could imagine randomness beacon designs for PoW chains, but we've no real interest in designing or implementing one afaik.

[jimmychu0807]

Made minor edit on some links. Overall lgtm.

[JoshOrndorff]

I made almost the same change in the PoW recipe. I wish we didn't have to do it this way, but I don't have any better idea.

[burdges]

There are still both CollectiveFlipRandomnessSource and BabeRandomnessSource` in this, which should never happen in real code. Why does it happen here?