Add op pool for slashings and exits

[dapplion]

Motivation

Our current operation pools have a critical flaw:

• They don't verify that the existing items are valid in the state when proposing blocks. This will very likely invalidate all blocks Lodestar tries to propose in the epochs close to slashings and exists being propagated through the network.

Also the gossip validation of slashings and exits required db reads, specially for attester slashings where all values had to be read.

Description

• Use in-memory cache for attester slashing, proposer slashing and voluntary exists
• Use this fast memory cache for gossip validation, with a fast index for attester slashing intersected indexes
• Read from DB once on start-up, and persist to DB once on shutdown

[codeclimate]

Code Climate has analyzed commit 62279f7 and detected 1 issue on this pull request.

Here's the issue category breakdown:

Category	Count
Complexity	1

View more on Code Climate.

[github-actions]

Performance Report

✔️ no performance regression detected

Full benchmark results

Benchmark suite	Current: 6b1ff9e	Previous: 7beccf0	Ratio
BeaconState.hashTreeRoot - No change	701.00 ns/op	856.00 ns/op	0.82
BeaconState.hashTreeRoot - 1 full validator	98.295 us/op	99.771 us/op	0.99
BeaconState.hashTreeRoot - 32 full validator	1.3135 ms/op	1.4010 ms/op	0.94
BeaconState.hashTreeRoot - 512 full validator	16.541 ms/op	18.219 ms/op	0.91
BeaconState.hashTreeRoot - 1 validator.effectiveBalance	92.507 us/op	101.80 us/op	0.91
BeaconState.hashTreeRoot - 32 validator.effectiveBalance	1.2492 ms/op	1.4201 ms/op	0.88
BeaconState.hashTreeRoot - 512 validator.effectiveBalance	18.548 ms/op	20.908 ms/op	0.89
BeaconState.hashTreeRoot - 1 balances	66.735 us/op	73.824 us/op	0.90
BeaconState.hashTreeRoot - 32 balances	587.77 us/op	609.42 us/op	0.96
BeaconState.hashTreeRoot - 512 balances	5.4684 ms/op	5.7907 ms/op	0.94
BeaconState.hashTreeRoot - 250000 balances	107.60 ms/op	111.63 ms/op	0.96
processSlot - 1 slots	52.531 us/op	56.165 us/op	0.94
processSlot - 32 slots	2.8847 ms/op	3.0096 ms/op	0.96
getCommitteeAssignments - req 1 vs - 250000 vc	4.9690 ms/op	5.2904 ms/op	0.94
getCommitteeAssignments - req 100 vs - 250000 vc	6.9101 ms/op	7.5268 ms/op	0.92
getCommitteeAssignments - req 1000 vs - 250000 vc	7.7968 ms/op	8.1685 ms/op	0.95
computeProposers - vc 250000	24.257 ms/op	27.063 ms/op	0.90
computeEpochShuffling - vc 250000	184.47 ms/op	201.24 ms/op	0.92
getNextSyncCommittee - vc 250000	423.29 ms/op	421.40 ms/op	1.00
altair processBlock - 250000 vs - 7PWei normalcase	82.586 ms/op	107.05 ms/op	0.77
altair processBlock - 250000 vs - 7PWei worstcase	235.80 ms/op	270.00 ms/op	0.87
altair processEpoch - pyrmont_e62330	514.56 ms/op	567.05 ms/op	0.91
pyrmont_e62330 - altair beforeProcessEpoch	153.66 ms/op	170.85 ms/op	0.90
pyrmont_e62330 - altair processJustificationAndFinalization	135.89 us/op	123.69 us/op	1.10
pyrmont_e62330 - altair processInactivityUpdates	74.992 ms/op	79.463 ms/op	0.94
pyrmont_e62330 - altair processRewardsAndPenalties	54.316 ms/op	62.626 ms/op	0.87
pyrmont_e62330 - altair processRegistryUpdates	29.847 us/op	24.492 us/op	1.22
pyrmont_e62330 - altair processSlashings	7.8940 us/op	6.3760 us/op	1.24
pyrmont_e62330 - altair processEth1DataReset	5.7010 us/op	5.4880 us/op	1.04
pyrmont_e62330 - altair processEffectiveBalanceUpdates	6.2617 ms/op	6.3310 ms/op	0.99
pyrmont_e62330 - altair processSlashingsReset	35.774 us/op	35.083 us/op	1.02
pyrmont_e62330 - altair processRandaoMixesReset	50.198 us/op	47.354 us/op	1.06
pyrmont_e62330 - altair processHistoricalRootsUpdate	8.8590 us/op	7.4000 us/op	1.20
pyrmont_e62330 - altair processParticipationFlagUpdates	49.270 ms/op	48.152 ms/op	1.02
pyrmont_e62330 - altair processSyncCommitteeUpdates	5.7800 us/op	5.7680 us/op	1.00
pyrmont_e62330 - altair afterProcessEpoch	116.76 ms/op	129.40 ms/op	0.90
altair processInactivityUpdates - 250000 normalcase	204.36 ms/op	227.68 ms/op	0.90
altair processInactivityUpdates - 250000 worstcase	204.76 ms/op	222.92 ms/op	0.92
altair processParticipationFlagUpdates - 250000 anycase	79.641 ms/op	93.029 ms/op	0.86
altair processRewardsAndPenalties - 250000 normalcase	110.12 ms/op	123.03 ms/op	0.90
altair processRewardsAndPenalties - 250000 worstcase	114.27 ms/op	137.97 ms/op	0.83
altair processSyncCommitteeUpdates - 250000	416.35 ms/op	428.39 ms/op	0.97
Tree 40 250000 create	557.80 ms/op	599.72 ms/op	0.93
Tree 40 250000 get(125000)	265.90 ns/op	303.99 ns/op	0.87
Tree 40 250000 set(125000)	1.4512 us/op	1.9546 us/op	0.74
Tree 40 250000 toArray()	36.299 ms/op	45.631 ms/op	0.80
Tree 40 250000 iterate all - toArray() + loop	36.183 ms/op	52.471 ms/op	0.69
Tree 40 250000 iterate all - get(i)	106.67 ms/op	119.57 ms/op	0.89
MutableVector 250000 create	23.371 ms/op	22.299 ms/op	1.05
MutableVector 250000 get(125000)	13.470 ns/op	15.720 ns/op	0.86
MutableVector 250000 set(125000)	524.60 ns/op	604.09 ns/op	0.87
MutableVector 250000 toArray()	7.0766 ms/op	8.0982 ms/op	0.87
MutableVector 250000 iterate all - toArray() + loop	7.2773 ms/op	8.2924 ms/op	0.88
MutableVector 250000 iterate all - get(i)	3.3582 ms/op	3.6617 ms/op	0.92
Array 250000 create	4.5835 ms/op	5.2334 ms/op	0.88
Array 250000 clone - spread	1.7335 ms/op	1.8370 ms/op	0.94
Array 250000 get(125000)	0.85300 ns/op	0.91200 ns/op	0.94
Array 250000 set(125000)	0.90200 ns/op	0.92400 ns/op	0.98
Array 250000 iterate all - loop	127.11 us/op	134.75 us/op	0.94
aggregationBits - 2048 els - readonlyValues	223.27 us/op	226.61 us/op	0.99
aggregationBits - 2048 els - zipIndexesInBitList	38.724 us/op	42.430 us/op	0.91
ssz.Root.equals	1.2870 us/op	1.4320 us/op	0.90
ssz.Root.equals with valueOf()	1.5030 us/op	1.6090 us/op	0.93
byteArrayEquals with valueOf()	1.4890 us/op	1.5740 us/op	0.95
phase0 processBlock - 250000 vs - 7PWei normalcase	14.053 ms/op	15.068 ms/op	0.93
phase0 processBlock - 250000 vs - 7PWei worstcase	92.658 ms/op	100.23 ms/op	0.92
phase0 afterProcessEpoch - 250000 vs - 7PWei	210.64 ms/op	219.85 ms/op	0.96
phase0 beforeProcessEpoch - 250000 vs - 7PWei	673.87 ms/op	731.65 ms/op	0.92
phase0 processEpoch - mainnet_e58758	891.23 ms/op	996.42 ms/op	0.89
mainnet_e58758 - phase0 beforeProcessEpoch	579.45 ms/op	617.41 ms/op	0.94
mainnet_e58758 - phase0 processJustificationAndFinalization	132.77 us/op	151.82 us/op	0.87
mainnet_e58758 - phase0 processRewardsAndPenalties	77.080 ms/op	86.890 ms/op	0.89
mainnet_e58758 - phase0 processRegistryUpdates	112.62 us/op	95.330 us/op	1.18
mainnet_e58758 - phase0 processSlashings	8.6890 us/op	6.3080 us/op	1.38
mainnet_e58758 - phase0 processEth1DataReset	7.4040 us/op	5.5250 us/op	1.34
mainnet_e58758 - phase0 processEffectiveBalanceUpdates	9.5252 ms/op	9.2954 ms/op	1.02
mainnet_e58758 - phase0 processSlashingsReset	41.943 us/op	30.763 us/op	1.36
mainnet_e58758 - phase0 processRandaoMixesReset	53.715 us/op	42.651 us/op	1.26
mainnet_e58758 - phase0 processHistoricalRootsUpdate	12.886 us/op	7.4120 us/op	1.74
mainnet_e58758 - phase0 processParticipationRecordUpdates	36.298 us/op	29.298 us/op	1.24
mainnet_e58758 - phase0 afterProcessEpoch	185.14 ms/op	197.23 ms/op	0.94
phase0 processEffectiveBalanceUpdates - 250000 normalcase	11.238 ms/op	10.678 ms/op	1.05
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5	1.4217 s/op	1.8499 s/op	0.77
phase0 processRegistryUpdates - 250000 normalcase	111.61 us/op	99.222 us/op	1.12
phase0 processRegistryUpdates - 250000 badcase_full_deposits	3.8249 ms/op	5.1211 ms/op	0.75
phase0 processRegistryUpdates - 250000 worstcase 0.5	1.9705 s/op	2.3972 s/op	0.82
phase0 getAttestationDeltas - 250000 normalcase	35.380 ms/op	44.099 ms/op	0.80
phase0 getAttestationDeltas - 250000 worstcase	34.625 ms/op	40.741 ms/op	0.85
phase0 processSlashings - 250000 worstcase	39.312 ms/op	37.389 ms/op	1.05
shuffle list - 16384 els	14.445 ms/op	14.261 ms/op	1.01
shuffle list - 250000 els	191.07 ms/op	194.97 ms/op	0.98
getEffectiveBalances - 250000 vs - 7PWei	11.635 ms/op	11.410 ms/op	1.02
getPubkeys - index2pubkey - req 1000 vs - 250000 vc	2.2273 ms/op	2.5079 ms/op	0.89
getPubkeys - validatorsArr - req 1000 vs - 250000 vc	1.0255 ms/op	1.0649 ms/op	0.96
BLS verify - blst-native	2.3410 ms/op	2.2624 ms/op	1.03
BLS verifyMultipleSignatures 3 - blst-native	4.9241 ms/op	4.1758 ms/op	1.18
BLS verifyMultipleSignatures 8 - blst-native	10.526 ms/op	9.5684 ms/op	1.10
BLS verifyMultipleSignatures 32 - blst-native	39.130 ms/op	34.697 ms/op	1.13
BLS aggregatePubkeys 32 - blst-native	52.495 us/op	47.997 us/op	1.09
BLS aggregatePubkeys 128 - blst-native	210.66 us/op	181.50 us/op	1.16
getAttestationsForBlock	82.657 ms/op	92.343 ms/op	0.90
CheckpointStateCache - add get delete	17.123 us/op	18.805 us/op	0.91
validate gossip signedAggregateAndProof - struct	5.8474 ms/op	5.2051 ms/op	1.12
validate gossip signedAggregateAndProof - treeBacked	5.6061 ms/op	5.3160 ms/op	1.05
validate gossip attestation - struct	2.6615 ms/op	2.4692 ms/op	1.08
validate gossip attestation - treeBacked	2.6831 ms/op	2.3738 ms/op	1.13

by benchmarkbot/action

[codecov]

Codecov Report

Merging #3134 (62279f7) into master (7beccf0) will decrease coverage by 0.44%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3134      +/-   ##
==========================================
- Coverage   58.78%   58.33%   -0.45%     
==========================================
  Files         374      375       +1     
  Lines        9091     9171      +80     
  Branches     1348     1360      +12     
==========================================
+ Hits         5344     5350       +6     
- Misses       3365     3439      +74     
  Partials      382      382              

[wemeetagain]

LGTM 🎉