From 7d4b97240b3825966c8783f6df54300f7c80684a Mon Sep 17 00:00:00 2001 From: Paul Hauner Date: Mon, 6 Apr 2020 17:46:33 +1000 Subject: [PATCH 01/44] Redefine attestation propogation condition --- specs/phase0/p2p-interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index 24a1d43761..4c50892a3b 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -316,7 +316,7 @@ Attestation subnets are used to propagate unaggregated attestations to subsectio - The attestation's committee index (`attestation.data.index`) is for the correct subnet. - `attestation.data.slot` is within the last `ATTESTATION_PROPAGATION_SLOT_RANGE` slots (within a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) -- i.e. `attestation.data.slot + ATTESTATION_PROPAGATION_SLOT_RANGE >= current_slot >= attestation.data.slot` (a client MAY queue future attestations for processing at the appropriate slot). - The attestation is unaggregated -- that is, it has exactly one participating validator (`len([bit for bit in attestation.aggregation_bits if bit == 0b1]) == 1`). - - The attestation is the first valid attestation received for the participating validator for the slot, `attestation.data.slot`. + - There has been no other attestation seen on an attestation subnet that has an indentical `attestation.data.slot` and participating validator index. - The block being voted for (`attestation.data.beacon_block_root`) passes validation. - The signature of `attestation` is valid. From bdf087d7f3a32133543f3a736fcbbb438ff87154 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 6 Apr 2020 09:57:23 -0600 Subject: [PATCH 02/44] add notes about how to handle peer discovery and gossip topics prior to genesis --- specs/phase0/p2p-interface.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index 24a1d43761..c94b2c7e35 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -105,6 +105,7 @@ It consists of four main sections: - [Discovery](#discovery) - [Why are we using discv5 and not libp2p Kademlia DHT?](#why-are-we-using-discv5-and-not-libp2p-kademlia-dht) - [What is the difference between an ENR and a multiaddr, and why are we using ENRs?](#what-is-the-difference-between-an-enr-and-a-multiaddr-and-why-are-we-using-enrs) + - [Why do we not form ENRs and find peers until genesis block/state is known?](#why-do-we-not-form-enrs-and-find-peers-until-genesis-blockstate-is-known) - [Compression/Encoding](#compressionencoding) - [Why are we using SSZ for encoding?](#why-are-we-using-ssz-for-encoding) - [Why are we compressing, and at which layers?](#why-are-we-compressing-and-at-which-layers) @@ -247,6 +248,8 @@ Topics are plain UTF-8 strings and are encoded on the wire as determined by prot - `Name` - see table below - `Encoding` - the encoding strategy describes a specific representation of bytes that will be transmitted over the wire. See the [Encodings](#Encoding-strategies) section for further details. +*Note*: `ForkDigestValue` is composed of values that are not known until the genesis block/state are available. Due to this, clients SHOULD NOT subscribe to gossipsub topics until these genesis values are known. + Each gossipsub [message](https://github.com/libp2p/go-libp2p-pubsub/blob/master/pb/rpc.proto#L17-L24) has a maximum size of `GOSSIP_MAX_SIZE`. Clients MUST reject (fail validation) messages that are over this size limit. Likewise, clients MUST NOT emit or propagate messages larger than this limit. The `message-id` of a gossipsub message MUST be: @@ -752,6 +755,8 @@ where the fields of `ENRForkID` are defined as * `next_fork_version` is the fork version corresponding to the next planned hard fork at a future epoch. If no future fork is planned, set `next_fork_version = current_fork_version` to signal this fact * `next_fork_epoch` is the epoch at which the next fork is planned and the `current_fork_version` will be updated. If no future fork is planned, set `next_fork_epoch = FAR_FUTURE_EPOCH` to signal this fact +*Note*: `fork_digest` is composed of values that are not not known until the genesis block/state are available. Due to this, clients SHOULD NOT form ENRs and begin peer discovery until genesis values are known. + Clients SHOULD connect to peers with `fork_digest`, `next_fork_version`, and `next_fork_epoch` that match local values. Clients MAY connect to peers with the same `fork_digest` but a different `next_fork_version`/`next_fork_epoch`. Unless `ENRForkID` is manually updated to matching prior to the earlier `next_fork_epoch` of the two clients, these connecting clients will be unable to successfully interact starting at the earlier `next_fork_epoch`. @@ -1092,6 +1097,12 @@ discv5 uses ENRs and we will presumably need to: 1. Add `multiaddr` to the dictionary, so that nodes can advertise their multiaddr under a reserved namespace in ENRs. – and/or – 2. Define a bi-directional conversion function between multiaddrs and the corresponding denormalized fields in an ENR (ip, ip6, tcp, tcp6, etc.), for compatibility with nodes that do not support multiaddr natively (e.g. Eth 1.0 nodes). +### Why do we not form ENRs and find peers until genesis block/state is known? + +Although client software might very well be running locally prior to the solidification of the eth2 genesis state and block, clients cannot form valid ENRs prior to this point. ENRs contain `fork_digest` which utilizes the `genesis_validators_root` for a cleaner separation between chains so prior to knowing genesis, we cannot use `fork_digest` to cleanly find peers on our intended chain. Once genesis data is known, we can then form ENRs and safely find peers. + +When using an eth1 deposit contract for deposits, `fork_digest` will be known at least `MIN_GENESIS_DELAY` (24 hours in mainnet configuration) before `genesis_time`, providing ample time to find peers and form initial connections and gossip subnets prior to genesis. + ## Compression/Encoding ### Why are we using SSZ for encoding? From 13d1303db82610d94e5701b713be2b1012dc290a Mon Sep 17 00:00:00 2001 From: protolambda Date: Mon, 6 Apr 2020 18:40:09 +0200 Subject: [PATCH 03/44] update remerkleable; mul/div bound checks, update config loading --- setup.py | 2 +- tests/core/pyspec/eth2spec/config/config_util.py | 13 +++++++++---- .../test_process_justification_and_finalization.py | 4 ++-- 3 files changed, 12 insertions(+), 7 deletions(-) diff --git a/setup.py b/setup.py index 911eb65b0e..d1c62fb72f 100644 --- a/setup.py +++ b/setup.py @@ -499,7 +499,7 @@ def run(self): "pycryptodome==3.9.4", "py_ecc==2.0.0", "dataclasses==0.6", - "remerkleable==0.1.12", + "remerkleable==0.1.13", "ruamel.yaml==0.16.5", "lru-dict==1.1.6" ] diff --git a/tests/core/pyspec/eth2spec/config/config_util.py b/tests/core/pyspec/eth2spec/config/config_util.py index 64c533f2d1..4c5768a294 100644 --- a/tests/core/pyspec/eth2spec/config/config_util.py +++ b/tests/core/pyspec/eth2spec/config/config_util.py @@ -8,13 +8,18 @@ # Access to overwrite spec constants based on configuration # This is called by the spec module after declaring its globals, and applies the loaded presets. -def apply_constants_config(spec_globals: Dict[str, Any]) -> None: +def apply_constants_config(spec_globals: Dict[str, Any], warn_if_unknown: bool = False) -> None: global config for k, v in config.items(): - if k.startswith('DOMAIN_'): - spec_globals[k] = spec_globals['DomainType'](v) # domain types are defined as bytes in the configs + # the spec should have default values for everything, if not, the config key is invalid. + if k in spec_globals: + # Keep the same type as the default value indicates (which may be an SSZ basic type subclass, e.g. 'Gwei') + spec_globals[k] = spec_globals[k].__class__(v) else: - spec_globals[k] = v + # Note: Phase 0 spec will not know the phase 1 config values. + # Yet, during debugging you can enable explicit warnings. + if warn_if_unknown: + print(f"WARNING: unknown config key: '{k}' with value: '{v}'") # Load presets from a file, and then prepares the global config setting. This does not apply the config. diff --git a/tests/core/pyspec/eth2spec/test/phase_0/epoch_processing/test_process_justification_and_finalization.py b/tests/core/pyspec/eth2spec/test/phase_0/epoch_processing/test_process_justification_and_finalization.py index 9f9e1c316b..09af2126db 100644 --- a/tests/core/pyspec/eth2spec/test/phase_0/epoch_processing/test_process_justification_and_finalization.py +++ b/tests/core/pyspec/eth2spec/test/phase_0/epoch_processing/test_process_justification_and_finalization.py @@ -24,7 +24,7 @@ def add_mock_attestations(spec, state, epoch, source, target, sufficient_support raise Exception(f"cannot include attestations in epoch ${epoch} from epoch ${current_epoch}") total_balance = spec.get_total_active_balance(state) - remaining_balance = total_balance * 2 // 3 + remaining_balance = int(total_balance * 2 // 3) # can become negative start_slot = spec.compute_start_slot_at_epoch(epoch) for slot in range(start_slot, start_slot + spec.SLOTS_PER_EPOCH): @@ -42,7 +42,7 @@ def add_mock_attestations(spec, state, epoch, source, target, sufficient_support aggregation_bits = [0] * len(committee) for v in range(len(committee) * 2 // 3 + 1): if remaining_balance > 0: - remaining_balance -= state.validators[v].effective_balance + remaining_balance -= int(state.validators[v].effective_balance) aggregation_bits[v] = 1 else: break From 021cb98dbb808713e8b14c37f049e7662bfbc2c0 Mon Sep 17 00:00:00 2001 From: Paul Hauner Date: Tue, 7 Apr 2020 07:05:51 +1000 Subject: [PATCH 04/44] Use epoch for attestation subnet seen-ness. --- specs/phase0/p2p-interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index 4c50892a3b..8614cd00af 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -316,7 +316,7 @@ Attestation subnets are used to propagate unaggregated attestations to subsectio - The attestation's committee index (`attestation.data.index`) is for the correct subnet. - `attestation.data.slot` is within the last `ATTESTATION_PROPAGATION_SLOT_RANGE` slots (within a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) -- i.e. `attestation.data.slot + ATTESTATION_PROPAGATION_SLOT_RANGE >= current_slot >= attestation.data.slot` (a client MAY queue future attestations for processing at the appropriate slot). - The attestation is unaggregated -- that is, it has exactly one participating validator (`len([bit for bit in attestation.aggregation_bits if bit == 0b1]) == 1`). - - There has been no other attestation seen on an attestation subnet that has an indentical `attestation.data.slot` and participating validator index. + - There has been no other attestation seen on an attestation subnet that has an indentical `attestation.data.target.epoch` and participating validator index. - The block being voted for (`attestation.data.beacon_block_root`) passes validation. - The signature of `attestation` is valid. From 616385a094067e89750608de5d56b7cf320df347 Mon Sep 17 00:00:00 2001 From: Paul Hauner Date: Tue, 7 Apr 2020 07:45:15 +1000 Subject: [PATCH 05/44] Fix spelling mistake --- specs/phase0/p2p-interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index 8614cd00af..e2ca054da5 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -316,7 +316,7 @@ Attestation subnets are used to propagate unaggregated attestations to subsectio - The attestation's committee index (`attestation.data.index`) is for the correct subnet. - `attestation.data.slot` is within the last `ATTESTATION_PROPAGATION_SLOT_RANGE` slots (within a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) -- i.e. `attestation.data.slot + ATTESTATION_PROPAGATION_SLOT_RANGE >= current_slot >= attestation.data.slot` (a client MAY queue future attestations for processing at the appropriate slot). - The attestation is unaggregated -- that is, it has exactly one participating validator (`len([bit for bit in attestation.aggregation_bits if bit == 0b1]) == 1`). - - There has been no other attestation seen on an attestation subnet that has an indentical `attestation.data.target.epoch` and participating validator index. + - There has been no other attestation seen on an attestation subnet that has an identical `attestation.data.target.epoch` and participating validator index. - The block being voted for (`attestation.data.beacon_block_root`) passes validation. - The signature of `attestation` is valid. From c96a3366fade983df68ba80850c1e409cc170c0a Mon Sep 17 00:00:00 2001 From: Paul Hauner Date: Tue, 7 Apr 2020 16:07:41 +1000 Subject: [PATCH 06/44] Tighten aggregate attn propogation condition --- specs/phase0/p2p-interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index c94b2c7e35..d01e4deaff 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -290,7 +290,7 @@ There are two primary global topics used to propagate beacon blocks and aggregat - `beacon_aggregate_and_proof` - This topic is used to propagate aggregated attestations (as `SignedAggregateAndProof`s) to subscribing nodes (typically validators) to be included in future blocks. The following validations MUST pass before forwarding the `signed_aggregate_and_proof` on the network. (We define the following for convenience -- `aggregate_and_proof = signed_aggregate_and_proof.message` and `aggregate = aggregate_and_proof.aggregate`) - `aggregate.data.slot` is within the last `ATTESTATION_PROPAGATION_SLOT_RANGE` slots (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) -- i.e. `aggregate.data.slot + ATTESTATION_PROPAGATION_SLOT_RANGE >= current_slot >= aggregate.data.slot` (a client MAY queue future aggregates for processing at the appropriate slot). - The aggregate attestation defined by `hash_tree_root(aggregate)` has _not_ already been seen (via aggregate gossip, within a verified block, or through the creation of an equivalent aggregate locally). - - The `aggregate` is the first valid aggregate received for the aggregator with index `aggregate_and_proof.aggregator_index` for the slot `aggregate.data.slot`. + - The `aggregate` is the first valid aggregate received for the aggregator with index `aggregate_and_proof.aggregator_index` for the epoch `aggregate.data.target.epoch`. - The block being voted for (`aggregate.data.beacon_block_root`) passes validation. - `aggregate_and_proof.selection_proof` selects the validator as an aggregator for the slot -- i.e. `is_aggregator(state, aggregate.data.slot, aggregate.data.index, aggregate_and_proof.selection_proof)` returns `True`. - The aggregator's validator index is within the aggregate's committee -- i.e. `aggregate_and_proof.aggregator_index in get_attesting_indices(state, aggregate.data, aggregate.aggregation_bits)`. From 79d6b49a904cbdba4de9b8c98eba649f381f93d4 Mon Sep 17 00:00:00 2001 From: Giuseppe Bertone Date: Fri, 10 Apr 2020 17:38:37 +0200 Subject: [PATCH 07/44] Fixed target compile_deposit_contract Path of validator_registration.vy contract was wrong --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index e8f3d21bc5..e53aaf8a2a 100644 --- a/Makefile +++ b/Makefile @@ -117,7 +117,7 @@ install_deposit_contract_compiler: compile_deposit_contract: cd $(DEPOSIT_CONTRACT_COMPILER_DIR); . venv/bin/activate; \ - python3.7 deposit_contract/compile.py contracts/validator_registration.vy + python3.7 deposit_contract/compile.py ../contracts/validator_registration.vy test_compile_deposit_contract: cd $(DEPOSIT_CONTRACT_COMPILER_DIR); . venv/bin/activate; \ From e58cfedb6830d54ab22b08db56a1773f4b0d3797 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Thu, 16 Apr 2020 11:12:24 -0600 Subject: [PATCH 08/44] clarify ssz_snappy for gossip --- specs/phase0/p2p-interface.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index c461c2d3b9..7d733e48e8 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -345,7 +345,9 @@ Topics are post-fixed with an encoding. Encodings define how the payload of a go #### Mainnet -- `ssz_snappy` - All objects are SSZ-encoded and then compressed with [Snappy](https://github.com/google/snappy). Example: The beacon aggregate attestation topic string is `/eth2/beacon_aggregate_and_proof/ssz_snappy`, and the data field of a gossipsub message is an `AggregateAndProof` that has been SSZ-encoded and then compressed with Snappy. +- `ssz_snappy` - All objects are SSZ-encoded and then compressed with [Snappy](https://github.com/google/snappy) block compression. Example: The beacon aggregate attestation topic string is `/eth2/beacon_aggregate_and_proof/ssz_snappy`, and the data field of a gossipsub message is an `AggregateAndProof` that has been SSZ-encoded and then compressed with Snappy. + +Snappy has two formats: "block" and "frames" (streaming). Gossip messages remain relatively small (100s of bytes to 100s of kilobytes) so [basic snappy block compression](https://github.com/google/snappy/blob/master/format_description.txt) is used to avoid the additional overhead associated with snappy frames. Implementations MUST use a single encoding. Changing an encoding will require coordination between participating implementations. @@ -448,7 +450,7 @@ Here, `result` represents the 1-byte response code. The token of the negotiated protocol ID specifies the type of encoding to be used for the req/resp interaction. Two values are possible at this time: - `ssz`: the contents are [SSZ-encoded](../../ssz/simple-serialize.md). This encoding type MUST be supported by all clients. For objects containing a single field, only the field is SSZ-encoded not a container with a single field. For example, the `BeaconBlocksByRoot` request is an SSZ-encoded list of `Root`'s. -- `ssz_snappy`: The contents are SSZ-encoded and then compressed with [Snappy](https://github.com/google/snappy). MAY be supported in the interoperability testnet; MUST be supported in mainnet. +- `ssz_snappy`: The contents are SSZ-encoded and then compressed with [Snappy](https://github.com/google/snappy) frames compression. MAY be supported in the interoperability testnet; MUST be supported in mainnet. #### SSZ-encoding strategy (with or without Snappy) From 4915014a19bb8d67efac833ecb065072846e6a5d Mon Sep 17 00:00:00 2001 From: Jacek Sieka Date: Mon, 20 Apr 2020 20:01:57 +0200 Subject: [PATCH 09/44] simplify block range request description There's room for ambiguity as to what `count` means - this clarifies that it always relates to the slot, and not the number of blocks in the response which allows clients to request ranges epoch by epoch (for example) without worrying about overlaps caused by empty slots. --- specs/phase0/p2p-interface.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index 7d733e48e8..eff6a7c4f3 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -463,7 +463,7 @@ Snappy has two formats: "block" and "frames" (streaming). To support large reque Since snappy frame contents [have a maximum size of `65536` bytes](https://github.com/google/snappy/blob/master/framing_format.txt#L104) and frame headers are just `identifier (1) + checksum (4)` bytes, the expected buffering of a single frame is acceptable. -**Encoding-dependent header:** Req/Resp protocols using the `ssz` or `ssz_snappy` encoding strategies MUST encode the length of the raw SSZ bytes, encoded as an unsigned [protobuf varint](https://developers.google.com/protocol-buffers/docs/encoding#varints). +**Encoding-dependent header:** Req/Resp protocols using the `ssz` or `ssz_snappy` encoding strategies MUST encode the length of the raw SSZ bytes, encoded as an unsigned [protobuf varint](https://developers.google.com/protocol-buffers/docs/encoding#varints). *Writing*: By first computing and writing the SSZ byte length, the SSZ encoder can then directly write the chunk contents to the stream. If Snappy is applied, it can be passed through a buffered Snappy writer to compress frame by frame. @@ -577,7 +577,7 @@ Response Content: ) ``` -Requests count beacon blocks from the peer starting from `start_slot`, leading up to the current head block as selected by fork choice. `step` defines the slot increment between blocks. For example, requesting blocks starting at `start_slot` 2 with a step value of 2 would return the blocks at slots [2, 4, 6, …]. In cases where a slot is empty for a given slot number, no block is returned. For example, if slot 4 were empty in the previous example, the returned array would contain [2, 6, …]. A step value of 1 returns all blocks on the range `[start_slot, start_slot + count)`. +Requests beacon blocks in the slot range `[start_slot, start_slot + count * step)`, leading up to the current head block as selected by fork choice. `step` defines the slot increment between blocks. For example, requesting blocks starting at `start_slot` 2 with a step value of 2 would return the blocks at slots [2, 4, 6, …]. In cases where a slot is empty for a given slot number, no block is returned. For example, if slot 4 were empty in the previous example, the returned array would contain [2, 6, …]. `BeaconBlocksByRange` is primarily used to sync historical blocks. From 508811d6417640c10d4866b12965bbe9e1ed6c13 Mon Sep 17 00:00:00 2001 From: Justin Date: Tue, 21 Apr 2020 08:50:42 +0100 Subject: [PATCH 10/44] =?UTF-8?q?Fix=20#1735=E2=80=94remove=20redundant=20?= =?UTF-8?q?check?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit As per #1735 the check `if not len(indices) <= MAX_VALIDATORS_PER_COMMITTEE: return False` is redundant. As such this PR should be purely cosmetic. --- specs/phase0/beacon-chain.md | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/specs/phase0/beacon-chain.md b/specs/phase0/beacon-chain.md index 23fa5ceee2..a093c1098e 100644 --- a/specs/phase0/beacon-chain.md +++ b/specs/phase0/beacon-chain.md @@ -684,14 +684,10 @@ def is_slashable_attestation_data(data_1: AttestationData, data_2: AttestationDa ```python def is_valid_indexed_attestation(state: BeaconState, indexed_attestation: IndexedAttestation) -> bool: """ - Check if ``indexed_attestation`` has valid indices and signature. + Check if ``indexed_attestation`` has sorted and unique indices and a valid aggregate signature. """ - indices = indexed_attestation.attesting_indices - - # Verify max number of indices - if not len(indices) <= MAX_VALIDATORS_PER_COMMITTEE: - return False # Verify indices are sorted and unique + indices = indexed_attestation.attesting_indices if not indices == sorted(set(indices)): return False # Verify aggregate signature From e2a320ef32f0a7567a7d62008a35c2c4bcd9c312 Mon Sep 17 00:00:00 2001 From: Justin Date: Tue, 21 Apr 2020 08:59:53 +0100 Subject: [PATCH 11/44] Partial fix for #1701 Clarify that state transitions with `uint64` overflows are invalid. --- specs/phase0/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/phase0/beacon-chain.md b/specs/phase0/beacon-chain.md index 23fa5ceee2..01bf36c993 100644 --- a/specs/phase0/beacon-chain.md +++ b/specs/phase0/beacon-chain.md @@ -1195,7 +1195,7 @@ Let `genesis_block = BeaconBlock(state_root=hash_tree_root(genesis_state))`. ## Beacon chain state transition function -The post-state corresponding to a pre-state `state` and a signed block `signed_block` is defined as `state_transition(state, signed_block)`. State transitions that trigger an unhandled exception (e.g. a failed `assert` or an out-of-range list access) are considered invalid. +The post-state corresponding to a pre-state `state` and a signed block `signed_block` is defined as `state_transition(state, signed_block)`. State transitions that trigger an unhandled exception (e.g. a failed `assert` or an out-of-range list access) are considered invalid. State transitions that cause a `uint64` overflow are also considered invalid. ```python def state_transition(state: BeaconState, signed_block: SignedBeaconBlock, validate_result: bool=True) -> BeaconState: From 3436021e72019349f64910822ef862cddac086d4 Mon Sep 17 00:00:00 2001 From: Justin Date: Tue, 21 Apr 2020 15:34:55 +0100 Subject: [PATCH 12/44] Update beacon-chain.md --- specs/phase0/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/phase0/beacon-chain.md b/specs/phase0/beacon-chain.md index 01bf36c993..87b2076866 100644 --- a/specs/phase0/beacon-chain.md +++ b/specs/phase0/beacon-chain.md @@ -1195,7 +1195,7 @@ Let `genesis_block = BeaconBlock(state_root=hash_tree_root(genesis_state))`. ## Beacon chain state transition function -The post-state corresponding to a pre-state `state` and a signed block `signed_block` is defined as `state_transition(state, signed_block)`. State transitions that trigger an unhandled exception (e.g. a failed `assert` or an out-of-range list access) are considered invalid. State transitions that cause a `uint64` overflow are also considered invalid. +The post-state corresponding to a pre-state `state` and a signed block `signed_block` is defined as `state_transition(state, signed_block)`. State transitions that trigger an unhandled exception (e.g. a failed `assert` or an out-of-range list access) are considered invalid. State transitions that cause a `uint64` overflow or underflow are also considered invalid. ```python def state_transition(state: BeaconState, signed_block: SignedBeaconBlock, validate_result: bool=True) -> BeaconState: From 3d4122a2f6010d393f52f63d62068ce0641068b9 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Thu, 16 Apr 2020 11:56:15 -0600 Subject: [PATCH 13/44] add note about distributing bootnode ENRs prior to genesis --- .circleci/config.yml | 22 +++++++++++----------- specs/phase0/p2p-interface.md | 2 +- 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 5c4b77e784..3a67e55281 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -79,16 +79,16 @@ jobs: # Restore git repo at point close to target branch/revision, to speed up checkout - restore_cache: keys: - - v2-specs-repo-{{ .Branch }}-{{ .Revision }} - - v2-specs-repo-{{ .Branch }}- - - v2-specs-repo- + - v3-specs-repo-{{ .Branch }}-{{ .Revision }} + - v3-specs-repo-{{ .Branch }}- + - v3-specs-repo- - checkout - run: name: Clean up git repo to reduce cache size command: git gc # Save the git checkout as a cache, to make cloning next time faster. - save_cache: - key: v2-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} paths: - ~/specs-repo install_pyspec_test: @@ -97,7 +97,7 @@ jobs: working_directory: ~/specs-repo steps: - restore_cache: - key: v2-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_pyspec_cached_venv - run: name: Install pyspec requirements @@ -109,7 +109,7 @@ jobs: working_directory: ~/specs-repo steps: - restore_cache: - key: v2-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_pyspec_cached_venv - run: name: Run py-tests @@ -140,7 +140,7 @@ jobs: working_directory: ~/specs-repo steps: - restore_cache: - key: v2-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_pyspec_cached_venv - run: name: Run linter @@ -152,7 +152,7 @@ jobs: working_directory: ~/specs-repo steps: - restore_cache: - key: v2-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_deposit_contract_compiler_cached_venv - run: name: Install deposit contract compiler requirements @@ -164,7 +164,7 @@ jobs: working_directory: ~/specs-repo steps: - restore_cache: - key: v2-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_deposit_contract_tester_cached_venv - run: name: Install deposit contract tester requirements @@ -176,7 +176,7 @@ jobs: working_directory: ~/specs-repo steps: - restore_cache: - key: v2-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_deposit_contract_compiler_cached_venv - run: name: Run deposit contract compile test @@ -187,7 +187,7 @@ jobs: working_directory: ~/specs-repo steps: - restore_cache: - key: v2-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_deposit_contract_tester_cached_venv - run: name: Run deposit contract test diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index c461c2d3b9..9573eef119 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -755,7 +755,7 @@ where the fields of `ENRForkID` are defined as * `next_fork_version` is the fork version corresponding to the next planned hard fork at a future epoch. If no future fork is planned, set `next_fork_version = current_fork_version` to signal this fact * `next_fork_epoch` is the epoch at which the next fork is planned and the `current_fork_version` will be updated. If no future fork is planned, set `next_fork_epoch = FAR_FUTURE_EPOCH` to signal this fact -*Note*: `fork_digest` is composed of values that are not not known until the genesis block/state are available. Due to this, clients SHOULD NOT form ENRs and begin peer discovery until genesis values are known. +*Note*: `fork_digest` is composed of values that are not not known until the genesis block/state are available. Due to this, clients SHOULD NOT form ENRs and begin peer discovery until genesis values are known. One notable exception to this rule is the distribution of bootnode ENRs prior to genesis. In this case, bootnode ENRs SHOULD be initially distributed with `eth2` field set as `ENRForkID(fork_digest=compute_fork_digest(GENESIS_FORK_VERSION, b'\x00'*32), next_fork_version=GENESIS_FORK_VERSION, next_fork_epoch=FAR_FUTURE_EPOCH)`. After genesis values are known, the bootnodes SHOULD update ENRs to participate in normal discovery operations. Clients SHOULD connect to peers with `fork_digest`, `next_fork_version`, and `next_fork_epoch` that match local values. From 11d164748c58580a34edb798a91387b720e45701 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 22 Apr 2020 14:45:01 -0600 Subject: [PATCH 14/44] add 'valid' when de-deduplication of attestations on gossip subnets --- specs/phase0/p2p-interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index c66ec0b6ec..d3b9150d17 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -319,7 +319,7 @@ Attestation subnets are used to propagate unaggregated attestations to subsectio - The attestation's committee index (`attestation.data.index`) is for the correct subnet. - `attestation.data.slot` is within the last `ATTESTATION_PROPAGATION_SLOT_RANGE` slots (within a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) -- i.e. `attestation.data.slot + ATTESTATION_PROPAGATION_SLOT_RANGE >= current_slot >= attestation.data.slot` (a client MAY queue future attestations for processing at the appropriate slot). - The attestation is unaggregated -- that is, it has exactly one participating validator (`len([bit for bit in attestation.aggregation_bits if bit == 0b1]) == 1`). - - There has been no other attestation seen on an attestation subnet that has an identical `attestation.data.target.epoch` and participating validator index. + - There has been no other valid attestation seen on an attestation subnet that has an identical `attestation.data.target.epoch` and participating validator index. - The block being voted for (`attestation.data.beacon_block_root`) passes validation. - The signature of `attestation` is valid. From bf806b9efaebf957d1399adac4bd839a58db537c Mon Sep 17 00:00:00 2001 From: Paul Hauner Date: Fri, 24 Apr 2020 15:01:18 +1000 Subject: [PATCH 15/44] Require "seen" aggregates to be valid --- specs/phase0/p2p-interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index d3b9150d17..92194920bf 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -289,7 +289,7 @@ There are two primary global topics used to propagate beacon blocks and aggregat - The block is proposed by the expected `proposer_index` for the block's slot in the context of the current shuffling (defined by `parent_root`/`slot`). If the `proposer_index` cannot immediately be verified against the expected shuffling, the block MAY be queued for later processing while proposers for the block's branch are calculated. - `beacon_aggregate_and_proof` - This topic is used to propagate aggregated attestations (as `SignedAggregateAndProof`s) to subscribing nodes (typically validators) to be included in future blocks. The following validations MUST pass before forwarding the `signed_aggregate_and_proof` on the network. (We define the following for convenience -- `aggregate_and_proof = signed_aggregate_and_proof.message` and `aggregate = aggregate_and_proof.aggregate`) - `aggregate.data.slot` is within the last `ATTESTATION_PROPAGATION_SLOT_RANGE` slots (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) -- i.e. `aggregate.data.slot + ATTESTATION_PROPAGATION_SLOT_RANGE >= current_slot >= aggregate.data.slot` (a client MAY queue future aggregates for processing at the appropriate slot). - - The aggregate attestation defined by `hash_tree_root(aggregate)` has _not_ already been seen (via aggregate gossip, within a verified block, or through the creation of an equivalent aggregate locally). + - The valid aggregate attestation defined by `hash_tree_root(aggregate)` has _not_ already been seen (via aggregate gossip, within a verified block, or through the creation of an equivalent aggregate locally). - The `aggregate` is the first valid aggregate received for the aggregator with index `aggregate_and_proof.aggregator_index` for the epoch `aggregate.data.target.epoch`. - The block being voted for (`aggregate.data.beacon_block_root`) passes validation. - `aggregate_and_proof.selection_proof` selects the validator as an aggregator for the slot -- i.e. `is_aggregator(state, aggregate.data.slot, aggregate.data.index, aggregate_and_proof.selection_proof)` returns `True`. From 56535e3dbe6127728e37fb6c871e790eaa6dc96b Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Fri, 24 Apr 2020 10:21:47 -0600 Subject: [PATCH 16/44] bump version to v0.11.2 --- tests/core/pyspec/eth2spec/VERSION.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/core/pyspec/eth2spec/VERSION.txt b/tests/core/pyspec/eth2spec/VERSION.txt index 027934ea1a..a8839f70de 100644 --- a/tests/core/pyspec/eth2spec/VERSION.txt +++ b/tests/core/pyspec/eth2spec/VERSION.txt @@ -1 +1 @@ -0.11.1 \ No newline at end of file +0.11.2 \ No newline at end of file From 2129f8a281ce01e722c1e7026ee938b35c64ac8c Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Fri, 24 Apr 2020 16:00:06 -0600 Subject: [PATCH 17/44] fix requirements.txt for bls gens --- tests/generators/bls/requirements.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/generators/bls/requirements.txt b/tests/generators/bls/requirements.txt index 7b1d2f4a9d..24ea127c47 100644 --- a/tests/generators/bls/requirements.txt +++ b/tests/generators/bls/requirements.txt @@ -1,3 +1,4 @@ py_ecc==2.0.0 eth-utils==1.6.0 ../../core/gen_helpers +../../../ From 0c67aaa68e83d0050d4791b82d1dfce3943eef09 Mon Sep 17 00:00:00 2001 From: protolambda Date: Sat, 25 Apr 2020 00:05:37 +0200 Subject: [PATCH 18/44] Include fork digest in example gossip topic name --- specs/phase0/p2p-interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index d3b9150d17..7197581dc8 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -345,7 +345,7 @@ Topics are post-fixed with an encoding. Encodings define how the payload of a go #### Mainnet -- `ssz_snappy` - All objects are SSZ-encoded and then compressed with [Snappy](https://github.com/google/snappy) block compression. Example: The beacon aggregate attestation topic string is `/eth2/beacon_aggregate_and_proof/ssz_snappy`, and the data field of a gossipsub message is an `AggregateAndProof` that has been SSZ-encoded and then compressed with Snappy. +- `ssz_snappy` - All objects are SSZ-encoded and then compressed with [Snappy](https://github.com/google/snappy) block compression. Example: The beacon aggregate attestation topic string is `/eth2/446a7232/beacon_aggregate_and_proof/ssz_snappy`, the fork digest is `446a7232` and the data field of a gossipsub message is an `AggregateAndProof` that has been SSZ-encoded and then compressed with Snappy. Snappy has two formats: "block" and "frames" (streaming). Gossip messages remain relatively small (100s of bytes to 100s of kilobytes) so [basic snappy block compression](https://github.com/google/snappy/blob/master/format_description.txt) is used to avoid the additional overhead associated with snappy frames. From 1a81c873af99397d4f0cc69eb2a462d037d01b54 Mon Sep 17 00:00:00 2001 From: Paul Hauner Date: Sun, 26 Apr 2020 16:24:16 +1000 Subject: [PATCH 19/44] Remove redundant check in fork choice --- specs/phase0/fork-choice.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/specs/phase0/fork-choice.md b/specs/phase0/fork-choice.md index 18c7a15806..35e2c5f567 100644 --- a/specs/phase0/fork-choice.md +++ b/specs/phase0/fork-choice.md @@ -278,8 +278,6 @@ def validate_on_attestation(store: Store, attestation: Attestation) -> None: # Attestations target be for a known block. If target block is unknown, delay consideration until the block is found assert target.root in store.blocks - # Attestations cannot be from future epochs. If they are, delay consideration until the epoch arrives - assert get_current_slot(store) >= compute_start_slot_at_epoch(target.epoch) # Attestations must be for a known block. If block is unknown, delay consideration until the block is found assert attestation.data.beacon_block_root in store.blocks From c841aa102bad17c0e06d0c1bb033d44591cad619 Mon Sep 17 00:00:00 2001 From: Raw Pong Ghmoa <58883403+q9f@users.noreply.github.com> Date: Sun, 26 Apr 2020 10:09:22 +0200 Subject: [PATCH 20/44] genesis: clarify that eth1 timestamp can be less than min genesis time --- specs/phase0/beacon-chain.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/specs/phase0/beacon-chain.md b/specs/phase0/beacon-chain.md index cdf38dc1e4..3ef4081ce0 100644 --- a/specs/phase0/beacon-chain.md +++ b/specs/phase0/beacon-chain.md @@ -1170,6 +1170,8 @@ def initialize_beacon_state_from_eth1(eth1_block_hash: Bytes32, return state ``` +*Note*: The ETH1 block with `eth1_timestamp` meeting the minimum genesis active validator count criteria can also occur before `MIN_GENESIS_TIME`. + ### Genesis state Let `genesis_state = candidate_state` whenever `is_valid_genesis_state(candidate_state) is True` for the first time. From 3cc1fb901760f1c7ab35bcb69f4ce7c5dce78180 Mon Sep 17 00:00:00 2001 From: terence tsao Date: Mon, 27 Apr 2020 14:34:50 -0700 Subject: [PATCH 21/44] Remove `/` --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index b6e25d5701..fed65eedb6 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ This repository hosts the current Eth2 specifications. Discussions about design ## Specs -Core specifications for Eth2 clients be found in [specs/](specs/). These are divided into phases. Each subsequent phase depends upon the prior. The current phases specified are: +Core specifications for Eth2 clients be found in [specs](specs/). These are divided into phases. Each subsequent phase depends upon the prior. The current phases specified are: ### Phase 0 * [The Beacon Chain](specs/phase0/beacon-chain.md) From fa66475da46d334ed2214a7b72143829af9cafab Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 29 Apr 2020 00:04:44 +0800 Subject: [PATCH 22/44] Move `sanity` to under `phase_0` --- .../core/pyspec/eth2spec/test/{ => phase_0}/sanity/__init__.py | 0 .../pyspec/eth2spec/test/{ => phase_0}/sanity/test_blocks.py | 0 .../pyspec/eth2spec/test/{ => phase_0}/sanity/test_slots.py | 0 tests/generators/sanity/main.py | 2 +- 4 files changed, 1 insertion(+), 1 deletion(-) rename tests/core/pyspec/eth2spec/test/{ => phase_0}/sanity/__init__.py (100%) rename tests/core/pyspec/eth2spec/test/{ => phase_0}/sanity/test_blocks.py (100%) rename tests/core/pyspec/eth2spec/test/{ => phase_0}/sanity/test_slots.py (100%) diff --git a/tests/core/pyspec/eth2spec/test/sanity/__init__.py b/tests/core/pyspec/eth2spec/test/phase_0/sanity/__init__.py similarity index 100% rename from tests/core/pyspec/eth2spec/test/sanity/__init__.py rename to tests/core/pyspec/eth2spec/test/phase_0/sanity/__init__.py diff --git a/tests/core/pyspec/eth2spec/test/sanity/test_blocks.py b/tests/core/pyspec/eth2spec/test/phase_0/sanity/test_blocks.py similarity index 100% rename from tests/core/pyspec/eth2spec/test/sanity/test_blocks.py rename to tests/core/pyspec/eth2spec/test/phase_0/sanity/test_blocks.py diff --git a/tests/core/pyspec/eth2spec/test/sanity/test_slots.py b/tests/core/pyspec/eth2spec/test/phase_0/sanity/test_slots.py similarity index 100% rename from tests/core/pyspec/eth2spec/test/sanity/test_slots.py rename to tests/core/pyspec/eth2spec/test/phase_0/sanity/test_slots.py diff --git a/tests/generators/sanity/main.py b/tests/generators/sanity/main.py index 74c85a9e8c..89fb89838b 100644 --- a/tests/generators/sanity/main.py +++ b/tests/generators/sanity/main.py @@ -5,7 +5,7 @@ from gen_from_tests.gen import generate_from_tests from eth2spec.test.context import PHASE0 -from eth2spec.test.sanity import test_blocks, test_slots +from eth2spec.test.phase_0.sanity import test_blocks, test_slots from eth2spec.config import config_util from eth2spec.phase0 import spec as spec_phase0 from eth2spec.phase1 import spec as spec_phase1 From 6a40f71a31724aab57ca90ff0a00fc5a536f62d6 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 29 Apr 2020 20:29:48 -0600 Subject: [PATCH 23/44] add note about beacon committees not going into attnets --- specs/phase0/validator.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/specs/phase0/validator.md b/specs/phase0/validator.md index bc75104032..69be712796 100644 --- a/specs/phase0/validator.md +++ b/specs/phase0/validator.md @@ -524,6 +524,8 @@ Because Phase 0 does not have shards and thus does not have Shard Committees, th * Maintain advertisement of the randomly selected subnets in their node's ENR `attnets` entry by setting the randomly selected `subnet_id` bits to `True` (e.g. `ENR["attnets"][subnet_id] = True`) for all persistent attestation subnets * Set the lifetime of each random subscription to a random number of epochs between `EPOCHS_PER_RANDOM_SUBNET_SUBSCRIPTION` and `2 * EPOCHS_PER_RANDOM_SUBNET_SUBSCRIPTION]`. At the end of life for a subscription, select a new random subnet, update subnet subscriptions, and publish an updated ENR +*Note*: Short lived beacon committee assignments should not be added in into the ENR `attnets` entry. + *Note*: When preparing for a hard fork, a validator must select and subscribe to random subnets of the future fork versioning at least `EPOCHS_PER_RANDOM_SUBNET_SUBSCRIPTION` epochs in advance of the fork. These new subnets for the fork are maintained in addition to those for the current fork until the fork occurs. After the fork occurs, let the subnets from the previous fork reach the end of life with no replacements. ## How to avoid slashing From 2dc515665181c5cb30e60148fb3be57073cb3a77 Mon Sep 17 00:00:00 2001 From: Paul Hauner Date: Thu, 30 Apr 2020 16:27:02 +1000 Subject: [PATCH 24/44] Add message about delaying consideration --- specs/phase0/fork-choice.md | 1 + 1 file changed, 1 insertion(+) diff --git a/specs/phase0/fork-choice.md b/specs/phase0/fork-choice.md index 35e2c5f567..60d398dd58 100644 --- a/specs/phase0/fork-choice.md +++ b/specs/phase0/fork-choice.md @@ -273,6 +273,7 @@ def validate_on_attestation(store: Store, attestation: Attestation) -> None: current_epoch = compute_epoch_at_slot(get_current_slot(store)) # Use GENESIS_EPOCH for previous when genesis to avoid underflow previous_epoch = current_epoch - 1 if current_epoch > GENESIS_EPOCH else GENESIS_EPOCH + # If attestation target is from a future epoch, delay consideration until the epoch arrives assert target.epoch in [current_epoch, previous_epoch] assert target.epoch == compute_epoch_at_slot(attestation.data.slot) From feb27a14be1ae2b64a96f0469ffaf0bd68223c67 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Tue, 31 Mar 2020 12:37:36 +0800 Subject: [PATCH 25/44] beacon-chain.md: Replace block wrapper with signable pattern --- specs/phase1/beacon-chain.md | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/specs/phase1/beacon-chain.md b/specs/phase1/beacon-chain.md index 596b3818f8..d90afda16a 100644 --- a/specs/phase1/beacon-chain.md +++ b/specs/phase1/beacon-chain.md @@ -24,8 +24,9 @@ - [Extended `SignedBeaconBlock`](#extended-signedbeaconblock) - [Extended `BeaconState`](#extended-beaconstate) - [New containers](#new-containers) - - [`ShardBlockWrapper`](#shardblockwrapper) - - [`ShardSignableHeader`](#shardsignableheader) + - [`ShardBlock`](#shardblock) + - [`SignedShardBlock`](#signedshardblock) + - [`ShardBlockHeader`](#shardblockheader) - [`ShardState`](#shardstate) - [`ShardTransition`](#shardtransition) - [`CompactCommittee`](#compactcommittee) @@ -291,23 +292,28 @@ class BeaconState(Container): The following containers are new in Phase 1. -### `ShardBlockWrapper` - -_Wrapper for being broadcasted over the network._ +### `ShardBlock` ```python -class ShardBlockWrapper(Container): +class ShardBlock(Container): shard_parent_root: Root beacon_parent_root: Root slot: Slot body: ByteList[MAX_SHARD_BLOCK_SIZE] +``` + +### `SignedShardBlock` + +```python +class SignedShardBlock(Container): + message: ShardBlock signature: BLSSignature ``` -### `ShardSignableHeader` +### `ShardBlockHeader` ```python -class ShardSignableHeader(Container): +class ShardBlockHeader(Container): shard_parent_root: Root beacon_parent_root: Root slot: Slot @@ -700,7 +706,7 @@ def apply_shard_transition(state: BeaconState, shard: Shard, transition: ShardTr shard_parent_root = state.shard_states[shard].latest_block_root for i in range(len(offset_slots)): if any(transition.shard_data_roots): - headers.append(ShardSignableHeader( + headers.append(ShardBlockHeader( shard_parent_root=shard_parent_root, parent_hash=get_block_root_at_slot(state, get_previous_slot(state.slot)), slot=offset_slots[i], From e9f1e4186d0e6cf70f32e68cd97f33a61eec5a2e Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 2 Apr 2020 10:43:50 +0800 Subject: [PATCH 26/44] Add `proposer_index` to shard block --- specs/phase1/beacon-chain.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/specs/phase1/beacon-chain.md b/specs/phase1/beacon-chain.md index d90afda16a..0652fecada 100644 --- a/specs/phase1/beacon-chain.md +++ b/specs/phase1/beacon-chain.md @@ -299,6 +299,7 @@ class ShardBlock(Container): shard_parent_root: Root beacon_parent_root: Root slot: Slot + proposer_index: ValidatorIndex body: ByteList[MAX_SHARD_BLOCK_SIZE] ``` @@ -317,6 +318,7 @@ class ShardBlockHeader(Container): shard_parent_root: Root beacon_parent_root: Root slot: Slot + proposer_index: ValidatorIndex body_root: Root ``` From 5f69afea382740c696052a11e939329f46810f82 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 2 Apr 2020 10:53:46 +0800 Subject: [PATCH 27/44] Make `shard_state_transition` more like beacon state_transition function --- specs/phase1/fraud-proofs.md | 30 +++++++++++++++++++++++------- 1 file changed, 23 insertions(+), 7 deletions(-) diff --git a/specs/phase1/fraud-proofs.md b/specs/phase1/fraud-proofs.md index 0688f5f47a..19077ba88c 100644 --- a/specs/phase1/fraud-proofs.md +++ b/specs/phase1/fraud-proofs.md @@ -45,14 +45,30 @@ The proof verifies that one of the two conditions is false: ## Shard state transition function ```python -def shard_state_transition(shard: Shard, +def shard_state_transition(beacon_state: BeaconState, + shard: Shard, slot: Slot, pre_state: Root, previous_beacon_root: Root, - proposer_pubkey: BLSPubkey, - block_data: ByteList[MAX_SHARD_BLOCK_SIZE]) -> Root: + proposer_index: ValidatorIndex, + signed_block: SignedShardBlock, + validate_result: bool=True) -> Root: # We will add something more substantive in phase 2 - return hash(pre_state + hash_tree_root(previous_beacon_root) + hash_tree_root(block_data)) + + # Verify the proposer_index and signature + assert proposer_index == signed_block.message.proposer_index + if validate_result: + assert verify_shard_block_signature(beacon_state, signed_block) + + return hash(pre_state + hash_tree_root(previous_beacon_root) + hash_tree_root(signed_block.message.data)) +``` + +```python +def verify_shard_block_signature(beacon_state: BeaconState, + signed_block: SignedShardBlock) -> bool: + proposer = beacon_state.validators[signed_block.message.proposer_index] + signing_root = compute_signing_root(signed_block.message, get_domain(beacon_state, DOMAIN_SHARD_PROPOSAL)) + return bls.Verify(proposer.pubkey, signing_root, signed_block.signature) ``` ## Honest committee member behavior @@ -61,10 +77,10 @@ Suppose you are a committee member on shard `shard` at slot `current_slot`. Let * Initialize `proposals = []`, `shard_states = []`, `shard_state = state.shard_states[shard][-1]`, `start_slot = shard_state.slot`. * For `slot in get_offset_slots(state, start_slot)`, do the following: - * Look for all valid proposals for `slot`; that is, a Bytes `proposal` where `shard_state_transition(shard, slot, shard_state, get_block_root_at_slot(state, state.slot - 1), get_shard_proposer_index(state, shard, slot), proposal)` returns a result and does not throw an exception. Let `choices` be the set of non-empty valid proposals you discover. + * Look for all valid proposals for `slot`; that is, a SignedShardBlock `proposal` where `shard_state_transition(shard, slot, shard_state, get_block_root_at_slot(state, state.slot - 1), get_shard_proposer_index(state, slot, shard), proposal)` returns a result and does not throw an exception. Let `choices` be the set of non-empty valid proposals you discover. * If `len(choices) == 0`, do `proposals.append(make_empty_proposal(shard_state, slot))` * If `len(choices) == 1`, do `proposals.append(choices[0])` * If `len(choices) > 1`, let `winning_proposal` be the proposal with the largest number of total attestations from slots in `state.shard_next_slots[shard]....slot-1` supporting it or any of its descendants, breaking ties by choosing the first proposal locally seen. Do `proposals.append(winning_proposal)`. - * If `proposals[-1]` is NOT an empty proposal, set `shard_state = shard_state_transition(shard, slot, shard_state, get_block_root_at_slot(state, state.slot - 1), get_shard_proposer_index(state, shard, slot), proposals[-1])` and do `shard_states.append(shard_state)`. If it is an empty proposal, leave `shard_state` unchanged. + * If `proposals.message.data[-1]` is NOT an empty proposal, set `shard_state = shard_state_transition(shard, slot, shard_state, get_block_root_at_slot(state, state.slot - 1), get_shard_proposer_index(state, slot, shard), proposals[-1])` and do `shard_states.append(shard_state)`. If it is an empty proposal, leave `shard_state` unchanged. -Make an attestation using `shard_data_roots = [hash_tree_root(proposal) for proposal in proposals]` and `shard_state_roots = shard_states`. +Make an attestation using `shard_data_roots = [hash_tree_root(proposal.message.data) for proposal in proposals]` and `shard_state_roots = shard_states`. From be50020bf8134b82c5d7e46add6713f193ccddde Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 2 Apr 2020 11:20:22 +0800 Subject: [PATCH 28/44] Refactor `get_light_client_committee` to similar to `get_shard_committee` --- specs/phase1/beacon-chain.md | 38 +++++++++++++++++++++++------------- 1 file changed, 24 insertions(+), 14 deletions(-) diff --git a/specs/phase1/beacon-chain.md b/specs/phase1/beacon-chain.md index 0652fecada..fbfdce70ef 100644 --- a/specs/phase1/beacon-chain.md +++ b/specs/phase1/beacon-chain.md @@ -43,8 +43,8 @@ - [`get_active_shard_count`](#get_active_shard_count) - [`get_online_validator_indices`](#get_online_validator_indices) - [`get_shard_committee`](#get_shard_committee) - - [`get_shard_proposer_index`](#get_shard_proposer_index) - [`get_light_client_committee`](#get_light_client_committee) + - [`get_shard_proposer_index`](#get_shard_proposer_index) - [`get_indexed_attestation`](#get_indexed_attestation) - [`get_updated_gasprice`](#get_updated_gasprice) - [`get_start_shard`](#get_start_shard) @@ -465,29 +465,39 @@ def get_shard_committee(beacon_state: BeaconState, epoch: Epoch, shard: Shard) - source_epoch -= SHARD_COMMITTEE_PERIOD active_validator_indices = get_active_validator_indices(beacon_state, source_epoch) seed = get_seed(beacon_state, source_epoch, DOMAIN_SHARD_COMMITTEE) - return compute_committee(active_validator_indices, seed, shard, get_active_shard_count(beacon_state)) -``` - -#### `get_shard_proposer_index` - -```python -def get_shard_proposer_index(beacon_state: BeaconState, slot: Slot, shard: Shard) -> ValidatorIndex: - committee = get_shard_committee(beacon_state, compute_epoch_at_slot(slot), shard) - r = bytes_to_int(get_seed(beacon_state, get_current_epoch(beacon_state), DOMAIN_SHARD_COMMITTEE)[:8]) - return committee[r % len(committee)] + return compute_committee( + indices=active_validator_indices, + seed=seed, + index=shard, + count=get_active_shard_count(beacon_state) + ) ``` #### `get_light_client_committee` ```python def get_light_client_committee(beacon_state: BeaconState, epoch: Epoch) -> Sequence[ValidatorIndex]: - source_epoch = epoch - epoch % LIGHT_CLIENT_COMMITTEE_PERIOD + source_epoch = epoch - epoch % LIGHT_CLIENT_COMMITTEE_PERIOD if source_epoch > 0: source_epoch -= LIGHT_CLIENT_COMMITTEE_PERIOD active_validator_indices = get_active_validator_indices(beacon_state, source_epoch) seed = get_seed(beacon_state, source_epoch, DOMAIN_LIGHT_CLIENT) - active_shards = get_active_shard_count(beacon_state) - return compute_committee(active_validator_indices, seed, 0, active_shards)[:TARGET_COMMITTEE_SIZE] + active_shards_count = get_active_shard_count(beacon_state) + return compute_committee( + indices=active_validator_indices, + seed=seed, + index=0, + count=active_shards_count, + )[:TARGET_COMMITTEE_SIZE] +``` + +#### `get_shard_proposer_index` + +```python +def get_shard_proposer_index(beacon_state: BeaconState, slot: Slot, shard: Shard) -> ValidatorIndex: + committee = get_shard_committee(beacon_state, compute_epoch_at_slot(slot), shard) + r = bytes_to_int(get_seed(beacon_state, get_current_epoch(beacon_state), DOMAIN_SHARD_COMMITTEE)[:8]) + return committee[r % len(committee)] ``` #### `get_indexed_attestation` From 247a6c8fca24e98489295d56db58c32180158c88 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 2 Apr 2020 11:22:21 +0800 Subject: [PATCH 29/44] Add `verify_fraud_proof` function --- specs/phase1/fraud-proofs.md | 60 ++++++++++++++++++++++++++++++------ 1 file changed, 50 insertions(+), 10 deletions(-) diff --git a/specs/phase1/fraud-proofs.md b/specs/phase1/fraud-proofs.md index 19077ba88c..e72a8bb2ab 100644 --- a/specs/phase1/fraud-proofs.md +++ b/specs/phase1/fraud-proofs.md @@ -6,7 +6,8 @@ - [Table of contents](#table-of-contents) - [Introduction](#introduction) - [Fraud proofs](#fraud-proofs) - - [Shard state transition function](#shard-state-transition-function) + - [Shard state transition function](#shard-state-transition-function) + - [Verifying the proof](#verifying-the-proof) - [Honest committee member behavior](#honest-committee-member-behavior) @@ -32,17 +33,12 @@ This document describes the shard transition function and fraud proofs as part o TODO. The intent is to have a single universal fraud proof type, which contains the following parts: 1. An on-time attestation on some `shard` signing a `ShardTransition` -2. An index `i` of a particular position to focus on +2. An index `index` of a particular position to focus on 3. The `ShardTransition` itself -4. The full body of the block -5. A Merkle proof to the `shard_states` in the parent block the attestation is referencing +4. The full body of the block `ShardBlock` +5. A Merkle proof to the `shard_states` in the parent block `parent_block` the attestation is referencing -The proof verifies that one of the two conditions is false: - -1. `custody_bits[i][j] != generate_custody_bit(subkey, block_contents)` for any `j` -2. `execute_state_transition(shard, slot, transition.shard_states[i-1].data, hash_tree_root(parent), get_shard_proposer_index(state, shard, slot), block_contents) != transition.shard_states[i].data` (if `i=0` then instead use `parent.shard_states[shard][-1].data`) - -## Shard state transition function +### Shard state transition function ```python def shard_state_transition(beacon_state: BeaconState, @@ -71,6 +67,50 @@ def verify_shard_block_signature(beacon_state: BeaconState, return bls.Verify(proposer.pubkey, signing_root, signed_block.signature) ``` +### Verifying the proof + +```python +def verify_fraud_proof(beacon_state: BeaconState, + subkey: BLSPubkey, + attestation: Attestation, + index: uint64, + transition: ShardTransition, + signed_block: SignedShardBlock, + parent_block: ShardBlock) -> bool: + # 1. Check if `custody_bits[index][j] != generate_custody_bit(subkey, block_contents)` for any `j` + shard = get_shard(beacon_state, attestation) + slot = attestation.data.slot + custody_bits = attestation.custody_bits_blocks + for j in range(custody_bits[index]): + if custody_bits[index][j] != generate_custody_bit(subkey, signed_block): + return True + + # 2. Verify the shard state transition + if index == 0: + parent_data = parent_block.shard_states[shard][-1].data + else: + parent_data = parent_block.shard_states[shard][index].data + + if shard_state_transition( + beacon_state, + shard, + slot, + transition.shard_states[index - 1].data, + hash_tree_root(parent_block), + get_shard_proposer_index(beacon_state, slot, shard), + signed_block, + ) != parent_data: + return True + + return False +``` + +```python +def generate_custody_bit(subkey: BLSPubkey, block: ShardBlock) -> bool: + # TODO + ... +``` + ## Honest committee member behavior Suppose you are a committee member on shard `shard` at slot `current_slot`. Let `state` be the head beacon state you are building on, and let `QUARTER_PERIOD = SECONDS_PER_SLOT // 4`. `2 * QUARTER_PERIOD` seconds into slot `slot`, run the following procedure: From 849d3f83bf4fd95ef47fd1fcee45494dc4a49e83 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Mon, 6 Apr 2020 17:48:24 +0800 Subject: [PATCH 30/44] Apply @terencechain 's review feedback Co-Authored-By: terence tsao --- specs/phase1/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/phase1/beacon-chain.md b/specs/phase1/beacon-chain.md index fbfdce70ef..7c3ae786ba 100644 --- a/specs/phase1/beacon-chain.md +++ b/specs/phase1/beacon-chain.md @@ -720,7 +720,7 @@ def apply_shard_transition(state: BeaconState, shard: Shard, transition: ShardTr if any(transition.shard_data_roots): headers.append(ShardBlockHeader( shard_parent_root=shard_parent_root, - parent_hash=get_block_root_at_slot(state, get_previous_slot(state.slot)), + beacon_parent_root=get_block_root_at_slot(state, get_previous_slot(state.slot)), slot=offset_slots[i], body_root=transition.shard_data_roots[i] )) From 4e8a7ff1156dc3a12ccb718476782bc685de3a4f Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Mon, 6 Apr 2020 19:30:32 +0800 Subject: [PATCH 31/44] [squashed] shard transition wip Fix the wrong `get_shard_proposer_index` parameters order Phase 1 WIP Add shard transition basic test Fix lint error Fix --- specs/phase1/beacon-chain.md | 63 +++-- specs/phase1/fraud-proofs.md | 231 ++++++++++++++---- .../eth2spec/test/helpers/attestations.py | 40 ++- .../eth2spec/test/helpers/crosslinks.py | 28 +++ .../eth2spec/test/helpers/phase1/__init__.py | 0 .../test/helpers/phase1/attestations.py | 63 ----- .../test/helpers/phase1/shard_block.py | 71 ------ .../test/helpers/phase1/shard_state.py | 18 -- .../eth2spec/test/helpers/shard_block.py | 47 ++++ .../test_process_crosslink.py | 52 ++++ 10 files changed, 375 insertions(+), 238 deletions(-) create mode 100644 tests/core/pyspec/eth2spec/test/helpers/crosslinks.py delete mode 100644 tests/core/pyspec/eth2spec/test/helpers/phase1/__init__.py delete mode 100644 tests/core/pyspec/eth2spec/test/helpers/phase1/attestations.py delete mode 100644 tests/core/pyspec/eth2spec/test/helpers/phase1/shard_block.py delete mode 100644 tests/core/pyspec/eth2spec/test/helpers/phase1/shard_state.py create mode 100644 tests/core/pyspec/eth2spec/test/helpers/shard_block.py create mode 100644 tests/core/pyspec/eth2spec/test/phase_1/block_processing/test_process_crosslink.py diff --git a/specs/phase1/beacon-chain.md b/specs/phase1/beacon-chain.md index 7c3ae786ba..4b22f86333 100644 --- a/specs/phase1/beacon-chain.md +++ b/specs/phase1/beacon-chain.md @@ -460,16 +460,17 @@ def get_online_validator_indices(state: BeaconState) -> Set[ValidatorIndex]: ```python def get_shard_committee(beacon_state: BeaconState, epoch: Epoch, shard: Shard) -> Sequence[ValidatorIndex]: - source_epoch = epoch - epoch % SHARD_COMMITTEE_PERIOD + source_epoch = epoch - epoch % SHARD_COMMITTEE_PERIOD if source_epoch > 0: source_epoch -= SHARD_COMMITTEE_PERIOD active_validator_indices = get_active_validator_indices(beacon_state, source_epoch) seed = get_seed(beacon_state, source_epoch, DOMAIN_SHARD_COMMITTEE) + active_shards_count = get_active_shard_count(beacon_state) return compute_committee( indices=active_validator_indices, seed=seed, index=shard, - count=get_active_shard_count(beacon_state) + count=active_shards_count, ) ``` @@ -712,29 +713,35 @@ def apply_shard_transition(state: BeaconState, shard: Shard, transition: ShardTr ) assert transition.start_slot == offset_slots[0] - # Reconstruct shard headers headers = [] + header = ShardBlockHeader() proposers = [] - shard_parent_root = state.shard_states[shard].latest_block_root - for i in range(len(offset_slots)): - if any(transition.shard_data_roots): - headers.append(ShardBlockHeader( - shard_parent_root=shard_parent_root, - beacon_parent_root=get_block_root_at_slot(state, get_previous_slot(state.slot)), - slot=offset_slots[i], - body_root=transition.shard_data_roots[i] - )) - proposers.append(get_shard_proposer_index(state, shard, offset_slots[i])) - shard_parent_root = hash_tree_root(headers[-1]) - - # Verify correct calculation of gas prices and slots prev_gasprice = state.shard_states[shard].gasprice + shard_parent_root = state.shard_states[shard].latest_block_root + beacon_parent_root = get_block_root_at_slot(state, get_previous_slot(state.slot)) for i in range(len(offset_slots)): + shard_block_length = transition.shard_block_lengths[i] + is_empty_proposal = (shard_block_length == 0) shard_state = transition.shard_states[i] - block_length = transition.shard_block_lengths[i] - assert shard_state.gasprice == get_updated_gasprice(prev_gasprice, block_length) - assert shard_state.slot == offset_slots[i] - prev_gasprice = shard_state.gasprice + proposal_index = get_shard_proposer_index(state, offset_slots[i], shard) + # Reconstruct shard headers + header = ShardBlockHeader( + shard_parent_root=shard_parent_root, + beacon_parent_root=beacon_parent_root, + proposer_index=proposal_index, + slot=offset_slots[i], + body_root=transition.shard_data_roots[i] + ) + shard_parent_root = hash_tree_root(header) + + if not is_empty_proposal: + # Only add non-empty signature + headers.append(header) + proposers.append(proposal_index) + # Verify correct calculation of gas prices and slots + assert shard_state.gasprice == get_updated_gasprice(prev_gasprice, shard_block_length) + assert shard_state.slot == offset_slots[i] + prev_gasprice = shard_state.gasprice pubkeys = [state.validators[proposer].pubkey for proposer in proposers] signing_roots = [ @@ -745,7 +752,8 @@ def apply_shard_transition(state: BeaconState, shard: Shard, transition: ShardTr assert bls.AggregateVerify(zip(pubkeys, signing_roots), signature=transition.proposer_signature_aggregate) # Save updated state - state.shard_states[shard] = transition.shard_states[-1] + state.shard_states[shard] = transition.shard_states[len(transition.shard_states) - 1] + assert state.slot > 0 state.shard_states[shard].slot = state.slot - 1 ``` @@ -779,7 +787,9 @@ def process_crosslink_for_shard(state: BeaconState, # Attestation <-> shard transition consistency assert shard_transition_root == hash_tree_root(shard_transition) - assert attestation.data.head_shard_root == shard_transition.shard_data_roots[-1] + assert attestation.data.head_shard_root == shard_transition.shard_data_roots[ + len(shard_transition.shard_data_roots) - 1 + ] # Apply transition apply_shard_transition(state, shard, shard_transition) @@ -790,11 +800,11 @@ def process_crosslink_for_shard(state: BeaconState, increase_balance(state, beacon_proposer_index, proposer_reward) states_slots_lengths = zip( shard_transition.shard_states, - get_offset_slots(state, get_latest_slot_for_shard(state, shard)), + get_offset_slots(state, shard), shard_transition.shard_block_lengths ) for shard_state, slot, length in states_slots_lengths: - proposer_index = get_shard_proposer_index(state, shard, slot) + proposer_index = get_shard_proposer_index(state, slot, shard) decrease_balance(state, proposer_index, shard_state.gasprice * length) # Return winning transition root @@ -913,14 +923,13 @@ def process_light_client_signatures(state: BeaconState, block_body: BeaconBlockB total_reward += get_base_reward(state, participant_index) increase_balance(state, get_beacon_proposer_index(state), Gwei(total_reward // PROPOSER_REWARD_QUOTIENT)) - + slot = get_previous_slot(state.slot) - signing_root = compute_signing_root(get_block_root_at_slot(state, slot), + signing_root = compute_signing_root(get_block_root_at_slot(state, slot), get_domain(state, DOMAIN_LIGHT_CLIENT, compute_epoch_at_slot(slot))) assert bls.FastAggregateVerify(signer_pubkeys, signing_root, signature=block_body.light_client_signature) ``` - ### Epoch transition This epoch transition overrides the phase0 epoch transition: diff --git a/specs/phase1/fraud-proofs.md b/specs/phase1/fraud-proofs.md index e72a8bb2ab..30b94d8106 100644 --- a/specs/phase1/fraud-proofs.md +++ b/specs/phase1/fraud-proofs.md @@ -9,6 +9,8 @@ - [Shard state transition function](#shard-state-transition-function) - [Verifying the proof](#verifying-the-proof) - [Honest committee member behavior](#honest-committee-member-behavior) + - [Helper functions](#helper-functions) + - [Make attestations](#make-attestations) @@ -30,76 +32,76 @@ This document describes the shard transition function and fraud proofs as part o ## Fraud proofs -TODO. The intent is to have a single universal fraud proof type, which contains the following parts: - -1. An on-time attestation on some `shard` signing a `ShardTransition` -2. An index `index` of a particular position to focus on -3. The `ShardTransition` itself -4. The full body of the block `ShardBlock` -5. A Merkle proof to the `shard_states` in the parent block `parent_block` the attestation is referencing - ### Shard state transition function ```python def shard_state_transition(beacon_state: BeaconState, shard: Shard, slot: Slot, - pre_state: Root, - previous_beacon_root: Root, - proposer_index: ValidatorIndex, - signed_block: SignedShardBlock, - validate_result: bool=True) -> Root: - # We will add something more substantive in phase 2 - - # Verify the proposer_index and signature - assert proposer_index == signed_block.message.proposer_index - if validate_result: - assert verify_shard_block_signature(beacon_state, signed_block) - - return hash(pre_state + hash_tree_root(previous_beacon_root) + hash_tree_root(signed_block.message.data)) + shard_state: ShardState, + beacon_parent_root: Root, + signed_block: SignedShardBlock) -> None: + # Update shard state + shard_state.data = hash( + hash_tree_root(shard_state) + hash_tree_root(beacon_parent_root) + hash_tree_root(signed_block.message.body) + ) + shard_state.slot = slot + shard_state.latest_block_root = hash_tree_root(signed_block.message) ``` ```python def verify_shard_block_signature(beacon_state: BeaconState, signed_block: SignedShardBlock) -> bool: proposer = beacon_state.validators[signed_block.message.proposer_index] - signing_root = compute_signing_root(signed_block.message, get_domain(beacon_state, DOMAIN_SHARD_PROPOSAL)) + domain = get_domain(beacon_state, DOMAIN_SHARD_PROPOSAL, compute_epoch_at_slot(signed_block.message.slot)) + signing_root = compute_signing_root(signed_block.message, domain) return bls.Verify(proposer.pubkey, signing_root, signed_block.signature) ``` ### Verifying the proof +TODO. The intent is to have a single universal fraud proof type, which contains the following parts: + +1. An on-time attestation `attestation` on some shard `shard` signing a `transition: ShardTransition` +2. An index `offset_index` of a particular position to focus on +3. The `transition: ShardTransition` itself +4. The full body of the shard block `shard_block` +5. A Merkle proof to the `shard_states` in the parent block the attestation is referencing + +Call the following function to verify the proof: + ```python def verify_fraud_proof(beacon_state: BeaconState, - subkey: BLSPubkey, attestation: Attestation, - index: uint64, + offset_index: uint64, transition: ShardTransition, signed_block: SignedShardBlock, - parent_block: ShardBlock) -> bool: - # 1. Check if `custody_bits[index][j] != generate_custody_bit(subkey, block_contents)` for any `j` + subkey: BLSPubkey, + beacon_parent_block: BeaconBlock) -> bool: + # 1. Check if `custody_bits[offset_index][j] != generate_custody_bit(subkey, block_contents)` for any `j`. shard = get_shard(beacon_state, attestation) slot = attestation.data.slot custody_bits = attestation.custody_bits_blocks - for j in range(custody_bits[index]): - if custody_bits[index][j] != generate_custody_bit(subkey, signed_block): + for j in range(custody_bits[offset_index]): + if custody_bits[offset_index][j] != generate_custody_bit(subkey, signed_block): return True - # 2. Verify the shard state transition - if index == 0: - parent_data = parent_block.shard_states[shard][-1].data + # 2. Check if the shard state transition result is wrong between + # `transition.shard_states[offset_index - 1]` to `transition.shard_states[offset_index]`. + if offset_index == 0: + shard_state = beacon_parent_block.shard_transitions[shard][-1] else: - parent_data = parent_block.shard_states[shard][index].data - - if shard_state_transition( - beacon_state, - shard, - slot, - transition.shard_states[index - 1].data, - hash_tree_root(parent_block), - get_shard_proposer_index(beacon_state, slot, shard), - signed_block, - ) != parent_data: + shard_state = transition.shard_states[offset_index - 1].copy() # Not doing the actual state updates here. + + shard_state_transition( + beacon_state=beacon_state, + shard=shard, + slot=slot, + shard_state=shard_state, + beacon_parent_root=hash_tree_root(beacon_parent_block), + signed_block=signed_block, + ) + if shard_state.latest_block_root != transition.shard_states[offset_index].data: return True return False @@ -113,14 +115,141 @@ def generate_custody_bit(subkey: BLSPubkey, block: ShardBlock) -> bool: ## Honest committee member behavior -Suppose you are a committee member on shard `shard` at slot `current_slot`. Let `state` be the head beacon state you are building on, and let `QUARTER_PERIOD = SECONDS_PER_SLOT // 4`. `2 * QUARTER_PERIOD` seconds into slot `slot`, run the following procedure: +### Helper functions + +```python +def get_winning_proposal(beacon_state: BeaconState, proposals: Sequence[SignedShardBlock]) -> SignedShardBlock: + # TODO: Let `winning_proposal` be the proposal with the largest number of total attestationsfrom slots in + # `state.shard_next_slots[shard]....slot-1` supporting it or any of its descendants, breaking ties by choosing + # the first proposal locally seen. Do `proposals.append(winning_proposal)`. + return proposals[-1] # stub +``` + +```python +def get_empty_body_block(shard_parent_root: Root, + beacon_parent_root: Root, + slot: Slot, + proposer_index: ValidatorIndex) -> ShardBlock: + return ShardBlock( + shard_parent_root=shard_parent_root, + beacon_parent_root=beacon_parent_root, + slot=slot, + proposer_index=proposer_index, + ) +``` + +```python +def is_empty_body(proposal: ShardBlock) -> bool: + # TODO + return len(proposal.body) == 0 +``` + +```python +def compute_shard_data_roots(proposals: Sequence[SignedShardBlock]) -> Sequence[Root]: + return [hash_tree_root(proposal.message.body) for proposal in proposals] +``` + +### Make attestations -* Initialize `proposals = []`, `shard_states = []`, `shard_state = state.shard_states[shard][-1]`, `start_slot = shard_state.slot`. -* For `slot in get_offset_slots(state, start_slot)`, do the following: - * Look for all valid proposals for `slot`; that is, a SignedShardBlock `proposal` where `shard_state_transition(shard, slot, shard_state, get_block_root_at_slot(state, state.slot - 1), get_shard_proposer_index(state, slot, shard), proposal)` returns a result and does not throw an exception. Let `choices` be the set of non-empty valid proposals you discover. - * If `len(choices) == 0`, do `proposals.append(make_empty_proposal(shard_state, slot))` - * If `len(choices) == 1`, do `proposals.append(choices[0])` - * If `len(choices) > 1`, let `winning_proposal` be the proposal with the largest number of total attestations from slots in `state.shard_next_slots[shard]....slot-1` supporting it or any of its descendants, breaking ties by choosing the first proposal locally seen. Do `proposals.append(winning_proposal)`. - * If `proposals.message.data[-1]` is NOT an empty proposal, set `shard_state = shard_state_transition(shard, slot, shard_state, get_block_root_at_slot(state, state.slot - 1), get_shard_proposer_index(state, slot, shard), proposals[-1])` and do `shard_states.append(shard_state)`. If it is an empty proposal, leave `shard_state` unchanged. +Suppose you are a committee member on shard `shard` at slot `current_slot` and you have received shard blocks `shard_blocks`. Let `state` be the head beacon state you are building on, and let `QUARTER_PERIOD = SECONDS_PER_SLOT // 4`. `2 * QUARTER_PERIOD` seconds into slot `current_slot`, run `get_shard_transition(beacon_state, shard, shard_blocks)` to get `shard_transition`. -Make an attestation using `shard_data_roots = [hash_tree_root(proposal.message.data) for proposal in proposals]` and `shard_state_roots = shard_states`. +```python +def get_shard_transition(beacon_state: BeaconState, + shard: Shard, + shard_blocks: Sequence[SignedShardBlock]) -> ShardTransition: + offset_slots = get_offset_slots(beacon_state, shard) + start_slot = offset_slots[0] + proposals, shard_states, shard_data_roots = get_shard_state_transition_result(beacon_state, shard, shard_blocks) + + assert len(proposals) > 0 + assert len(shard_data_roots) > 0 + + shard_block_lengths = [] + proposer_signatures = [] + for proposal in proposals: + shard_block_lengths.append(len(proposal.message.body)) + if proposal.signature != BLSSignature(): + proposer_signatures.append(proposal.signature) + + proposer_signature_aggregate = bls.Aggregate(proposer_signatures) + + return ShardTransition( + start_slot=start_slot, + shard_block_lengths=shard_block_lengths, + shard_data_roots=shard_data_roots, + shard_states=shard_states, + proposer_signature_aggregate=proposer_signature_aggregate, + ) +``` + +```python +def get_shard_state_transition_result( + beacon_state: BeaconState, + shard: Shard, + shard_blocks: Sequence[SignedShardBlock], + validate_result: bool=True, +) -> Tuple[Sequence[SignedShardBlock], Sequence[ShardState], Sequence[Root]]: + proposals = [] + shard_states = [] + shard_state = beacon_state.shard_states[shard].copy() + shard_parent_root = beacon_state.shard_states[shard].latest_block_root + for slot in get_offset_slots(beacon_state, shard): + choices = [] + beacon_parent_root = get_block_root_at_slot(beacon_state, get_previous_slot(beacon_state.slot)) + proposer_index = get_shard_proposer_index(beacon_state, slot, shard) + shard_blocks_at_slot = [block for block in shard_blocks if block.message.slot == slot] + for block in shard_blocks_at_slot: + temp_shard_state = shard_state.copy() # Not doing the actual state updates here. + # Try to apply state transition to temp_shard_state. + try: + # Verify the proposer_index and signature + assert block.message.proposer_index == proposer_index + if validate_result: + assert verify_shard_block_signature(beacon_state, block) + + shard_state_transition( + beacon_state=beacon_state, + shard=shard, + slot=slot, + shard_state=temp_shard_state, + beacon_parent_root=beacon_parent_root, + signed_block=block, + ) + except Exception: + pass # TODO: throw error in the test helper + else: + choices.append(block) + + if len(choices) == 0: + block_header = get_empty_body_block( + shard_parent_root=shard_parent_root, + beacon_parent_root=beacon_parent_root, + slot=slot, + proposer_index=proposer_index, + ) + block = SignedShardBlock(message=block_header) + proposals.append(block) + elif len(choices) == 1: + proposals.append(choices[0]) + else: + proposals.append(get_winning_proposal(beacon_state, choices)) + + shard_parent_root = hash_tree_root(proposals[-1].message) + + if not is_empty_body(proposals[-1].message): + # Apply state transition to shard_state. + shard_state_transition( + beacon_state=beacon_state, + shard=shard, + slot=slot, + shard_state=shard_state, + beacon_parent_root=beacon_parent_root, + signed_block=block, + ) + + shard_states.append(shard_state) + + shard_data_roots = compute_shard_data_roots(proposals) + + return proposals, shard_states, shard_data_roots +``` diff --git a/tests/core/pyspec/eth2spec/test/helpers/attestations.py b/tests/core/pyspec/eth2spec/test/helpers/attestations.py index 8215f5c5b0..c203b737a5 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/attestations.py +++ b/tests/core/pyspec/eth2spec/test/helpers/attestations.py @@ -1,6 +1,6 @@ from typing import List -from eth2spec.test.context import expect_assertion_error, PHASE0 +from eth2spec.test.context import expect_assertion_error, PHASE0, PHASE1 from eth2spec.test.helpers.state import state_transition_and_sign_block from eth2spec.test.helpers.block import build_empty_block_for_next_slot from eth2spec.test.helpers.keys import privkeys @@ -43,7 +43,7 @@ def run_attestation_processing(spec, state, attestation, valid=True): yield 'post', state -def build_attestation_data(spec, state, slot, index): +def build_attestation_data(spec, state, slot, index, shard_transition=None): assert state.slot >= slot if slot == state.slot: @@ -66,12 +66,21 @@ def build_attestation_data(spec, state, slot, index): source_epoch = state.current_justified_checkpoint.epoch source_root = state.current_justified_checkpoint.root + if spec.fork == PHASE1 and shard_transition is not None: + shard_transition_root = shard_transition.hash_tree_root() + head_shard_root = shard_transition.shard_data_roots[len(shard_transition.shard_data_roots) - 1] + else: + shard_transition_root = spec.Root() + head_shard_root = spec.Root() + return spec.AttestationData( slot=slot, index=index, beacon_block_root=block_root, source=spec.Checkpoint(epoch=source_epoch, root=source_root), target=spec.Checkpoint(epoch=spec.compute_epoch_at_slot(slot), root=epoch_boundary_root), + head_shard_root=head_shard_root, + shard_transition_root=shard_transition_root, ) @@ -89,7 +98,7 @@ def convert_to_valid_on_time_attestation(spec, state, attestation, signed=False) return attestation -def get_valid_on_time_attestation(spec, state, slot=None, index=None, signed=False): +def get_valid_on_time_attestation(spec, state, slot=None, index=None, shard_transition=None, signed=False): ''' Construct on-time attestation for next slot ''' @@ -98,7 +107,15 @@ def get_valid_on_time_attestation(spec, state, slot=None, index=None, signed=Fal if index is None: index = 0 - return get_valid_attestation(spec, state, slot=slot, index=index, signed=signed, on_time=True) + return get_valid_attestation( + spec, + state, + slot=slot, + index=index, + shard_transition=shard_transition, + signed=signed, + on_time=True, + ) def get_valid_late_attestation(spec, state, slot=None, index=None, signed=False): @@ -113,13 +130,20 @@ def get_valid_late_attestation(spec, state, slot=None, index=None, signed=False) return get_valid_attestation(spec, state, slot=slot, index=index, signed=signed, on_time=False) -def get_valid_attestation(spec, state, slot=None, index=None, empty=False, signed=False, on_time=True): +def get_valid_attestation(spec, + state, + slot=None, + index=None, + shard_transition=None, + empty=False, + signed=False, + on_time=True): if slot is None: slot = state.slot if index is None: index = 0 - attestation_data = build_attestation_data(spec, state, slot, index) + attestation_data = build_attestation_data(spec, state, slot=slot, index=index, shard_transition=shard_transition) beacon_committee = spec.get_beacon_committee( state, @@ -138,7 +162,7 @@ def get_valid_attestation(spec, state, slot=None, index=None, empty=False, signe if signed: sign_attestation(spec, state, attestation) - if spec.fork == 'phase1' and on_time: + if spec.fork == PHASE1 and on_time: attestation = convert_to_valid_on_time_attestation(spec, state, attestation, signed) return attestation @@ -210,7 +234,7 @@ def get_attestation_custody_signature(spec, state, attestation_data, block_index def sign_attestation(spec, state, attestation): - if spec.fork == 'phase1' and any(attestation.custody_bits_blocks): + if spec.fork == PHASE1 and any(attestation.custody_bits_blocks): sign_on_time_attestation(spec, state, attestation) return diff --git a/tests/core/pyspec/eth2spec/test/helpers/crosslinks.py b/tests/core/pyspec/eth2spec/test/helpers/crosslinks.py new file mode 100644 index 0000000000..ea5da89d96 --- /dev/null +++ b/tests/core/pyspec/eth2spec/test/helpers/crosslinks.py @@ -0,0 +1,28 @@ +from eth2spec.test.context import expect_assertion_error + + +def run_crosslinks_processing(spec, state, shard_transitions, attestations, valid=True): + """ + Run ``process_attestation``, yielding: + - pre-state ('pre') + - shard_transitions ('shard_transitions') + - attestations ('attestations') + - post-state ('post'). + If ``valid == False``, run expecting ``AssertionError`` + """ + # yield pre-state + yield 'pre', state + yield 'shard_transitions', shard_transitions + yield 'attestations', attestations + + # If the attestation is invalid, processing is aborted, and there is no post-state. + if not valid: + expect_assertion_error(lambda: spec.process_crosslinks(state, shard_transitions, attestations)) + yield 'post', None + return + + # process crosslinks + spec.process_crosslinks(state, shard_transitions, attestations) + + # yield post-state + yield 'post', state diff --git a/tests/core/pyspec/eth2spec/test/helpers/phase1/__init__.py b/tests/core/pyspec/eth2spec/test/helpers/phase1/__init__.py deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/tests/core/pyspec/eth2spec/test/helpers/phase1/attestations.py b/tests/core/pyspec/eth2spec/test/helpers/phase1/attestations.py deleted file mode 100644 index 0e16e1face..0000000000 --- a/tests/core/pyspec/eth2spec/test/helpers/phase1/attestations.py +++ /dev/null @@ -1,63 +0,0 @@ -from eth2spec.utils.ssz.ssz_typing import Bitlist -from eth2spec.utils import bls - -from eth2spec.test.helpers.keys import privkeys -import eth2spec.test.helpers.attestations as phase0_attestations - - -def get_valid_on_time_attestation(spec, state, index=None, signed=False): - ''' - Construct on-time attestation for next slot - ''' - if index is None: - index = 0 - - attestation = phase0_attestations.get_valid_attestation(spec, state, state.slot, index, False) - shard = spec.get_shard(state, attestation) - offset_slots = spec.compute_offset_slots(spec.get_latest_slot_for_shard(state, shard), state.slot + 1) - - for _ in offset_slots: - attestation.custody_bits_blocks.append( - Bitlist[spec.MAX_VALIDATORS_PER_COMMITTEE]([0 for _ in attestation.aggregation_bits]) - ) - - if signed: - sign_attestation(spec, state, attestation) - - return attestation - - -def sign_attestation(spec, state, attestation): - if not any(attestation.custody_bits_blocks): - phase0_attestations.sign_attestation(spec, state, attestation) - return - - committee = spec.get_beacon_committee(state, attestation.data.slot, attestation.data.index) - signatures = [] - for block_index, custody_bits in enumerate(attestation.custody_bits_blocks): - for participant, abit, cbit in zip(committee, attestation.aggregation_bits, custody_bits): - if not abit: - continue - signatures.append(get_attestation_custody_signature( - spec, - state, - attestation.data, - block_index, - cbit, - privkeys[participant] - )) - - attestation.signature = bls.Aggregate(signatures) - - -def get_attestation_custody_signature(spec, state, attestation_data, block_index, bit, privkey): - domain = spec.get_domain(state, spec.DOMAIN_BEACON_ATTESTER, attestation_data.target.epoch) - signing_root = spec.compute_signing_root( - spec.AttestationCustodyBitWrapper( - attestation_data.hash_tree_root(), - block_index, - bit, - ), - domain, - ) - return bls.Sign(privkey, signing_root) diff --git a/tests/core/pyspec/eth2spec/test/helpers/phase1/shard_block.py b/tests/core/pyspec/eth2spec/test/helpers/phase1/shard_block.py deleted file mode 100644 index 6ef0cf79bf..0000000000 --- a/tests/core/pyspec/eth2spec/test/helpers/phase1/shard_block.py +++ /dev/null @@ -1,71 +0,0 @@ -from eth2spec.test.helpers.keys import privkeys -from eth2spec.utils import bls -from eth2spec.utils.bls import only_with_bls -from eth2spec.utils.ssz.ssz_impl import ( - hash_tree_root, -) - -from .attestations import ( - sign_shard_attestation, -) - - -@only_with_bls() -def sign_shard_block(spec, beacon_state, shard_state, block, proposer_index=None): - if proposer_index is None: - proposer_index = spec.get_shard_proposer_index(beacon_state, shard_state.shard, block.slot) - - privkey = privkeys[proposer_index] - domain = spec.get_domain(beacon_state, spec.DOMAIN_SHARD_PROPOSER, spec.compute_epoch_of_shard_slot(block.slot)) - signing_root = spec.compute_signing_root(block, domain) - block.signature = bls.Sign(privkey, signing_root) - - -def build_empty_shard_block(spec, - beacon_state, - shard_state, - slot, - signed=False, - full_attestation=False): - if slot is None: - slot = shard_state.slot - - previous_beacon_header = beacon_state.latest_block_header.copy() - if previous_beacon_header.state_root == spec.Bytes32(): - previous_beacon_header.state_root = beacon_state.hash_tree_root() - beacon_block_root = hash_tree_root(previous_beacon_header) - - previous_block_header = shard_state.latest_block_header.copy() - if previous_block_header.state_root == spec.Bytes32(): - previous_block_header.state_root = shard_state.hash_tree_root() - parent_root = hash_tree_root(previous_block_header) - - block = spec.ShardBlock( - shard=shard_state.shard, - slot=slot, - beacon_block_root=beacon_block_root, - parent_root=parent_root, - block_size_sum=shard_state.block_size_sum + spec.SHARD_HEADER_SIZE, - ) - - if full_attestation: - shard_committee = spec.get_shard_committee(beacon_state, shard_state.shard, block.slot) - block.aggregation_bits = list( - (True,) * len(shard_committee) + - (False,) * (spec.MAX_PERIOD_COMMITTEE_SIZE * 2 - len(shard_committee)) - ) - else: - shard_committee = [] - - block.attestations = sign_shard_attestation( - spec, - beacon_state, - shard_state, - block, - participants=shard_committee, - ) - - if signed: - sign_shard_block(spec, beacon_state, shard_state, block) - - return block diff --git a/tests/core/pyspec/eth2spec/test/helpers/phase1/shard_state.py b/tests/core/pyspec/eth2spec/test/helpers/phase1/shard_state.py deleted file mode 100644 index 24240b5fa5..0000000000 --- a/tests/core/pyspec/eth2spec/test/helpers/phase1/shard_state.py +++ /dev/null @@ -1,18 +0,0 @@ -from eth2spec.test.helpers.phase1.shard_block import sign_shard_block - - -def configure_shard_state(spec, beacon_state, shard=0): - beacon_state.slot = spec.Slot(spec.SHARD_GENESIS_EPOCH * spec.SLOTS_PER_EPOCH) - shard_state = spec.get_genesis_shard_state(spec.Shard(shard)) - shard_state.slot = spec.ShardSlot(spec.SHARD_GENESIS_EPOCH * spec.SHARD_SLOTS_PER_EPOCH) - return beacon_state, shard_state - - -def shard_state_transition_and_sign_block(spec, beacon_state, shard_state, block): - """ - Shard state transition via the provided ``block`` - then package the block with the state root and signature. - """ - spec.shard_state_transition(beacon_state, shard_state, block) - block.state_root = shard_state.hash_tree_root() - sign_shard_block(spec, beacon_state, shard_state, block) diff --git a/tests/core/pyspec/eth2spec/test/helpers/shard_block.py b/tests/core/pyspec/eth2spec/test/helpers/shard_block.py new file mode 100644 index 0000000000..0cb8a3dfd4 --- /dev/null +++ b/tests/core/pyspec/eth2spec/test/helpers/shard_block.py @@ -0,0 +1,47 @@ +from eth2spec.test.helpers.keys import privkeys +from eth2spec.utils import bls +from eth2spec.utils.bls import only_with_bls + + +@only_with_bls() +def sign_shard_block(spec, beacon_state, shard, block, proposer_index=None): + slot = block.message.slot + if proposer_index is None: + proposer_index = spec.get_shard_proposer_index(beacon_state, slot, shard) + + privkey = privkeys[proposer_index] + domain = spec.get_domain(beacon_state, spec.DOMAIN_SHARD_PROPOSAL, spec.compute_epoch_at_slot(slot)) + signing_root = spec.compute_signing_root(block.message, domain) + block.signature = bls.Sign(privkey, signing_root) + + +def build_empty_shard_block(spec, + beacon_state, + shard, + slot, + body=None, + signed=False): + shard_state = beacon_state.shard_states[shard] + if slot is None: + slot = shard_state.slot + + if body is None: + body = [] + + proposer_index = spec.get_shard_proposer_index(beacon_state, slot, shard) + + block = spec.ShardBlock( + shard_parent_root=shard_state.latest_block_root, + beacon_parent_root=spec.get_block_root_at_slot(beacon_state, spec.get_previous_slot(beacon_state.slot)), + slot=slot, + proposer_index=proposer_index, + body=body, + ) + signed_block = spec.SignedShardBlock( + message=block, + ) + + if signed: + sign_shard_block(spec, beacon_state, shard, signed_block, proposer_index=proposer_index) + + return signed_block diff --git a/tests/core/pyspec/eth2spec/test/phase_1/block_processing/test_process_crosslink.py b/tests/core/pyspec/eth2spec/test/phase_1/block_processing/test_process_crosslink.py new file mode 100644 index 0000000000..1d5fe7c02f --- /dev/null +++ b/tests/core/pyspec/eth2spec/test/phase_1/block_processing/test_process_crosslink.py @@ -0,0 +1,52 @@ +from eth2spec.test.context import ( + with_all_phases_except, + spec_state_test, + always_bls, +) +from eth2spec.test.helpers.attestations import ( + get_valid_on_time_attestation, +) +from eth2spec.test.helpers.crosslinks import ( + run_crosslinks_processing, +) +from eth2spec.test.helpers.shard_block import build_empty_shard_block +from eth2spec.test.helpers.state import next_epoch, next_slot + + +@with_all_phases_except(['phase0']) +@spec_state_test +@always_bls +def test_basic_crosslinks(spec, state): + next_epoch(spec, state) + next_epoch(spec, state) + state = spec.upgrade_to_phase1(state) + next_slot(spec, state) + + committee_index = spec.CommitteeIndex(0) + shard = spec.compute_shard_from_committee_index(state, committee_index, state.slot) + shard_block = build_empty_shard_block(spec, state, shard, body=b'\x12' * 10, slot=state.slot, signed=True) + shard_blocks = [shard_block] + + next_slot(spec, state) + + shard_transition = spec.get_shard_transition(state, shard, shard_blocks) + shard_transitions = [spec.ShardTransition()] * len(state.shard_states) + shard_transitions[shard] = shard_transition + + attestation = get_valid_on_time_attestation( + spec, + state, + slot=state.slot, + index=committee_index, + shard_transition=shard_transition, + signed=True, + ) + attestations = [attestation] + + offset_slots = spec.get_offset_slots(state, shard) + + yield from run_crosslinks_processing(spec, state, shard_transitions, attestations) + + shard_state = state.shard_states[shard] + assert shard_state.slot == offset_slots[-1] + assert shard_state.latest_block_root == shard_block.message.hash_tree_root() From afa12caf1f3d2595c79e1a53e901c9639e98f500 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Sat, 11 Apr 2020 01:12:40 +0800 Subject: [PATCH 32/44] Refactor `get_shard_state_transition_result` --- specs/phase1/fraud-proofs.md | 184 +++++++++++++++++++++-------------- 1 file changed, 112 insertions(+), 72 deletions(-) diff --git a/specs/phase1/fraud-proofs.md b/specs/phase1/fraud-proofs.md index 30b94d8106..5af904ff55 100644 --- a/specs/phase1/fraud-proofs.md +++ b/specs/phase1/fraud-proofs.md @@ -149,6 +149,118 @@ def compute_shard_data_roots(proposals: Sequence[SignedShardBlock]) -> Sequence[ return [hash_tree_root(proposal.message.body) for proposal in proposals] ``` +```python +def get_proposal_choices_at_slot(beacon_state: BeaconState, + shard_state: ShardState, + slot: Slot, + shard: Shard, + shard_blocks: Sequence[SignedShardBlock], + validate_result: bool=True) -> Sequence[SignedShardBlock]: + choices = [] + beacon_parent_root = get_block_root_at_slot(beacon_state, get_previous_slot(beacon_state.slot)) + proposer_index = get_shard_proposer_index(beacon_state, slot, shard) + shard_blocks_at_slot = [block for block in shard_blocks if block.message.slot == slot] + for block in shard_blocks_at_slot: + temp_shard_state = shard_state.copy() # Not doing the actual state updates here. + # Try to apply state transition to temp_shard_state. + try: + # Verify the proposer_index and signature + assert block.message.proposer_index == proposer_index + if validate_result: + assert verify_shard_block_signature(beacon_state, block) + + shard_state_transition( + beacon_state=beacon_state, + shard=shard, + slot=slot, + shard_state=temp_shard_state, + beacon_parent_root=beacon_parent_root, + signed_block=block, + ) + except Exception: + pass # TODO: throw error in the test helper + else: + choices.append(block) + return choices +``` + +```python +def get_proposal_at_slot(beacon_state: BeaconState, + shard_state: ShardState, + slot: Shard, + shard: Shard, + shard_parent_root: Root, + shard_blocks: Sequence[SignedShardBlock], + validate_result: bool=True) -> Tuple[SignedShardBlock, ShardState, Root]: + beacon_parent_root = get_block_root_at_slot(beacon_state, get_previous_slot(beacon_state.slot)) + proposer_index = get_shard_proposer_index(beacon_state, slot, shard) + shard_state = shard_state.copy() # Don't update the given shard_state + choices = get_proposal_choices_at_slot( + beacon_state=beacon_state, + shard_state=shard_state, + slot=slot, + shard=shard, + shard_blocks=shard_blocks, + validate_result=validate_result, + ) + if len(choices) == 0: + block_header = get_empty_body_block( + shard_parent_root=shard_parent_root, + beacon_parent_root=beacon_parent_root, + slot=slot, + proposer_index=proposer_index, + ) + proposal = SignedShardBlock(message=block_header) + elif len(choices) == 1: + proposal = choices[0] + else: + proposal = get_winning_proposal(beacon_state, choices) + + shard_parent_root = hash_tree_root(proposal.message) + + if not is_empty_body(proposal.message): + # Apply state transition to shard_state. + shard_state_transition( + beacon_state=beacon_state, + shard=shard, + slot=slot, + shard_state=shard_state, + beacon_parent_root=beacon_parent_root, + signed_block=proposal, + ) + + return proposal, shard_state, shard_parent_root +``` + +```python +def get_shard_state_transition_result( + beacon_state: BeaconState, + shard: Shard, + shard_blocks: Sequence[SignedShardBlock], + validate_result: bool=True, +) -> Tuple[Sequence[SignedShardBlock], Sequence[ShardState], Sequence[Root]]: + proposals = [] + shard_states = [] + shard_state = beacon_state.shard_states[shard].copy() + shard_parent_root = beacon_state.shard_states[shard].latest_block_root + for slot in get_offset_slots(beacon_state, shard): + proposal, shard_state, shard_parent_root = get_proposal_at_slot( + beacon_state=beacon_state, + shard_state=shard_state, + slot=slot, + shard=shard, + shard_parent_root=shard_parent_root, + shard_blocks=shard_blocks, + validate_result=validate_result, + ) + shard_states.append(shard_state) + proposals.append(proposal) + + shard_data_roots = compute_shard_data_roots(proposals) + + return proposals, shard_states, shard_data_roots +``` + ### Make attestations Suppose you are a committee member on shard `shard` at slot `current_slot` and you have received shard blocks `shard_blocks`. Let `state` be the head beacon state you are building on, and let `QUARTER_PERIOD = SECONDS_PER_SLOT // 4`. `2 * QUARTER_PERIOD` seconds into slot `current_slot`, run `get_shard_transition(beacon_state, shard, shard_blocks)` to get `shard_transition`. @@ -181,75 +293,3 @@ def get_shard_transition(beacon_state: BeaconState, proposer_signature_aggregate=proposer_signature_aggregate, ) ``` - -```python -def get_shard_state_transition_result( - beacon_state: BeaconState, - shard: Shard, - shard_blocks: Sequence[SignedShardBlock], - validate_result: bool=True, -) -> Tuple[Sequence[SignedShardBlock], Sequence[ShardState], Sequence[Root]]: - proposals = [] - shard_states = [] - shard_state = beacon_state.shard_states[shard].copy() - shard_parent_root = beacon_state.shard_states[shard].latest_block_root - for slot in get_offset_slots(beacon_state, shard): - choices = [] - beacon_parent_root = get_block_root_at_slot(beacon_state, get_previous_slot(beacon_state.slot)) - proposer_index = get_shard_proposer_index(beacon_state, slot, shard) - shard_blocks_at_slot = [block for block in shard_blocks if block.message.slot == slot] - for block in shard_blocks_at_slot: - temp_shard_state = shard_state.copy() # Not doing the actual state updates here. - # Try to apply state transition to temp_shard_state. - try: - # Verify the proposer_index and signature - assert block.message.proposer_index == proposer_index - if validate_result: - assert verify_shard_block_signature(beacon_state, block) - - shard_state_transition( - beacon_state=beacon_state, - shard=shard, - slot=slot, - shard_state=temp_shard_state, - beacon_parent_root=beacon_parent_root, - signed_block=block, - ) - except Exception: - pass # TODO: throw error in the test helper - else: - choices.append(block) - - if len(choices) == 0: - block_header = get_empty_body_block( - shard_parent_root=shard_parent_root, - beacon_parent_root=beacon_parent_root, - slot=slot, - proposer_index=proposer_index, - ) - block = SignedShardBlock(message=block_header) - proposals.append(block) - elif len(choices) == 1: - proposals.append(choices[0]) - else: - proposals.append(get_winning_proposal(beacon_state, choices)) - - shard_parent_root = hash_tree_root(proposals[-1].message) - - if not is_empty_body(proposals[-1].message): - # Apply state transition to shard_state. - shard_state_transition( - beacon_state=beacon_state, - shard=shard, - slot=slot, - shard_state=shard_state, - beacon_parent_root=beacon_parent_root, - signed_block=block, - ) - - shard_states.append(shard_state) - - shard_data_roots = compute_shard_data_roots(proposals) - - return proposals, shard_states, shard_data_roots -``` From e645d6b5fa7490134c1b30d98ae29aaa4cb46f00 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Mon, 13 Apr 2020 18:23:01 +0800 Subject: [PATCH 33/44] Rename `build_empty_shard_block` to `build_shard_block` --- .../core/pyspec/eth2spec/test/helpers/shard_block.py | 12 ++++++------ .../block_processing/test_process_crosslink.py | 4 ++-- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/helpers/shard_block.py b/tests/core/pyspec/eth2spec/test/helpers/shard_block.py index 0cb8a3dfd4..8f95cf73a7 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/shard_block.py +++ b/tests/core/pyspec/eth2spec/test/helpers/shard_block.py @@ -15,12 +15,12 @@ def sign_shard_block(spec, beacon_state, shard, block, proposer_index=None): block.signature = bls.Sign(privkey, signing_root) -def build_empty_shard_block(spec, - beacon_state, - shard, - slot, - body=None, - signed=False): +def build_shard_block(spec, + beacon_state, + shard, + slot, + body=None, + signed=False): shard_state = beacon_state.shard_states[shard] if slot is None: slot = shard_state.slot diff --git a/tests/core/pyspec/eth2spec/test/phase_1/block_processing/test_process_crosslink.py b/tests/core/pyspec/eth2spec/test/phase_1/block_processing/test_process_crosslink.py index 1d5fe7c02f..1585a1a451 100644 --- a/tests/core/pyspec/eth2spec/test/phase_1/block_processing/test_process_crosslink.py +++ b/tests/core/pyspec/eth2spec/test/phase_1/block_processing/test_process_crosslink.py @@ -9,7 +9,7 @@ from eth2spec.test.helpers.crosslinks import ( run_crosslinks_processing, ) -from eth2spec.test.helpers.shard_block import build_empty_shard_block +from eth2spec.test.helpers.shard_block import build_shard_block from eth2spec.test.helpers.state import next_epoch, next_slot @@ -24,7 +24,7 @@ def test_basic_crosslinks(spec, state): committee_index = spec.CommitteeIndex(0) shard = spec.compute_shard_from_committee_index(state, committee_index, state.slot) - shard_block = build_empty_shard_block(spec, state, shard, body=b'\x12' * 10, slot=state.slot, signed=True) + shard_block = build_shard_block(spec, state, shard, body=b'\x12' * 10, slot=state.slot, signed=True) shard_blocks = [shard_block] next_slot(spec, state) From 9724cb832d9f909b21f88683721389ff90e70756 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 16 Apr 2020 17:18:11 +0800 Subject: [PATCH 34/44] Apply suggestions from code review from @djrtwo Co-Authored-By: Danny Ryan --- specs/phase1/beacon-chain.md | 8 ++++---- specs/phase1/fraud-proofs.md | 6 +++--- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/specs/phase1/beacon-chain.md b/specs/phase1/beacon-chain.md index 4b22f86333..2410c534d6 100644 --- a/specs/phase1/beacon-chain.md +++ b/specs/phase1/beacon-chain.md @@ -465,12 +465,12 @@ def get_shard_committee(beacon_state: BeaconState, epoch: Epoch, shard: Shard) - source_epoch -= SHARD_COMMITTEE_PERIOD active_validator_indices = get_active_validator_indices(beacon_state, source_epoch) seed = get_seed(beacon_state, source_epoch, DOMAIN_SHARD_COMMITTEE) - active_shards_count = get_active_shard_count(beacon_state) + active_shard_count = get_active_shard_count(beacon_state) return compute_committee( indices=active_validator_indices, seed=seed, index=shard, - count=active_shards_count, + count=active_shard_count, ) ``` @@ -483,12 +483,12 @@ def get_light_client_committee(beacon_state: BeaconState, epoch: Epoch) -> Seque source_epoch -= LIGHT_CLIENT_COMMITTEE_PERIOD active_validator_indices = get_active_validator_indices(beacon_state, source_epoch) seed = get_seed(beacon_state, source_epoch, DOMAIN_LIGHT_CLIENT) - active_shards_count = get_active_shard_count(beacon_state) + active_shard_count = get_active_shard_count(beacon_state) return compute_committee( indices=active_validator_indices, seed=seed, index=0, - count=active_shards_count, + count=active_shard_count, )[:TARGET_COMMITTEE_SIZE] ``` diff --git a/specs/phase1/fraud-proofs.md b/specs/phase1/fraud-proofs.md index 5af904ff55..b476508a4d 100644 --- a/specs/phase1/fraud-proofs.md +++ b/specs/phase1/fraud-proofs.md @@ -89,7 +89,7 @@ def verify_fraud_proof(beacon_state: BeaconState, # 2. Check if the shard state transition result is wrong between # `transition.shard_states[offset_index - 1]` to `transition.shard_states[offset_index]`. if offset_index == 0: - shard_state = beacon_parent_block.shard_transitions[shard][-1] + shard_state = beacon_parent_block.shard_transitions[shard].shard_states[-1] else: shard_state = transition.shard_states[offset_index - 1].copy() # Not doing the actual state updates here. @@ -119,7 +119,7 @@ def generate_custody_bit(subkey: BLSPubkey, block: ShardBlock) -> bool: ```python def get_winning_proposal(beacon_state: BeaconState, proposals: Sequence[SignedShardBlock]) -> SignedShardBlock: - # TODO: Let `winning_proposal` be the proposal with the largest number of total attestationsfrom slots in + # TODO: Let `winning_proposal` be the proposal with the largest number of total attestations from slots in # `state.shard_next_slots[shard]....slot-1` supporting it or any of its descendants, breaking ties by choosing # the first proposal locally seen. Do `proposals.append(winning_proposal)`. return proposals[-1] # stub @@ -263,7 +263,7 @@ def get_shard_state_transition_result( ### Make attestations -Suppose you are a committee member on shard `shard` at slot `current_slot` and you have received shard blocks `shard_blocks`. Let `state` be the head beacon state you are building on, and let `QUARTER_PERIOD = SECONDS_PER_SLOT // 4`. `2 * QUARTER_PERIOD` seconds into slot `current_slot`, run `get_shard_transition(beacon_state, shard, shard_blocks)` to get `shard_transition`. +Suppose you are a committee member on shard `shard` at slot `current_slot` and you have received shard blocks `shard_blocks` since the latest successful crosslink for `shard` into the beacon chain. Let `state` be the head beacon state you are building on, and let `QUARTER_PERIOD = SECONDS_PER_SLOT // 4`. `2 * QUARTER_PERIOD` seconds into slot `current_slot`, run `get_shard_transition(beacon_state, shard, shard_blocks)` to get `shard_transition`. ```python def get_shard_transition(beacon_state: BeaconState, From 85d5a9abaf0dfbf9d652da3be3f606880b54943d Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 16 Apr 2020 19:43:48 +0800 Subject: [PATCH 35/44] [squashed] shard chain updates wip PR feedback from Danny and some refactor 1. Add stub `PHASE_1_GENESIS_SLOT` 2. Rename `get_updated_gasprice` to `compute_updated_gasprice` 3. Rename `compute_shard_data_roots` to `compute_shard_body_roots` Apply shard transition for the skipped slots Refactor `shard_state_transition` Get `beacon_parent_root` from offset slot Add more test Add `verify_shard_block_message` Add `> 0` Keep `beacon_parent_block` unchanged in `is_valid_fraud_proof` Remove some lines Fix type Refactor + simplify skipped slot processing --- configs/mainnet.yaml | 2 + configs/minimal.yaml | 2 + specs/phase1/beacon-chain.md | 68 ++++---- specs/phase1/fraud-proofs.md | 164 ++++++++---------- specs/phase1/phase1-fork.md | 9 +- .../eth2spec/test/helpers/shard_block.py | 2 +- .../test_process_crosslink.py | 53 +++++- 7 files changed, 166 insertions(+), 134 deletions(-) diff --git a/configs/mainnet.yaml b/configs/mainnet.yaml index 6d71cfa47c..f5f38de5eb 100644 --- a/configs/mainnet.yaml +++ b/configs/mainnet.yaml @@ -161,6 +161,8 @@ DOMAIN_CUSTODY_BIT_SLASHING: 0x83000000 # Phase 1: Upgrade from Phase 0 # --------------------------------------------------------------- PHASE_1_FORK_VERSION: 0x01000000 +# [STUB] +PHASE_1_GENESIS_SLOT: 32 INITIAL_ACTIVE_SHARDS: 64 # Phase 1: General diff --git a/configs/minimal.yaml b/configs/minimal.yaml index 9daf428b4a..7fc255f59d 100644 --- a/configs/minimal.yaml +++ b/configs/minimal.yaml @@ -162,6 +162,8 @@ DOMAIN_CUSTODY_BIT_SLASHING: 0x83000000 # --------------------------------------------------------------- # [customized] for testnet distinction PHASE_1_FORK_VERSION: 0x01000001 +# [customized] for testing +PHASE_1_GENESIS_SLOT: 8 # [customized] reduced for testing INITIAL_ACTIVE_SHARDS: 4 diff --git a/specs/phase1/beacon-chain.md b/specs/phase1/beacon-chain.md index 2410c534d6..5c67fe4f2a 100644 --- a/specs/phase1/beacon-chain.md +++ b/specs/phase1/beacon-chain.md @@ -39,6 +39,7 @@ - [`committee_to_compact_committee`](#committee_to_compact_committee) - [`compute_shard_from_committee_index`](#compute_shard_from_committee_index) - [`compute_offset_slots`](#compute_offset_slots) + - [`compute_updated_gasprice`](#compute_updated_gasprice) - [Beacon state accessors](#beacon-state-accessors) - [`get_active_shard_count`](#get_active_shard_count) - [`get_online_validator_indices`](#get_online_validator_indices) @@ -46,7 +47,6 @@ - [`get_light_client_committee`](#get_light_client_committee) - [`get_shard_proposer_index`](#get_shard_proposer_index) - [`get_indexed_attestation`](#get_indexed_attestation) - - [`get_updated_gasprice`](#get_updated_gasprice) - [`get_start_shard`](#get_start_shard) - [`get_shard`](#get_shard) - [`get_latest_slot_for_shard`](#get_latest_slot_for_shard) @@ -439,6 +439,20 @@ def compute_offset_slots(start_slot: Slot, end_slot: Slot) -> Sequence[Slot]: return [Slot(start_slot + x) for x in SHARD_BLOCK_OFFSETS if start_slot + x < end_slot] ``` +#### `compute_updated_gasprice` + +```python +def compute_updated_gasprice(prev_gasprice: Gwei, length: uint8) -> Gwei: + if length > TARGET_SHARD_BLOCK_SIZE: + delta = (prev_gasprice * (length - TARGET_SHARD_BLOCK_SIZE) + // TARGET_SHARD_BLOCK_SIZE // GASPRICE_ADJUSTMENT_COEFFICIENT) + return min(prev_gasprice + delta, MAX_GASPRICE) + else: + delta = (prev_gasprice * (TARGET_SHARD_BLOCK_SIZE - length) + // TARGET_SHARD_BLOCK_SIZE // GASPRICE_ADJUSTMENT_COEFFICIENT) + return max(prev_gasprice, MIN_GASPRICE + delta) - delta +``` + ### Beacon state accessors #### `get_active_shard_count` @@ -512,20 +526,6 @@ def get_indexed_attestation(beacon_state: BeaconState, attestation: Attestation) ) ``` -#### `get_updated_gasprice` - -```python -def get_updated_gasprice(prev_gasprice: Gwei, length: uint8) -> Gwei: - if length > TARGET_SHARD_BLOCK_SIZE: - delta = (prev_gasprice * (length - TARGET_SHARD_BLOCK_SIZE) - // TARGET_SHARD_BLOCK_SIZE // GASPRICE_ADJUSTMENT_COEFFICIENT) - return min(prev_gasprice + delta, MAX_GASPRICE) - else: - delta = (prev_gasprice * (TARGET_SHARD_BLOCK_SIZE - length) - // TARGET_SHARD_BLOCK_SIZE // GASPRICE_ADJUSTMENT_COEFFICIENT) - return max(prev_gasprice, MIN_GASPRICE + delta) - delta -``` - #### `get_start_shard` ```python @@ -617,7 +617,6 @@ def is_valid_indexed_attestation(state: BeaconState, indexed_attestation: Indexe return bls.AggregateVerify(zip(all_pubkeys, all_signing_roots), signature=attestation.signature) ``` - ### Block processing ```python @@ -630,7 +629,6 @@ def process_block(state: BeaconState, block: BeaconBlock) -> None: process_operations(state, block.body) ``` - #### Operations ```python @@ -714,34 +712,31 @@ def apply_shard_transition(state: BeaconState, shard: Shard, transition: ShardTr assert transition.start_slot == offset_slots[0] headers = [] - header = ShardBlockHeader() proposers = [] prev_gasprice = state.shard_states[shard].gasprice shard_parent_root = state.shard_states[shard].latest_block_root - beacon_parent_root = get_block_root_at_slot(state, get_previous_slot(state.slot)) for i in range(len(offset_slots)): shard_block_length = transition.shard_block_lengths[i] - is_empty_proposal = (shard_block_length == 0) + is_empty_proposal = shard_block_length == 0 shard_state = transition.shard_states[i] - proposal_index = get_shard_proposer_index(state, offset_slots[i], shard) - # Reconstruct shard headers - header = ShardBlockHeader( - shard_parent_root=shard_parent_root, - beacon_parent_root=beacon_parent_root, - proposer_index=proposal_index, - slot=offset_slots[i], - body_root=transition.shard_data_roots[i] - ) - shard_parent_root = hash_tree_root(header) - if not is_empty_proposal: - # Only add non-empty signature + proposal_index = get_shard_proposer_index(state, offset_slots[i], shard) + # Reconstruct shard headers + header = ShardBlockHeader( + shard_parent_root=shard_parent_root, + beacon_parent_root=get_block_root_at_slot(state, offset_slots[i]), + proposer_index=proposal_index, + slot=offset_slots[i], + body_root=transition.shard_data_roots[i] + ) + shard_parent_root = hash_tree_root(header) headers.append(header) proposers.append(proposal_index) - # Verify correct calculation of gas prices and slots - assert shard_state.gasprice == get_updated_gasprice(prev_gasprice, shard_block_length) - assert shard_state.slot == offset_slots[i] - prev_gasprice = shard_state.gasprice + + # Verify correct calculation of gas prices and slots + assert shard_state.gasprice == compute_updated_gasprice(prev_gasprice, shard_block_length) + assert shard_state.slot == offset_slots[i] + prev_gasprice = shard_state.gasprice pubkeys = [state.validators[proposer].pubkey for proposer in proposers] signing_roots = [ @@ -753,7 +748,6 @@ def apply_shard_transition(state: BeaconState, shard: Shard, transition: ShardTr # Save updated state state.shard_states[shard] = transition.shard_states[len(transition.shard_states) - 1] - assert state.slot > 0 state.shard_states[shard].slot = state.slot - 1 ``` diff --git a/specs/phase1/fraud-proofs.md b/specs/phase1/fraud-proofs.md index b476508a4d..c2178c10ef 100644 --- a/specs/phase1/fraud-proofs.md +++ b/specs/phase1/fraud-proofs.md @@ -36,17 +36,48 @@ This document describes the shard transition function and fraud proofs as part o ```python def shard_state_transition(beacon_state: BeaconState, - shard: Shard, - slot: Slot, shard_state: ShardState, - beacon_parent_root: Root, signed_block: SignedShardBlock) -> None: # Update shard state - shard_state.data = hash( - hash_tree_root(shard_state) + hash_tree_root(beacon_parent_root) + hash_tree_root(signed_block.message.body) + prev_gasprice = shard_state.gasprice + if len(signed_block.message.body) == 0: + latest_block_root = shard_state.latest_block_root + else: + latest_block_root = hash_tree_root(signed_block.message) + + shard_state.data = compute_shard_transition_data( + beacon_state, + shard_state, + signed_block.message.beacon_parent_root, + signed_block.message.body, + ) + shard_state.gasprice = compute_updated_gasprice(prev_gasprice, len(signed_block.message.body)) + shard_state.slot = signed_block.message.slot + shard_state.latest_block_root = latest_block_root +``` + +```python +def compute_shard_transition_data(beacon_state: BeaconState, + shard_state: ShardState, + beacon_parent_root: Root, + shard_body_root: Root) -> Bytes32: + return hash( + hash_tree_root(shard_state) + beacon_parent_root + shard_body_root ) - shard_state.slot = slot - shard_state.latest_block_root = hash_tree_root(signed_block.message) +``` + +```python +def verify_shard_block_message(beacon_state: BeaconState, + shard_state: ShardState, + block: ShardBlock, + slot: Slot, + shard: Shard) -> bool: + assert block.shard_parent_root == shard_state.latest_block_root + assert block.beacon_parent_root == get_block_root_at_slot(beacon_state, slot) + assert block.slot == slot + assert block.proposer_index == get_shard_proposer_index(beacon_state, slot, shard) + assert 0 < len(block.body) <= MAX_SHARD_BLOCK_SIZE + return True ``` ```python @@ -67,20 +98,20 @@ TODO. The intent is to have a single universal fraud proof type, which contains 3. The `transition: ShardTransition` itself 4. The full body of the shard block `shard_block` 5. A Merkle proof to the `shard_states` in the parent block the attestation is referencing +6. The `subkey` to generate the custody bit Call the following function to verify the proof: ```python -def verify_fraud_proof(beacon_state: BeaconState, - attestation: Attestation, - offset_index: uint64, - transition: ShardTransition, - signed_block: SignedShardBlock, - subkey: BLSPubkey, - beacon_parent_block: BeaconBlock) -> bool: +def is_valid_fraud_proof(beacon_state: BeaconState, + attestation: Attestation, + offset_index: uint64, + transition: ShardTransition, + signed_block: SignedShardBlock, + subkey: BLSPubkey, + beacon_parent_block: BeaconBlock) -> bool: # 1. Check if `custody_bits[offset_index][j] != generate_custody_bit(subkey, block_contents)` for any `j`. shard = get_shard(beacon_state, attestation) - slot = attestation.data.slot custody_bits = attestation.custody_bits_blocks for j in range(custody_bits[offset_index]): if custody_bits[offset_index][j] != generate_custody_bit(subkey, signed_block): @@ -89,19 +120,12 @@ def verify_fraud_proof(beacon_state: BeaconState, # 2. Check if the shard state transition result is wrong between # `transition.shard_states[offset_index - 1]` to `transition.shard_states[offset_index]`. if offset_index == 0: - shard_state = beacon_parent_block.shard_transitions[shard].shard_states[-1] + shard_state = beacon_parent_block.shard_transitions[shard].shard_states[-1].copy() else: shard_state = transition.shard_states[offset_index - 1].copy() # Not doing the actual state updates here. - shard_state_transition( - beacon_state=beacon_state, - shard=shard, - slot=slot, - shard_state=shard_state, - beacon_parent_root=hash_tree_root(beacon_parent_block), - signed_block=signed_block, - ) - if shard_state.latest_block_root != transition.shard_states[offset_index].data: + shard_state_transition(beacon_state, shard_state, signed_block) + if shard_state.data != transition.shard_states[offset_index].data: return True return False @@ -126,26 +150,7 @@ def get_winning_proposal(beacon_state: BeaconState, proposals: Sequence[SignedSh ``` ```python -def get_empty_body_block(shard_parent_root: Root, - beacon_parent_root: Root, - slot: Slot, - proposer_index: ValidatorIndex) -> ShardBlock: - return ShardBlock( - shard_parent_root=shard_parent_root, - beacon_parent_root=beacon_parent_root, - slot=slot, - proposer_index=proposer_index, - ) -``` - -```python -def is_empty_body(proposal: ShardBlock) -> bool: - # TODO - return len(proposal.body) == 0 -``` - -```python -def compute_shard_data_roots(proposals: Sequence[SignedShardBlock]) -> Sequence[Root]: +def compute_shard_body_roots(proposals: Sequence[SignedShardBlock]) -> Sequence[Root]: return [hash_tree_root(proposal.message.body) for proposal in proposals] ``` @@ -155,28 +160,23 @@ def get_proposal_choices_at_slot(beacon_state: BeaconState, slot: Slot, shard: Shard, shard_blocks: Sequence[SignedShardBlock], - validate_result: bool=True) -> Sequence[SignedShardBlock]: + validate_signature: bool=True) -> Sequence[SignedShardBlock]: + """ + Return the valid shard blocks at the given ``slot``. + Note that this function doesn't change the state. + """ choices = [] - beacon_parent_root = get_block_root_at_slot(beacon_state, get_previous_slot(beacon_state.slot)) - proposer_index = get_shard_proposer_index(beacon_state, slot, shard) shard_blocks_at_slot = [block for block in shard_blocks if block.message.slot == slot] for block in shard_blocks_at_slot: temp_shard_state = shard_state.copy() # Not doing the actual state updates here. # Try to apply state transition to temp_shard_state. try: - # Verify the proposer_index and signature - assert block.message.proposer_index == proposer_index - if validate_result: + # Verify block message and signature + assert verify_shard_block_message(beacon_state, temp_shard_state, block.message, slot, shard) + if validate_signature: assert verify_shard_block_signature(beacon_state, block) - shard_state_transition( - beacon_state=beacon_state, - shard=shard, - slot=slot, - shard_state=temp_shard_state, - beacon_parent_root=beacon_parent_root, - signed_block=block, - ) + shard_state_transition(beacon_state, temp_shard_state, block) except Exception: pass # TODO: throw error in the test helper else: @@ -189,11 +189,12 @@ def get_proposal_at_slot(beacon_state: BeaconState, shard_state: ShardState, slot: Shard, shard: Shard, - shard_parent_root: Root, shard_blocks: Sequence[SignedShardBlock], - validate_result: bool=True) -> Tuple[SignedShardBlock, ShardState, Root]: - beacon_parent_root = get_block_root_at_slot(beacon_state, get_previous_slot(beacon_state.slot)) - proposer_index = get_shard_proposer_index(beacon_state, slot, shard) + validate_signature: bool=True) -> Tuple[SignedShardBlock, ShardState]: + """ + Return ``proposal``, ``shard_state`` of the given ``slot``. + Note that this function doesn't change the state. + """ shard_state = shard_state.copy() # Don't update the given shard_state choices = get_proposal_choices_at_slot( beacon_state=beacon_state, @@ -201,35 +202,20 @@ def get_proposal_at_slot(beacon_state: BeaconState, slot=slot, shard=shard, shard_blocks=shard_blocks, - validate_result=validate_result, + validate_signature=validate_signature, ) if len(choices) == 0: - block_header = get_empty_body_block( - shard_parent_root=shard_parent_root, - beacon_parent_root=beacon_parent_root, - slot=slot, - proposer_index=proposer_index, - ) - proposal = SignedShardBlock(message=block_header) + block = ShardBlock(slot=slot) + proposal = SignedShardBlock(message=block) elif len(choices) == 1: proposal = choices[0] else: proposal = get_winning_proposal(beacon_state, choices) - shard_parent_root = hash_tree_root(proposal.message) - - if not is_empty_body(proposal.message): - # Apply state transition to shard_state. - shard_state_transition( - beacon_state=beacon_state, - shard=shard, - slot=slot, - shard_state=shard_state, - beacon_parent_root=beacon_parent_root, - signed_block=proposal, - ) + # Apply state transition + shard_state_transition(beacon_state, shard_state, proposal) - return proposal, shard_state, shard_parent_root + return proposal, shard_state ``` ```python @@ -237,26 +223,24 @@ def get_shard_state_transition_result( beacon_state: BeaconState, shard: Shard, shard_blocks: Sequence[SignedShardBlock], - validate_result: bool=True, + validate_signature: bool=True, ) -> Tuple[Sequence[SignedShardBlock], Sequence[ShardState], Sequence[Root]]: proposals = [] shard_states = [] shard_state = beacon_state.shard_states[shard].copy() - shard_parent_root = beacon_state.shard_states[shard].latest_block_root for slot in get_offset_slots(beacon_state, shard): - proposal, shard_state, shard_parent_root = get_proposal_at_slot( + proposal, shard_state = get_proposal_at_slot( beacon_state=beacon_state, shard_state=shard_state, slot=slot, shard=shard, - shard_parent_root=shard_parent_root, shard_blocks=shard_blocks, - validate_result=validate_result, + validate_signature=validate_signature, ) shard_states.append(shard_state) proposals.append(proposal) - shard_data_roots = compute_shard_data_roots(proposals) + shard_data_roots = compute_shard_body_roots(proposals) return proposals, shard_states, shard_data_roots ``` diff --git a/specs/phase1/phase1-fork.md b/specs/phase1/phase1-fork.md index 173fceeb47..e95c12c72f 100644 --- a/specs/phase1/phase1-fork.md +++ b/specs/phase1/phase1-fork.md @@ -7,7 +7,7 @@ - [Introduction](#introduction) - [Configuration](#configuration) - [Fork to Phase 1](#fork-to-phase-1) - - [Fork trigger.](#fork-trigger) + - [Fork trigger](#fork-trigger) - [Upgrading the state](#upgrading-the-state) @@ -35,17 +35,18 @@ Warning: this configuration is not definitive. | Name | Value | | - | - | | `PHASE_1_FORK_VERSION` | `Version('0x01000000')` | +| `PHASE_1_GENESIS_SLOT` | `2**5` **TBD** | | `INITIAL_ACTIVE_SHARDS` | `2**6` (= 64) | ## Fork to Phase 1 -### Fork trigger. +### Fork trigger -TBD. Social consensus, along with state conditions such as epoch boundary, finality, deposits, active validator count, etc. may be part of the decision process to trigger the fork. +TBD. Social consensus, along with state conditions such as epoch boundary, finality, deposits, active validator count, etc. may be part of the decision process to trigger the fork. For now we assume the condition will be triggered at slot `PHASE_1_GENESIS_SLOT`, where `PHASE_1_GENESIS_SLOT % SLOTS_PER_EPOCH == 0`. ### Upgrading the state -After `process_slots` of Phase 0 finishes, but before the first Phase 1 block is processed, an irregular state change is made to upgrade to Phase 1. +After `process_slots` of Phase 0 finishes, if `state.slot == PHASE_1_GENESIS_SLOT`, an irregular state change is made to upgrade to Phase 1. ```python def upgrade_to_phase1(pre: phase0.BeaconState) -> BeaconState: diff --git a/tests/core/pyspec/eth2spec/test/helpers/shard_block.py b/tests/core/pyspec/eth2spec/test/helpers/shard_block.py index 8f95cf73a7..25d868b65e 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/shard_block.py +++ b/tests/core/pyspec/eth2spec/test/helpers/shard_block.py @@ -32,7 +32,7 @@ def build_shard_block(spec, block = spec.ShardBlock( shard_parent_root=shard_state.latest_block_root, - beacon_parent_root=spec.get_block_root_at_slot(beacon_state, spec.get_previous_slot(beacon_state.slot)), + beacon_parent_root=spec.get_block_root_at_slot(beacon_state, spec.get_previous_slot(slot)), slot=slot, proposer_index=proposer_index, body=body, diff --git a/tests/core/pyspec/eth2spec/test/phase_1/block_processing/test_process_crosslink.py b/tests/core/pyspec/eth2spec/test/phase_1/block_processing/test_process_crosslink.py index 1585a1a451..6d1e65678b 100644 --- a/tests/core/pyspec/eth2spec/test/phase_1/block_processing/test_process_crosslink.py +++ b/tests/core/pyspec/eth2spec/test/phase_1/block_processing/test_process_crosslink.py @@ -10,7 +10,7 @@ run_crosslinks_processing, ) from eth2spec.test.helpers.shard_block import build_shard_block -from eth2spec.test.helpers.state import next_epoch, next_slot +from eth2spec.test.helpers.state import next_epoch, next_slot, next_slots @with_all_phases_except(['phase0']) @@ -24,7 +24,8 @@ def test_basic_crosslinks(spec, state): committee_index = spec.CommitteeIndex(0) shard = spec.compute_shard_from_committee_index(state, committee_index, state.slot) - shard_block = build_shard_block(spec, state, shard, body=b'\x12' * 10, slot=state.slot, signed=True) + body = b'1' * spec.MAX_SHARD_BLOCK_SIZE + shard_block = build_shard_block(spec, state, shard, body=body, slot=state.slot, signed=True) shard_blocks = [shard_block] next_slot(spec, state) @@ -43,10 +44,58 @@ def test_basic_crosslinks(spec, state): ) attestations = [attestation] + pre_gasprice = state.shard_states[shard].gasprice offset_slots = spec.get_offset_slots(state, shard) + assert len(offset_slots) == 1 yield from run_crosslinks_processing(spec, state, shard_transitions, attestations) shard_state = state.shard_states[shard] assert shard_state.slot == offset_slots[-1] assert shard_state.latest_block_root == shard_block.message.hash_tree_root() + assert shard_state == shard_transition.shard_states[len(shard_transition.shard_states) - 1] + assert shard_state.gasprice > pre_gasprice + + +@with_all_phases_except(['phase0']) +@spec_state_test +@always_bls +def test_multiple_offset_slots(spec, state): + next_epoch(spec, state) + next_epoch(spec, state) + state = spec.upgrade_to_phase1(state) + next_slot(spec, state) + + committee_index = spec.CommitteeIndex(0) + shard = spec.compute_shard_from_committee_index(state, committee_index, state.slot) + body = b'1' * spec.MAX_SHARD_BLOCK_SIZE + shard_block = build_shard_block(spec, state, shard, body=body, slot=state.slot, signed=True) + shard_blocks = [shard_block] + + next_slots(spec, state, 3) + + shard_transition = spec.get_shard_transition(state, shard, shard_blocks) + shard_transitions = [spec.ShardTransition()] * len(state.shard_states) + shard_transitions[shard] = shard_transition + + attestation = get_valid_on_time_attestation( + spec, + state, + slot=state.slot, + index=committee_index, + shard_transition=shard_transition, + signed=True, + ) + attestations = [attestation] + + pre_gasprice = state.shard_states[shard].gasprice + offset_slots = spec.get_offset_slots(state, shard) + assert len(offset_slots) == 3 + + yield from run_crosslinks_processing(spec, state, shard_transitions, attestations) + + shard_state = state.shard_states[shard] + assert shard_state.slot == offset_slots[-1] + assert shard_state.latest_block_root == shard_block.message.hash_tree_root() + assert shard_state == shard_transition.shard_states[len(shard_transition.shard_states) - 1] + assert shard_state.gasprice > pre_gasprice From 40483b587b1945eace6a1aeb187608782179e3a6 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Mon, 20 Apr 2020 16:57:29 +0800 Subject: [PATCH 36/44] [squashed] shard chain updates wip Use `ShardBlock` in `shard_state_transition` PR feedback 1. Rename `ShardState.data` -> `ShardState.transition_digest` 2. Rename `compute_shard_transition_data` to `compute_shard_transition_digest` 3. Add `assert state.slot > PHASE_1_GENESIS_SLOT` just in case, may move it later Add `get_post_shard_state` as a pure function wrapper --- specs/phase1/beacon-chain.md | 5 ++- specs/phase1/fraud-proofs.md | 61 +++++++++++++++++++++--------------- 2 files changed, 40 insertions(+), 26 deletions(-) diff --git a/specs/phase1/beacon-chain.md b/specs/phase1/beacon-chain.md index 5c67fe4f2a..1fdf7ec4ce 100644 --- a/specs/phase1/beacon-chain.md +++ b/specs/phase1/beacon-chain.md @@ -328,7 +328,7 @@ class ShardBlockHeader(Container): class ShardState(Container): slot: Slot gasprice: Gwei - data: Bytes32 + transition_digest: Bytes32 latest_block_root: Root ``` @@ -701,6 +701,9 @@ def validate_attestation(state: BeaconState, attestation: Attestation) -> None: ```python def apply_shard_transition(state: BeaconState, shard: Shard, transition: ShardTransition) -> None: + # TODO: only need to check it once when phase 1 starts + assert state.slot > PHASE_1_GENESIS_SLOT + # Correct data root count offset_slots = get_offset_slots(state, shard) assert ( diff --git a/specs/phase1/fraud-proofs.md b/specs/phase1/fraud-proofs.md index c2178c10ef..a5068cd07a 100644 --- a/specs/phase1/fraud-proofs.md +++ b/specs/phase1/fraud-proofs.md @@ -37,30 +37,43 @@ This document describes the shard transition function and fraud proofs as part o ```python def shard_state_transition(beacon_state: BeaconState, shard_state: ShardState, - signed_block: SignedShardBlock) -> None: + block: ShardBlock) -> None: # Update shard state prev_gasprice = shard_state.gasprice - if len(signed_block.message.body) == 0: + if len(block.body) == 0: latest_block_root = shard_state.latest_block_root else: - latest_block_root = hash_tree_root(signed_block.message) + latest_block_root = hash_tree_root(block) - shard_state.data = compute_shard_transition_data( + shard_state.transition_digest = compute_shard_transition_digest( beacon_state, shard_state, - signed_block.message.beacon_parent_root, - signed_block.message.body, + block.beacon_parent_root, + block.body, ) - shard_state.gasprice = compute_updated_gasprice(prev_gasprice, len(signed_block.message.body)) - shard_state.slot = signed_block.message.slot + shard_state.gasprice = compute_updated_gasprice(prev_gasprice, len(block.body)) + shard_state.slot = block.slot shard_state.latest_block_root = latest_block_root ``` ```python -def compute_shard_transition_data(beacon_state: BeaconState, - shard_state: ShardState, - beacon_parent_root: Root, - shard_body_root: Root) -> Bytes32: +def get_post_shard_state(beacon_state: BeaconState, + shard_state: ShardState, + block: ShardBlock) -> ShardState: + """ + A pure function that returns a new post ShardState instead of modifying the given `shard_state`. + """ + post_state = shard_state.copy() + shard_state_transition(beacon_state, post_state, block) + return post_state +``` + +```python +def compute_shard_transition_digest(beacon_state: BeaconState, + shard_state: ShardState, + beacon_parent_root: Root, + shard_body_root: Root) -> Bytes32: + # TODO: use SSZ hash tree root return hash( hash_tree_root(shard_state) + beacon_parent_root + shard_body_root ) @@ -107,25 +120,25 @@ def is_valid_fraud_proof(beacon_state: BeaconState, attestation: Attestation, offset_index: uint64, transition: ShardTransition, - signed_block: SignedShardBlock, + block: ShardBlock, subkey: BLSPubkey, beacon_parent_block: BeaconBlock) -> bool: # 1. Check if `custody_bits[offset_index][j] != generate_custody_bit(subkey, block_contents)` for any `j`. shard = get_shard(beacon_state, attestation) custody_bits = attestation.custody_bits_blocks for j in range(custody_bits[offset_index]): - if custody_bits[offset_index][j] != generate_custody_bit(subkey, signed_block): + if custody_bits[offset_index][j] != generate_custody_bit(subkey, block): return True # 2. Check if the shard state transition result is wrong between # `transition.shard_states[offset_index - 1]` to `transition.shard_states[offset_index]`. if offset_index == 0: - shard_state = beacon_parent_block.shard_transitions[shard].shard_states[-1].copy() + shard_state = beacon_parent_block.shard_transitions[shard].shard_states[-1] else: - shard_state = transition.shard_states[offset_index - 1].copy() # Not doing the actual state updates here. + shard_state = transition.shard_states[offset_index - 1] # Not doing the actual state updates here. - shard_state_transition(beacon_state, shard_state, signed_block) - if shard_state.data != transition.shard_states[offset_index].data: + shard_state = get_post_shard_state(beacon_state, shard_state, block) + if shard_state.transition_digest != transition.shard_states[offset_index].transition_digest: return True return False @@ -168,15 +181,14 @@ def get_proposal_choices_at_slot(beacon_state: BeaconState, choices = [] shard_blocks_at_slot = [block for block in shard_blocks if block.message.slot == slot] for block in shard_blocks_at_slot: - temp_shard_state = shard_state.copy() # Not doing the actual state updates here. - # Try to apply state transition to temp_shard_state. try: # Verify block message and signature - assert verify_shard_block_message(beacon_state, temp_shard_state, block.message, slot, shard) + # TODO these validations should have been checked upon receiving shard blocks. + assert verify_shard_block_message(beacon_state, shard_state, block.message, slot, shard) if validate_signature: assert verify_shard_block_signature(beacon_state, block) - shard_state_transition(beacon_state, temp_shard_state, block) + shard_state = get_post_shard_state(beacon_state, shard_state, block.message) except Exception: pass # TODO: throw error in the test helper else: @@ -195,7 +207,6 @@ def get_proposal_at_slot(beacon_state: BeaconState, Return ``proposal``, ``shard_state`` of the given ``slot``. Note that this function doesn't change the state. """ - shard_state = shard_state.copy() # Don't update the given shard_state choices = get_proposal_choices_at_slot( beacon_state=beacon_state, shard_state=shard_state, @@ -213,7 +224,7 @@ def get_proposal_at_slot(beacon_state: BeaconState, proposal = get_winning_proposal(beacon_state, choices) # Apply state transition - shard_state_transition(beacon_state, shard_state, proposal) + shard_state = get_post_shard_state(beacon_state, shard_state, proposal.message) return proposal, shard_state ``` @@ -227,7 +238,7 @@ def get_shard_state_transition_result( ) -> Tuple[Sequence[SignedShardBlock], Sequence[ShardState], Sequence[Root]]: proposals = [] shard_states = [] - shard_state = beacon_state.shard_states[shard].copy() + shard_state = beacon_state.shard_states[shard] for slot in get_offset_slots(beacon_state, shard): proposal, shard_state = get_proposal_at_slot( beacon_state=beacon_state, From c8a473ba24e130bd44c2cd28b64ce58e23b1814a Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Tue, 28 Apr 2020 10:46:13 +0800 Subject: [PATCH 37/44] Apply suggestions from code review Co-Authored-By: Danny Ryan --- specs/phase1/beacon-chain.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/specs/phase1/beacon-chain.md b/specs/phase1/beacon-chain.md index 1fdf7ec4ce..fc9687545a 100644 --- a/specs/phase1/beacon-chain.md +++ b/specs/phase1/beacon-chain.md @@ -497,12 +497,11 @@ def get_light_client_committee(beacon_state: BeaconState, epoch: Epoch) -> Seque source_epoch -= LIGHT_CLIENT_COMMITTEE_PERIOD active_validator_indices = get_active_validator_indices(beacon_state, source_epoch) seed = get_seed(beacon_state, source_epoch, DOMAIN_LIGHT_CLIENT) - active_shard_count = get_active_shard_count(beacon_state) return compute_committee( indices=active_validator_indices, seed=seed, index=0, - count=active_shard_count, + count=get_active_shard_count(beacon_state), )[:TARGET_COMMITTEE_SIZE] ``` From 524ba166d13d954e5d512b44bb9a5855f6f77116 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Tue, 28 Apr 2020 11:28:21 +0800 Subject: [PATCH 38/44] [squashed] shard chain updates wip Fix wrong field names Fix `build_attestation_data` and other PR feedback from Danny and Terence 1. Rename `get_previous_slot` to `compute_previous_slot` 2. Break down `build_empty_block` into `get_state_and_beacon_parent_root_at_slot`, use it in `build_shard_block` 3. Set defult `slot` to `shard_state.slot + 1` in `build_shard_block` Update `verify_shard_block_message`: check beacon_parent_root at fork choice rule stage instead of state transition Fix `beacon-chain.md` 1. Fix typo `attestation.slot == state.slot` -> `attestation.data.slot == state.slot` in `is_winning_attestation` 2. Check `verify_shard_transition_false_positives` **after** `process_operations` 3. Fix `shard_attestations` filter in `process_crosslinks`: since attestations come from block, should use `attestation.data.slot + MIN_ATTESTATION_INCLUSION_DELAY == state.slot` 4. [TBD] Allow empty `light_client_signature` to make the tests pass 5. [TBD] Add `is_shard_attestation`, filter out empty `ShardTransition()` Rework `test_process_crosslink` Add basic phase 1 `test_blocks` Add more test cases Revert `is_shard_attestation` and fix test cases backward compatibility. Remove `test_process_beacon_block_no_shard_transition` and consider it as invalid case. --- specs/phase1/beacon-chain.md | 41 +++-- specs/phase1/fraud-proofs.md | 3 +- specs/phase1/phase1-fork.md | 4 +- .../eth2spec/test/helpers/attestations.py | 60 ++++++-- .../pyspec/eth2spec/test/helpers/block.py | 27 ++-- .../eth2spec/test/helpers/shard_block.py | 49 +++++- .../test/phase_0/sanity/test_blocks.py | 11 +- .../test_process_crosslink.py | 113 ++++++-------- .../eth2spec/test/phase_1/sanity/__init__.py | 0 .../test/phase_1/sanity/test_blocks.py | 141 ++++++++++++++++++ 10 files changed, 335 insertions(+), 114 deletions(-) create mode 100644 tests/core/pyspec/eth2spec/test/phase_1/sanity/__init__.py create mode 100644 tests/core/pyspec/eth2spec/test/phase_1/sanity/test_blocks.py diff --git a/specs/phase1/beacon-chain.md b/specs/phase1/beacon-chain.md index fc9687545a..611cc19552 100644 --- a/specs/phase1/beacon-chain.md +++ b/specs/phase1/beacon-chain.md @@ -33,7 +33,7 @@ - [`AttestationCustodyBitWrapper`](#attestationcustodybitwrapper) - [Helper functions](#helper-functions) - [Misc](#misc-1) - - [`get_previous_slot`](#get_previous_slot) + - [`compute_previous_slot`](#compute_previous_slot) - [`pack_compact_validator`](#pack_compact_validator) - [`unpack_compact_validator`](#unpack_compact_validator) - [`committee_to_compact_committee`](#committee_to_compact_committee) @@ -52,6 +52,7 @@ - [`get_latest_slot_for_shard`](#get_latest_slot_for_shard) - [`get_offset_slots`](#get_offset_slots) - [Predicates](#predicates) + - [`is_shard_attestation`](#is_shard_attestation) - [`is_winning_attestation`](#is_winning_attestation) - [Updated `is_valid_indexed_attestation`](#updated-is_valid_indexed_attestation) - [Block processing](#block-processing) @@ -369,10 +370,10 @@ class AttestationCustodyBitWrapper(Container): ### Misc -#### `get_previous_slot` +#### `compute_previous_slot` ```python -def get_previous_slot(slot: Slot) -> Slot: +def compute_previous_slot(slot: Slot) -> Slot: if slot > 0: return Slot(slot - 1) else: @@ -556,6 +557,21 @@ def get_offset_slots(state: BeaconState, shard: Shard) -> Sequence[Slot]: ### Predicates +#### `is_shard_attestation` + +```python +def is_shard_attestation(state: BeaconState, + attestation: Attestation, + committee_index: CommitteeIndex) -> bool: + if not ( + attestation.data.index == committee_index + and attestation.data.slot + MIN_ATTESTATION_INCLUSION_DELAY == state.slot + ): + return False + + return True +``` + #### `is_winning_attestation` ```python @@ -568,7 +584,7 @@ def is_winning_attestation(state: BeaconState, ``winning_root`` formed by ``committee_index`` committee at the current slot. """ return ( - attestation.slot == state.slot + attestation.data.slot == state.slot and attestation.data.index == committee_index and attestation.data.shard_transition_root == winning_root ) @@ -623,9 +639,9 @@ def process_block(state: BeaconState, block: BeaconBlock) -> None: process_block_header(state, block) process_randao(state, block.body) process_eth1_data(state, block.body) - verify_shard_transition_false_positives(state, block.body) process_light_client_signatures(state, block.body) process_operations(state, block.body) + verify_shard_transition_false_positives(state, block.body) ``` #### Operations @@ -684,7 +700,7 @@ def validate_attestation(state: BeaconState, attestation: Attestation) -> None: # Correct data root count assert len(attestation.custody_bits_blocks) == len(get_offset_slots(state, shard)) # Correct parent block root - assert data.beacon_block_root == get_block_root_at_slot(state, get_previous_slot(state.slot)) + assert data.beacon_block_root == get_block_root_at_slot(state, compute_previous_slot(state.slot)) # Type 2: no shard transition, no custody bits else: # Ensure delayed attestation @@ -821,11 +837,12 @@ def process_crosslinks(state: BeaconState, for committee_index in map(CommitteeIndex, range(committee_count)): shard = compute_shard_from_committee_index(state, committee_index, state.slot) # All attestations in the block for this committee/shard and current slot + shard_transition = shard_transitions[shard] shard_attestations = [ attestation for attestation in attestations - if attestation.data.index == committee_index and attestation.data.slot == state.slot + if is_shard_attestation(state, attestation, committee_index) ] - shard_transition = shard_transitions[shard] + winning_root = process_crosslink_for_shard(state, committee_index, shard_transition, shard_attestations) if winning_root != Root(): # Mark relevant pending attestations as creating a successful crosslink @@ -920,10 +937,14 @@ def process_light_client_signatures(state: BeaconState, block_body: BeaconBlockB increase_balance(state, get_beacon_proposer_index(state), Gwei(total_reward // PROPOSER_REWARD_QUOTIENT)) - slot = get_previous_slot(state.slot) + slot = compute_previous_slot(state.slot) signing_root = compute_signing_root(get_block_root_at_slot(state, slot), get_domain(state, DOMAIN_LIGHT_CLIENT, compute_epoch_at_slot(slot))) - assert bls.FastAggregateVerify(signer_pubkeys, signing_root, signature=block_body.light_client_signature) + if len(signer_pubkeys) == 0: + # TODO: Or disallow empty light_client_signature? + return + else: + assert bls.FastAggregateVerify(signer_pubkeys, signing_root, signature=block_body.light_client_signature) ``` ### Epoch transition diff --git a/specs/phase1/fraud-proofs.md b/specs/phase1/fraud-proofs.md index a5068cd07a..c676e142a4 100644 --- a/specs/phase1/fraud-proofs.md +++ b/specs/phase1/fraud-proofs.md @@ -86,7 +86,6 @@ def verify_shard_block_message(beacon_state: BeaconState, slot: Slot, shard: Shard) -> bool: assert block.shard_parent_root == shard_state.latest_block_root - assert block.beacon_parent_root == get_block_root_at_slot(beacon_state, slot) assert block.slot == slot assert block.proposer_index == get_shard_proposer_index(beacon_state, slot, shard) assert 0 < len(block.body) <= MAX_SHARD_BLOCK_SIZE @@ -124,7 +123,6 @@ def is_valid_fraud_proof(beacon_state: BeaconState, subkey: BLSPubkey, beacon_parent_block: BeaconBlock) -> bool: # 1. Check if `custody_bits[offset_index][j] != generate_custody_bit(subkey, block_contents)` for any `j`. - shard = get_shard(beacon_state, attestation) custody_bits = attestation.custody_bits_blocks for j in range(custody_bits[offset_index]): if custody_bits[offset_index][j] != generate_custody_bit(subkey, block): @@ -133,6 +131,7 @@ def is_valid_fraud_proof(beacon_state: BeaconState, # 2. Check if the shard state transition result is wrong between # `transition.shard_states[offset_index - 1]` to `transition.shard_states[offset_index]`. if offset_index == 0: + shard = get_shard(beacon_state, attestation) shard_state = beacon_parent_block.shard_transitions[shard].shard_states[-1] else: shard_state = transition.shard_states[offset_index - 1] # Not doing the actual state updates here. diff --git a/specs/phase1/phase1-fork.md b/specs/phase1/phase1-fork.md index e95c12c72f..cc7d8f33ee 100644 --- a/specs/phase1/phase1-fork.md +++ b/specs/phase1/phase1-fork.md @@ -103,7 +103,7 @@ def upgrade_to_phase1(pre: phase0.BeaconState) -> BeaconState: ShardState( slot=pre.slot, gasprice=MIN_GASPRICE, - data=Root(), + transition_digest=Root(), latest_block_root=Root(), ) for i in range(INITIAL_ACTIVE_SHARDS) ), @@ -111,7 +111,7 @@ def upgrade_to_phase1(pre: phase0.BeaconState) -> BeaconState: current_light_committee=CompactCommittee(), # computed after state creation next_light_committee=CompactCommittee(), # Custody game - custody_challenge_index=0, + exposed_derived_secrets=[] * EARLY_DERIVED_SECRET_PENALTY_MAX_FUTURE_EPOCHS, # exposed_derived_secrets will fully default to zeroes ) next_epoch = Epoch(epoch + 1) diff --git a/tests/core/pyspec/eth2spec/test/helpers/attestations.py b/tests/core/pyspec/eth2spec/test/helpers/attestations.py index c203b737a5..9e98e83ea5 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/attestations.py +++ b/tests/core/pyspec/eth2spec/test/helpers/attestations.py @@ -1,7 +1,7 @@ from typing import List from eth2spec.test.context import expect_assertion_error, PHASE0, PHASE1 -from eth2spec.test.helpers.state import state_transition_and_sign_block +from eth2spec.test.helpers.state import state_transition_and_sign_block, next_slot, transition_to from eth2spec.test.helpers.block import build_empty_block_for_next_slot from eth2spec.test.helpers.keys import privkeys from eth2spec.utils import bls @@ -43,7 +43,7 @@ def run_attestation_processing(spec, state, attestation, valid=True): yield 'post', state -def build_attestation_data(spec, state, slot, index, shard_transition=None): +def build_attestation_data(spec, state, slot, index, shard_transition=None, on_time=True): assert state.slot >= slot if slot == state.slot: @@ -66,28 +66,42 @@ def build_attestation_data(spec, state, slot, index, shard_transition=None): source_epoch = state.current_justified_checkpoint.epoch source_root = state.current_justified_checkpoint.root - if spec.fork == PHASE1 and shard_transition is not None: - shard_transition_root = shard_transition.hash_tree_root() - head_shard_root = shard_transition.shard_data_roots[len(shard_transition.shard_data_roots) - 1] - else: - shard_transition_root = spec.Root() - head_shard_root = spec.Root() - - return spec.AttestationData( + attestation_data = spec.AttestationData( slot=slot, index=index, beacon_block_root=block_root, source=spec.Checkpoint(epoch=source_epoch, root=source_root), target=spec.Checkpoint(epoch=spec.compute_epoch_at_slot(slot), root=epoch_boundary_root), - head_shard_root=head_shard_root, - shard_transition_root=shard_transition_root, ) + if spec.fork == PHASE1: + if shard_transition is not None: + lastest_shard_data_root_index = len(shard_transition.shard_data_roots) - 1 + attestation_data.head_shard_root = shard_transition.shard_data_roots[lastest_shard_data_root_index] + attestation_data.shard_transition_root = shard_transition.hash_tree_root() + else: + # No shard transition + shard = spec.get_shard(state, spec.Attestation(data=attestation_data)) + if on_time: + temp_state = state.copy() + next_slot(spec, temp_state) + shard_transition = spec.get_shard_transition(temp_state, shard, []) + lastest_shard_data_root_index = len(shard_transition.shard_data_roots) - 1 + attestation_data.head_shard_root = shard_transition.shard_data_roots[lastest_shard_data_root_index] + attestation_data.shard_transition_root = shard_transition.hash_tree_root() + else: + attestation_data.head_shard_root = state.shard_states[shard].transition_digest + attestation_data.shard_transition_root = spec.Root() + return attestation_data + def convert_to_valid_on_time_attestation(spec, state, attestation, signed=False): shard = spec.get_shard(state, attestation) - offset_slots = spec.compute_offset_slots(spec.get_latest_slot_for_shard(state, shard), state.slot + 1) - for offset_slot in offset_slots: + offset_slots = spec.compute_offset_slots( + spec.get_latest_slot_for_shard(state, shard), + attestation.data.slot + spec.MIN_ATTESTATION_INCLUSION_DELAY, + ) + for _ in offset_slots: attestation.custody_bits_blocks.append( Bitlist[spec.MAX_VALIDATORS_PER_COMMITTEE]([0 for _ in attestation.aggregation_bits]) ) @@ -143,7 +157,9 @@ def get_valid_attestation(spec, if index is None: index = 0 - attestation_data = build_attestation_data(spec, state, slot=slot, index=index, shard_transition=shard_transition) + attestation_data = build_attestation_data( + spec, state, slot=slot, index=index, shard_transition=shard_transition, on_time=on_time + ) beacon_committee = spec.get_beacon_committee( state, @@ -298,7 +314,21 @@ def next_epoch_with_attestations(spec, spec, post_state, slot_to_attest, index=index, signed=True, on_time=False) block.body.attestations.append(prev_attestation) + if spec.fork == PHASE1: + fill_block_shard_transitions_by_attestations(spec, post_state, block) + signed_block = state_transition_and_sign_block(spec, post_state, block) signed_blocks.append(signed_block) return state, signed_blocks, post_state + + +def fill_block_shard_transitions_by_attestations(spec, state, block): + block.body.shard_transitions = [spec.ShardTransition()] * spec.MAX_SHARDS + for attestation in block.body.attestations: + shard = spec.get_shard(state, attestation) + if attestation.data.slot == state.slot: + temp_state = state.copy() + transition_to(spec, temp_state, slot=block.slot) + shard_transition = spec.get_shard_transition(temp_state, shard, []) + block.body.shard_transitions[shard] = shard_transition diff --git a/tests/core/pyspec/eth2spec/test/helpers/block.py b/tests/core/pyspec/eth2spec/test/helpers/block.py index 96cc30e35d..e40d5e7d8d 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/block.py +++ b/tests/core/pyspec/eth2spec/test/helpers/block.py @@ -71,24 +71,31 @@ def build_empty_block(spec, state, slot=None): """ if slot is None: slot = state.slot - if slot < state.slot: - raise Exception("build_empty_block cannot build blocks for past slots") - if slot > state.slot: - # transition forward in copied state to grab relevant data from state - state = state.copy() - spec.process_slots(state, slot) + state, parent_block_root = get_state_and_beacon_parent_root_at_slot(spec, state, slot) empty_block = spec.BeaconBlock() empty_block.slot = slot empty_block.proposer_index = spec.get_beacon_proposer_index(state) empty_block.body.eth1_data.deposit_count = state.eth1_deposit_index - previous_block_header = state.latest_block_header.copy() - if previous_block_header.state_root == spec.Root(): - previous_block_header.state_root = hash_tree_root(state) - empty_block.parent_root = hash_tree_root(previous_block_header) + empty_block.parent_root = parent_block_root apply_randao_reveal(spec, state, empty_block) return empty_block def build_empty_block_for_next_slot(spec, state): return build_empty_block(spec, state, state.slot + 1) + + +def get_state_and_beacon_parent_root_at_slot(spec, state, slot): + if slot < state.slot: + raise Exception("Cannot build blocks for past slots") + if slot > state.slot: + # transition forward in copied state to grab relevant data from state + state = state.copy() + spec.process_slots(state, slot) + + previous_block_header = state.latest_block_header.copy() + if previous_block_header.state_root == spec.Root(): + previous_block_header.state_root = hash_tree_root(state) + beacon_parent_root = hash_tree_root(previous_block_header) + return state, beacon_parent_root diff --git a/tests/core/pyspec/eth2spec/test/helpers/shard_block.py b/tests/core/pyspec/eth2spec/test/helpers/shard_block.py index 25d868b65e..922f4d748c 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/shard_block.py +++ b/tests/core/pyspec/eth2spec/test/helpers/shard_block.py @@ -1,3 +1,6 @@ +from eth2spec.test.helpers.attestations import get_valid_on_time_attestation +from eth2spec.test.helpers.block import get_state_and_beacon_parent_root_at_slot +from eth2spec.test.helpers.state import next_slots from eth2spec.test.helpers.keys import privkeys from eth2spec.utils import bls from eth2spec.utils.bls import only_with_bls @@ -18,21 +21,22 @@ def sign_shard_block(spec, beacon_state, shard, block, proposer_index=None): def build_shard_block(spec, beacon_state, shard, - slot, + slot=None, body=None, signed=False): shard_state = beacon_state.shard_states[shard] if slot is None: - slot = shard_state.slot + slot = shard_state.slot + 1 if body is None: - body = [] + body = b'\x56' * 128 proposer_index = spec.get_shard_proposer_index(beacon_state, slot, shard) + beacon_state, beacon_parent_root = get_state_and_beacon_parent_root_at_slot(spec, beacon_state, slot) block = spec.ShardBlock( shard_parent_root=shard_state.latest_block_root, - beacon_parent_root=spec.get_block_root_at_slot(beacon_state, spec.get_previous_slot(slot)), + beacon_parent_root=beacon_parent_root, slot=slot, proposer_index=proposer_index, body=body, @@ -45,3 +49,40 @@ def build_shard_block(spec, sign_shard_block(spec, beacon_state, shard, signed_block, proposer_index=proposer_index) return signed_block + + +def build_shard_transitions_till_slot(spec, state, shards, shard_blocks, target_len_offset_slot): + state = state.copy() + next_slots(spec, state, target_len_offset_slot) + shard_transitions = [spec.ShardTransition()] * spec.MAX_SHARDS + for shard in shards: + offset_slots = spec.get_offset_slots(state, shard) + len_offset_slots = len(offset_slots) + assert len_offset_slots == target_len_offset_slot + shard_blocks_of_shard = shard_blocks[shard] + shard_transition = spec.get_shard_transition(state, shard, shard_blocks_of_shard) + if len(shard_blocks_of_shard) > 0: + shard_block_root = shard_blocks_of_shard[-1].message.hash_tree_root() + assert shard_transition.shard_states[len_offset_slots - 1].latest_block_root == shard_block_root + assert shard_transition.shard_states[len_offset_slots - 1].slot == offset_slots[-1] + shard_transitions[shard] = shard_transition + + return shard_transitions + + +def build_attestation_with_shard_transition(spec, state, slot, index, target_len_offset_slot, shard_transition=None): + state = state.copy() + next_slots(spec, state, target_len_offset_slot) + attestation = get_valid_on_time_attestation( + spec, + state, + slot=slot, + index=index, + shard_transition=shard_transition, + signed=True, + ) + assert attestation.data.slot == slot + if shard_transition is not None: + assert target_len_offset_slot == len(shard_transition.shard_states) + assert attestation.data.shard_transition_root == shard_transition.hash_tree_root() + return attestation diff --git a/tests/core/pyspec/eth2spec/test/phase_0/sanity/test_blocks.py b/tests/core/pyspec/eth2spec/test/phase_0/sanity/test_blocks.py index b6b6718724..29a9dcca28 100644 --- a/tests/core/pyspec/eth2spec/test/phase_0/sanity/test_blocks.py +++ b/tests/core/pyspec/eth2spec/test/phase_0/sanity/test_blocks.py @@ -8,10 +8,13 @@ from eth2spec.test.helpers.keys import privkeys, pubkeys from eth2spec.test.helpers.attester_slashings import get_valid_attester_slashing, get_indexed_attestation_participants from eth2spec.test.helpers.proposer_slashings import get_valid_proposer_slashing -from eth2spec.test.helpers.attestations import get_valid_attestation +from eth2spec.test.helpers.attestations import get_valid_attestation, fill_block_shard_transitions_by_attestations from eth2spec.test.helpers.deposits import prepare_state_and_deposit -from eth2spec.test.context import spec_state_test, with_all_phases, expect_assertion_error, always_bls, with_phases +from eth2spec.test.context import ( + spec_state_test, with_all_phases, expect_assertion_error, always_bls, with_phases, + PHASE1 +) @with_all_phases @@ -420,12 +423,14 @@ def test_attestation(spec, state): yield 'pre', state - attestation = get_valid_attestation(spec, state, signed=True) + attestation = get_valid_attestation(spec, state, signed=True, on_time=True) # Add to state via block transition pre_current_attestations_len = len(state.current_epoch_attestations) attestation_block = build_empty_block(spec, state, state.slot + spec.MIN_ATTESTATION_INCLUSION_DELAY) attestation_block.body.attestations.append(attestation) + if spec.fork == PHASE1: + fill_block_shard_transitions_by_attestations(spec, state, attestation_block) signed_attestation_block = state_transition_and_sign_block(spec, state, attestation_block) assert len(state.current_epoch_attestations) == pre_current_attestations_len + 1 diff --git a/tests/core/pyspec/eth2spec/test/phase_1/block_processing/test_process_crosslink.py b/tests/core/pyspec/eth2spec/test/phase_1/block_processing/test_process_crosslink.py index 6d1e65678b..912770da86 100644 --- a/tests/core/pyspec/eth2spec/test/phase_1/block_processing/test_process_crosslink.py +++ b/tests/core/pyspec/eth2spec/test/phase_1/block_processing/test_process_crosslink.py @@ -1,101 +1,78 @@ from eth2spec.test.context import ( + PHASE0, with_all_phases_except, spec_state_test, always_bls, ) -from eth2spec.test.helpers.attestations import ( - get_valid_on_time_attestation, +from eth2spec.test.helpers.crosslinks import run_crosslinks_processing +from eth2spec.test.helpers.shard_block import ( + build_attestation_with_shard_transition, + build_shard_block, + build_shard_transitions_till_slot, ) -from eth2spec.test.helpers.crosslinks import ( - run_crosslinks_processing, -) -from eth2spec.test.helpers.shard_block import build_shard_block -from eth2spec.test.helpers.state import next_epoch, next_slot, next_slots +from eth2spec.test.helpers.state import next_epoch, next_slot, transition_to -@with_all_phases_except(['phase0']) -@spec_state_test -@always_bls -def test_basic_crosslinks(spec, state): +def run_basic_crosslink_tests(spec, state, target_len_offset_slot): next_epoch(spec, state) next_epoch(spec, state) state = spec.upgrade_to_phase1(state) next_slot(spec, state) + # At the beginning, let `x = state.slot`, `state.shard_states[shard].slot == x - 1` + slot_x = state.slot committee_index = spec.CommitteeIndex(0) shard = spec.compute_shard_from_committee_index(state, committee_index, state.slot) - body = b'1' * spec.MAX_SHARD_BLOCK_SIZE - shard_block = build_shard_block(spec, state, shard, body=body, slot=state.slot, signed=True) - shard_blocks = [shard_block] + assert state.shard_states[shard].slot == slot_x - 1 - next_slot(spec, state) - - shard_transition = spec.get_shard_transition(state, shard, shard_blocks) - shard_transitions = [spec.ShardTransition()] * len(state.shard_states) - shard_transitions[shard] = shard_transition + # Create SignedShardBlock at slot `shard_state.slot + 1` -> x + body = b'\x56' * spec.MAX_SHARD_BLOCK_SIZE + shard_block = build_shard_block(spec, state, shard, body=body, signed=True) + shard_blocks = [shard_block] - attestation = get_valid_on_time_attestation( + # Attester creates `attestation` at slot x + # Use temporary next state to get ShardTransition of shard block + shard_transitions = build_shard_transitions_till_slot( spec, state, - slot=state.slot, + shards=[shard, ], + shard_blocks={shard: shard_blocks}, + target_len_offset_slot=target_len_offset_slot, + ) + shard_transition = shard_transitions[shard] + attestation = build_attestation_with_shard_transition( + spec, + state, + slot=slot_x + target_len_offset_slot - 1, index=committee_index, + target_len_offset_slot=target_len_offset_slot, shard_transition=shard_transition, - signed=True, ) - attestations = [attestation] - pre_gasprice = state.shard_states[shard].gasprice - offset_slots = spec.get_offset_slots(state, shard) - assert len(offset_slots) == 1 - yield from run_crosslinks_processing(spec, state, shard_transitions, attestations) + transition_to(spec, state, state.slot + spec.MIN_ATTESTATION_INCLUSION_DELAY) + pre_shard_state = state.shard_states[shard] + yield from run_crosslinks_processing(spec, state, shard_transitions, [attestation]) + # After state transition, + assert state.slot == slot_x + target_len_offset_slot shard_state = state.shard_states[shard] - assert shard_state.slot == offset_slots[-1] - assert shard_state.latest_block_root == shard_block.message.hash_tree_root() + assert shard_state != pre_shard_state assert shard_state == shard_transition.shard_states[len(shard_transition.shard_states) - 1] - assert shard_state.gasprice > pre_gasprice + + if target_len_offset_slot == 1: + assert shard_state.gasprice > pre_gasprice -@with_all_phases_except(['phase0']) +@with_all_phases_except([PHASE0]) @spec_state_test @always_bls -def test_multiple_offset_slots(spec, state): - next_epoch(spec, state) - next_epoch(spec, state) - state = spec.upgrade_to_phase1(state) - next_slot(spec, state) - - committee_index = spec.CommitteeIndex(0) - shard = spec.compute_shard_from_committee_index(state, committee_index, state.slot) - body = b'1' * spec.MAX_SHARD_BLOCK_SIZE - shard_block = build_shard_block(spec, state, shard, body=body, slot=state.slot, signed=True) - shard_blocks = [shard_block] - - next_slots(spec, state, 3) - - shard_transition = spec.get_shard_transition(state, shard, shard_blocks) - shard_transitions = [spec.ShardTransition()] * len(state.shard_states) - shard_transitions[shard] = shard_transition - - attestation = get_valid_on_time_attestation( - spec, - state, - slot=state.slot, - index=committee_index, - shard_transition=shard_transition, - signed=True, - ) - attestations = [attestation] - - pre_gasprice = state.shard_states[shard].gasprice - offset_slots = spec.get_offset_slots(state, shard) - assert len(offset_slots) == 3 +def test_basic_crosslinks(spec, state): + run_basic_crosslink_tests(spec, state, target_len_offset_slot=1) - yield from run_crosslinks_processing(spec, state, shard_transitions, attestations) - shard_state = state.shard_states[shard] - assert shard_state.slot == offset_slots[-1] - assert shard_state.latest_block_root == shard_block.message.hash_tree_root() - assert shard_state == shard_transition.shard_states[len(shard_transition.shard_states) - 1] - assert shard_state.gasprice > pre_gasprice +@with_all_phases_except([PHASE0]) +@spec_state_test +@always_bls +def test_multiple_offset_slots(spec, state): + run_basic_crosslink_tests(spec, state, target_len_offset_slot=3) diff --git a/tests/core/pyspec/eth2spec/test/phase_1/sanity/__init__.py b/tests/core/pyspec/eth2spec/test/phase_1/sanity/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/tests/core/pyspec/eth2spec/test/phase_1/sanity/test_blocks.py b/tests/core/pyspec/eth2spec/test/phase_1/sanity/test_blocks.py new file mode 100644 index 0000000000..168915d024 --- /dev/null +++ b/tests/core/pyspec/eth2spec/test/phase_1/sanity/test_blocks.py @@ -0,0 +1,141 @@ +from eth2spec.test.context import ( + PHASE0, + with_all_phases_except, + spec_state_test, + always_bls, +) +from eth2spec.test.helpers.block import build_empty_block +from eth2spec.test.helpers.shard_block import ( + build_attestation_with_shard_transition, + build_shard_block, + build_shard_transitions_till_slot, +) +from eth2spec.test.helpers.state import next_epoch, next_slot, state_transition_and_sign_block + + +@with_all_phases_except([PHASE0]) +@spec_state_test +@always_bls +def test_process_beacon_block_with_normal_shard_transition(spec, state): + next_epoch(spec, state) + next_epoch(spec, state) + state = spec.upgrade_to_phase1(state) + next_slot(spec, state) + + target_len_offset_slot = 1 + + # At the beginning, let `x = state.slot`, `state.shard_states[shard].slot == x - 1` + slot_x = state.slot + committee_index = spec.CommitteeIndex(0) + shard = spec.compute_shard_from_committee_index(state, committee_index, state.slot) + assert state.shard_states[shard].slot == slot_x - 1 + + # Create SignedShardBlock at slot `shard_state.slot + 1` -> x + body = b'\x56' * spec.MAX_SHARD_BLOCK_SIZE + shard_block = build_shard_block(spec, state, shard, body=body, signed=True) + shard_blocks = [shard_block] + + # Attester creates `attestation` at slot x + # Use temporary next state to get ShardTransition of shard block + shard_transitions = build_shard_transitions_till_slot( + spec, + state, + shards=[shard, ], + shard_blocks={shard: shard_blocks}, + target_len_offset_slot=target_len_offset_slot, + ) + shard_transition = shard_transitions[shard] + attestation = build_attestation_with_shard_transition( + spec, + state, + slot=slot_x + target_len_offset_slot - 1, + index=committee_index, + target_len_offset_slot=target_len_offset_slot, + shard_transition=shard_transition, + ) + pre_gasprice = state.shard_states[shard].gasprice + + # Propose beacon block at slot `x + 1` + pre_shard_state = state.shard_states[shard] + beacon_block_1 = build_empty_block(spec, state, slot=slot_x + target_len_offset_slot) + beacon_block_1.body.attestations = [attestation] + beacon_block_1.body.shard_transitions = shard_transitions + assert ( + beacon_block_1.slot == slot_x + target_len_offset_slot + == shard_transition.shard_states[0].slot + target_len_offset_slot + ) + state_transition_and_sign_block(spec, state, beacon_block_1) + + # After state transition + assert state.slot == slot_x + target_len_offset_slot + shard_state = state.shard_states[shard] + # latest_block_root has changed + assert shard_state.latest_block_root == shard_block.message.hash_tree_root() + assert shard_state != pre_shard_state + assert shard_state == shard_transition.shard_states[len(shard_transition.shard_states) - 1] + + if target_len_offset_slot == 1 and len(shard_blocks) > 0: + assert shard_state.gasprice > pre_gasprice + + +@with_all_phases_except([PHASE0]) +@spec_state_test +@always_bls +def test_process_beacon_block_with_empty_proposal_transition(spec, state): + next_epoch(spec, state) + next_epoch(spec, state) + state = spec.upgrade_to_phase1(state) + next_slot(spec, state) + + target_len_offset_slot = 1 + + # At the beginning, let `x = state.slot`, `state.shard_states[shard].slot == x - 1` + slot_x = state.slot + committee_index = spec.CommitteeIndex(0) + shard = spec.compute_shard_from_committee_index(state, committee_index, state.slot) + assert state.shard_states[shard].slot == slot_x - 1 + + # No new shard block + shard_blocks = [] + + # Attester creates `attestation` at slot x + # Use temporary next state to get ShardTransition of shard block + shard_transitions = build_shard_transitions_till_slot( + spec, + state, + shards=[shard, ], + shard_blocks={shard: shard_blocks}, + target_len_offset_slot=target_len_offset_slot, + ) + shard_transition = shard_transitions[shard] + attestation = build_attestation_with_shard_transition( + spec, + state, + slot=slot_x + target_len_offset_slot - 1, + index=committee_index, + target_len_offset_slot=target_len_offset_slot, + shard_transition=shard_transition, + ) + pre_gasprice = state.shard_states[shard].gasprice + + # Propose beacon block at slot `x + 1` + pre_shard_state = state.shard_states[shard] + beacon_block_1 = build_empty_block(spec, state, slot=slot_x + target_len_offset_slot) + beacon_block_1.body.attestations = [attestation] + beacon_block_1.body.shard_transitions = shard_transitions + assert ( + beacon_block_1.slot == slot_x + target_len_offset_slot + == shard_transition.shard_states[0].slot + target_len_offset_slot + ) + state_transition_and_sign_block(spec, state, beacon_block_1) + + # After state transition + assert state.slot == slot_x + target_len_offset_slot + shard_state = state.shard_states[shard] + # latest_block_root hasn't changed + assert shard_state.latest_block_root == pre_shard_state.latest_block_root + assert shard_state != pre_shard_state + assert shard_state == shard_transition.shard_states[len(shard_transition.shard_states) - 1] + + if target_len_offset_slot == 1 and len(shard_blocks) > 0: + assert shard_state.gasprice > pre_gasprice From e758fb76c2ee67afa63a581c5b03022c6a739875 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Sat, 2 May 2020 01:39:03 +0800 Subject: [PATCH 39/44] Check `head_shard_root` of all `transition_attestations` --- specs/phase1/beacon-chain.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specs/phase1/beacon-chain.md b/specs/phase1/beacon-chain.md index 611cc19552..f58a95fb76 100644 --- a/specs/phase1/beacon-chain.md +++ b/specs/phase1/beacon-chain.md @@ -788,6 +788,9 @@ def process_crosslink_for_shard(state: BeaconState, for attestation in transition_attestations: participants = get_attesting_indices(state, attestation.data, attestation.aggregation_bits) transition_participants = transition_participants.union(participants) + assert attestation.data.head_shard_root == shard_transition.shard_data_roots[ + len(shard_transition.shard_data_roots) - 1 + ] enough_online_stake = ( get_total_balance(state, online_indices.intersection(transition_participants)) * 3 >= @@ -799,9 +802,6 @@ def process_crosslink_for_shard(state: BeaconState, # Attestation <-> shard transition consistency assert shard_transition_root == hash_tree_root(shard_transition) - assert attestation.data.head_shard_root == shard_transition.shard_data_roots[ - len(shard_transition.shard_data_roots) - 1 - ] # Apply transition apply_shard_transition(state, shard, shard_transition) From ff850251130c807b1403a5c9a9cfd0ef17a28a01 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Sat, 2 May 2020 01:56:25 +0800 Subject: [PATCH 40/44] PR feedback from terence --- specs/phase1/beacon-chain.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/specs/phase1/beacon-chain.md b/specs/phase1/beacon-chain.md index f58a95fb76..c9d8eeb54c 100644 --- a/specs/phase1/beacon-chain.md +++ b/specs/phase1/beacon-chain.md @@ -735,8 +735,12 @@ def apply_shard_transition(state: BeaconState, shard: Shard, transition: ShardTr shard_parent_root = state.shard_states[shard].latest_block_root for i in range(len(offset_slots)): shard_block_length = transition.shard_block_lengths[i] - is_empty_proposal = shard_block_length == 0 shard_state = transition.shard_states[i] + # Verify correct calculation of gas prices and slots + assert shard_state.gasprice == compute_updated_gasprice(prev_gasprice, shard_block_length) + assert shard_state.slot == offset_slots[i] + # Collect the non-empty proposals result + is_empty_proposal = shard_block_length == 0 if not is_empty_proposal: proposal_index = get_shard_proposer_index(state, offset_slots[i], shard) # Reconstruct shard headers @@ -751,9 +755,6 @@ def apply_shard_transition(state: BeaconState, shard: Shard, transition: ShardTr headers.append(header) proposers.append(proposal_index) - # Verify correct calculation of gas prices and slots - assert shard_state.gasprice == compute_updated_gasprice(prev_gasprice, shard_block_length) - assert shard_state.slot == offset_slots[i] prev_gasprice = shard_state.gasprice pubkeys = [state.validators[proposer].pubkey for proposer in proposers] From 977cd73379e383cb4f1b08d342f0d790ab513915 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Sat, 2 May 2020 01:57:34 +0800 Subject: [PATCH 41/44] Refactor the tests --- .../eth2spec/test/helpers/shard_block.py | 33 ++-- .../pyspec/eth2spec/test/helpers/state.py | 10 ++ .../test_process_crosslink.py | 47 +++--- .../test/phase_1/sanity/test_blocks.py | 152 +++++++----------- 4 files changed, 105 insertions(+), 137 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/helpers/shard_block.py b/tests/core/pyspec/eth2spec/test/helpers/shard_block.py index 922f4d748c..ef65d24279 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/shard_block.py +++ b/tests/core/pyspec/eth2spec/test/helpers/shard_block.py @@ -1,6 +1,6 @@ from eth2spec.test.helpers.attestations import get_valid_on_time_attestation from eth2spec.test.helpers.block import get_state_and_beacon_parent_root_at_slot -from eth2spec.test.helpers.state import next_slots +from eth2spec.test.helpers.state import transition_to from eth2spec.test.helpers.keys import privkeys from eth2spec.utils import bls from eth2spec.utils.bls import only_with_bls @@ -51,18 +51,17 @@ def build_shard_block(spec, return signed_block -def build_shard_transitions_till_slot(spec, state, shards, shard_blocks, target_len_offset_slot): - state = state.copy() - next_slots(spec, state, target_len_offset_slot) +def build_shard_transitions_till_slot(spec, state, shard_blocks, on_time_slot): + temp_state = state.copy() + transition_to(spec, temp_state, on_time_slot) shard_transitions = [spec.ShardTransition()] * spec.MAX_SHARDS - for shard in shards: - offset_slots = spec.get_offset_slots(state, shard) + for shard, blocks in shard_blocks.items(): + offset_slots = spec.get_offset_slots(temp_state, shard) len_offset_slots = len(offset_slots) - assert len_offset_slots == target_len_offset_slot - shard_blocks_of_shard = shard_blocks[shard] - shard_transition = spec.get_shard_transition(state, shard, shard_blocks_of_shard) - if len(shard_blocks_of_shard) > 0: - shard_block_root = shard_blocks_of_shard[-1].message.hash_tree_root() + assert len_offset_slots == on_time_slot - state.shard_states[shard].slot - 1 + shard_transition = spec.get_shard_transition(temp_state, shard, blocks) + if len(blocks) > 0: + shard_block_root = blocks[-1].message.hash_tree_root() assert shard_transition.shard_states[len_offset_slots - 1].latest_block_root == shard_block_root assert shard_transition.shard_states[len_offset_slots - 1].slot == offset_slots[-1] shard_transitions[shard] = shard_transition @@ -70,19 +69,17 @@ def build_shard_transitions_till_slot(spec, state, shards, shard_blocks, target_ return shard_transitions -def build_attestation_with_shard_transition(spec, state, slot, index, target_len_offset_slot, shard_transition=None): - state = state.copy() - next_slots(spec, state, target_len_offset_slot) +def build_attestation_with_shard_transition(spec, state, index, on_time_slot, shard_transition=None): + temp_state = state.copy() + transition_to(spec, temp_state, on_time_slot - 1) attestation = get_valid_on_time_attestation( spec, - state, - slot=slot, + temp_state, index=index, shard_transition=shard_transition, signed=True, ) - assert attestation.data.slot == slot + assert attestation.data.slot == temp_state.slot if shard_transition is not None: - assert target_len_offset_slot == len(shard_transition.shard_states) assert attestation.data.shard_transition_root == shard_transition.hash_tree_root() return attestation diff --git a/tests/core/pyspec/eth2spec/test/helpers/state.py b/tests/core/pyspec/eth2spec/test/helpers/state.py index 46a7ce2b5e..f08f226028 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/state.py +++ b/tests/core/pyspec/eth2spec/test/helpers/state.py @@ -30,6 +30,16 @@ def transition_to(spec, state, slot): assert state.slot == slot +def transition_to_valid_shard_slot(spec, state): + """ + Transition to slot `spec.PHASE_1_GENESIS_SLOT + 1` and fork at `spec.PHASE_1_GENESIS_SLOT`. + """ + transition_to(spec, state, spec.PHASE_1_GENESIS_SLOT) + state = spec.upgrade_to_phase1(state) # `upgrade_to_phase1` is a pure function + next_slot(spec, state) + return state + + def next_epoch(spec, state): """ Transition to the start slot of the next epoch diff --git a/tests/core/pyspec/eth2spec/test/phase_1/block_processing/test_process_crosslink.py b/tests/core/pyspec/eth2spec/test/phase_1/block_processing/test_process_crosslink.py index 912770da86..1f066b3442 100644 --- a/tests/core/pyspec/eth2spec/test/phase_1/block_processing/test_process_crosslink.py +++ b/tests/core/pyspec/eth2spec/test/phase_1/block_processing/test_process_crosslink.py @@ -10,69 +10,64 @@ build_shard_block, build_shard_transitions_till_slot, ) -from eth2spec.test.helpers.state import next_epoch, next_slot, transition_to +from eth2spec.test.helpers.state import transition_to, transition_to_valid_shard_slot -def run_basic_crosslink_tests(spec, state, target_len_offset_slot): - next_epoch(spec, state) - next_epoch(spec, state) - state = spec.upgrade_to_phase1(state) - next_slot(spec, state) - +def run_basic_crosslink_tests(spec, state, target_len_offset_slot, valid=True): + state = transition_to_valid_shard_slot(spec, state) # At the beginning, let `x = state.slot`, `state.shard_states[shard].slot == x - 1` slot_x = state.slot committee_index = spec.CommitteeIndex(0) shard = spec.compute_shard_from_committee_index(state, committee_index, state.slot) assert state.shard_states[shard].slot == slot_x - 1 - # Create SignedShardBlock at slot `shard_state.slot + 1` -> x + # Create SignedShardBlock body = b'\x56' * spec.MAX_SHARD_BLOCK_SIZE shard_block = build_shard_block(spec, state, shard, body=body, signed=True) shard_blocks = [shard_block] - - # Attester creates `attestation` at slot x - # Use temporary next state to get ShardTransition of shard block + # Create a shard_transitions that would be included at beacon block `state.slot + target_len_offset_slot` shard_transitions = build_shard_transitions_till_slot( spec, state, - shards=[shard, ], shard_blocks={shard: shard_blocks}, - target_len_offset_slot=target_len_offset_slot, + on_time_slot=state.slot + target_len_offset_slot, ) shard_transition = shard_transitions[shard] + # Create an attestation that would be included at beacon block `state.slot + target_len_offset_slot` attestation = build_attestation_with_shard_transition( spec, state, - slot=slot_x + target_len_offset_slot - 1, index=committee_index, - target_len_offset_slot=target_len_offset_slot, + on_time_slot=state.slot + target_len_offset_slot, shard_transition=shard_transition, ) pre_gasprice = state.shard_states[shard].gasprice - transition_to(spec, state, state.slot + spec.MIN_ATTESTATION_INCLUSION_DELAY) + transition_to(spec, state, state.slot + target_len_offset_slot) pre_shard_state = state.shard_states[shard] - yield from run_crosslinks_processing(spec, state, shard_transitions, [attestation]) - # After state transition, - assert state.slot == slot_x + target_len_offset_slot - shard_state = state.shard_states[shard] - assert shard_state != pre_shard_state - assert shard_state == shard_transition.shard_states[len(shard_transition.shard_states) - 1] + yield from run_crosslinks_processing(spec, state, shard_transitions, [attestation], valid=valid) + + if valid: + # After state transition, + assert state.slot == slot_x + target_len_offset_slot + shard_state = state.shard_states[shard] + assert shard_state != pre_shard_state + assert shard_state == shard_transition.shard_states[len(shard_transition.shard_states) - 1] - if target_len_offset_slot == 1: - assert shard_state.gasprice > pre_gasprice + if target_len_offset_slot == 1: + assert shard_state.gasprice > pre_gasprice @with_all_phases_except([PHASE0]) @spec_state_test @always_bls def test_basic_crosslinks(spec, state): - run_basic_crosslink_tests(spec, state, target_len_offset_slot=1) + yield from run_basic_crosslink_tests(spec, state, target_len_offset_slot=1, valid=True) @with_all_phases_except([PHASE0]) @spec_state_test @always_bls def test_multiple_offset_slots(spec, state): - run_basic_crosslink_tests(spec, state, target_len_offset_slot=3) + yield from run_basic_crosslink_tests(spec, state, target_len_offset_slot=3, valid=True) diff --git a/tests/core/pyspec/eth2spec/test/phase_1/sanity/test_blocks.py b/tests/core/pyspec/eth2spec/test/phase_1/sanity/test_blocks.py index 168915d024..60af35d45d 100644 --- a/tests/core/pyspec/eth2spec/test/phase_1/sanity/test_blocks.py +++ b/tests/core/pyspec/eth2spec/test/phase_1/sanity/test_blocks.py @@ -1,3 +1,5 @@ +from typing import Dict, Sequence + from eth2spec.test.context import ( PHASE0, with_all_phases_except, @@ -10,69 +12,74 @@ build_shard_block, build_shard_transitions_till_slot, ) -from eth2spec.test.helpers.state import next_epoch, next_slot, state_transition_and_sign_block +from eth2spec.test.helpers.state import state_transition_and_sign_block, transition_to_valid_shard_slot + + +def run_beacon_block_with_shard_blocks(spec, state, shard_blocks, target_len_offset_slot, committee_index, valid=True): + shard_transitions = build_shard_transitions_till_slot( + spec, state, shard_blocks, on_time_slot=state.slot + target_len_offset_slot + ) + attestations = [ + build_attestation_with_shard_transition( + spec, + state, + on_time_slot=state.slot + target_len_offset_slot, + index=committee_index, + shard_transition=shard_transitions[shard], + ) + for shard in shard_blocks.keys() + ] + + # Propose beacon block at slot `x + 1` + beacon_block = build_empty_block(spec, state, slot=state.slot + target_len_offset_slot) + beacon_block.body.attestations = attestations + beacon_block.body.shard_transitions = shard_transitions + + pre_shard_states = state.shard_states.copy() + yield 'pre', state.copy() + yield 'block', beacon_block + state_transition_and_sign_block(spec, state, beacon_block) + if valid: + yield 'post', state + else: + yield 'post', None + return + + for shard in range(spec.get_active_shard_count(state)): + post_shard_state = state.shard_states[shard] + if shard in shard_blocks: + # Shard state has been changed to state_transition result + assert post_shard_state == shard_transitions[shard].shard_states[ + len(shard_transitions[shard].shard_states) - 1 + ] + assert beacon_block.slot == shard_transitions[shard].shard_states[0].slot + target_len_offset_slot + assert post_shard_state.slot == state.slot - 1 + if len(shard_blocks[shard]) == 0: + # `latest_block_root` is the same + assert post_shard_state.latest_block_root == pre_shard_states[shard].latest_block_root @with_all_phases_except([PHASE0]) @spec_state_test @always_bls def test_process_beacon_block_with_normal_shard_transition(spec, state): - next_epoch(spec, state) - next_epoch(spec, state) - state = spec.upgrade_to_phase1(state) - next_slot(spec, state) + state = transition_to_valid_shard_slot(spec, state) target_len_offset_slot = 1 - - # At the beginning, let `x = state.slot`, `state.shard_states[shard].slot == x - 1` - slot_x = state.slot committee_index = spec.CommitteeIndex(0) shard = spec.compute_shard_from_committee_index(state, committee_index, state.slot) - assert state.shard_states[shard].slot == slot_x - 1 + assert state.shard_states[shard].slot == state.slot - 1 + + pre_gasprice = state.shard_states[shard].gasprice - # Create SignedShardBlock at slot `shard_state.slot + 1` -> x + # Create SignedShardBlock at slot `shard_state.slot + 1` body = b'\x56' * spec.MAX_SHARD_BLOCK_SIZE shard_block = build_shard_block(spec, state, shard, body=body, signed=True) - shard_blocks = [shard_block] - - # Attester creates `attestation` at slot x - # Use temporary next state to get ShardTransition of shard block - shard_transitions = build_shard_transitions_till_slot( - spec, - state, - shards=[shard, ], - shard_blocks={shard: shard_blocks}, - target_len_offset_slot=target_len_offset_slot, - ) - shard_transition = shard_transitions[shard] - attestation = build_attestation_with_shard_transition( - spec, - state, - slot=slot_x + target_len_offset_slot - 1, - index=committee_index, - target_len_offset_slot=target_len_offset_slot, - shard_transition=shard_transition, - ) - pre_gasprice = state.shard_states[shard].gasprice + shard_blocks: Dict[spec.Shard, Sequence[spec.SignedShardBlock]] = {shard: [shard_block]} - # Propose beacon block at slot `x + 1` - pre_shard_state = state.shard_states[shard] - beacon_block_1 = build_empty_block(spec, state, slot=slot_x + target_len_offset_slot) - beacon_block_1.body.attestations = [attestation] - beacon_block_1.body.shard_transitions = shard_transitions - assert ( - beacon_block_1.slot == slot_x + target_len_offset_slot - == shard_transition.shard_states[0].slot + target_len_offset_slot - ) - state_transition_and_sign_block(spec, state, beacon_block_1) + yield from run_beacon_block_with_shard_blocks(spec, state, shard_blocks, target_len_offset_slot, committee_index) - # After state transition - assert state.slot == slot_x + target_len_offset_slot shard_state = state.shard_states[shard] - # latest_block_root has changed - assert shard_state.latest_block_root == shard_block.message.hash_tree_root() - assert shard_state != pre_shard_state - assert shard_state == shard_transition.shard_states[len(shard_transition.shard_states) - 1] if target_len_offset_slot == 1 and len(shard_blocks) > 0: assert shard_state.gasprice > pre_gasprice @@ -82,60 +89,19 @@ def test_process_beacon_block_with_normal_shard_transition(spec, state): @spec_state_test @always_bls def test_process_beacon_block_with_empty_proposal_transition(spec, state): - next_epoch(spec, state) - next_epoch(spec, state) - state = spec.upgrade_to_phase1(state) - next_slot(spec, state) + state = transition_to_valid_shard_slot(spec, state) target_len_offset_slot = 1 - - # At the beginning, let `x = state.slot`, `state.shard_states[shard].slot == x - 1` - slot_x = state.slot committee_index = spec.CommitteeIndex(0) shard = spec.compute_shard_from_committee_index(state, committee_index, state.slot) - assert state.shard_states[shard].slot == slot_x - 1 + assert state.shard_states[shard].slot == state.slot - 1 # No new shard block - shard_blocks = [] + shard_blocks = {} - # Attester creates `attestation` at slot x - # Use temporary next state to get ShardTransition of shard block - shard_transitions = build_shard_transitions_till_slot( - spec, - state, - shards=[shard, ], - shard_blocks={shard: shard_blocks}, - target_len_offset_slot=target_len_offset_slot, - ) - shard_transition = shard_transitions[shard] - attestation = build_attestation_with_shard_transition( - spec, - state, - slot=slot_x + target_len_offset_slot - 1, - index=committee_index, - target_len_offset_slot=target_len_offset_slot, - shard_transition=shard_transition, - ) pre_gasprice = state.shard_states[shard].gasprice - # Propose beacon block at slot `x + 1` - pre_shard_state = state.shard_states[shard] - beacon_block_1 = build_empty_block(spec, state, slot=slot_x + target_len_offset_slot) - beacon_block_1.body.attestations = [attestation] - beacon_block_1.body.shard_transitions = shard_transitions - assert ( - beacon_block_1.slot == slot_x + target_len_offset_slot - == shard_transition.shard_states[0].slot + target_len_offset_slot - ) - state_transition_and_sign_block(spec, state, beacon_block_1) - - # After state transition - assert state.slot == slot_x + target_len_offset_slot - shard_state = state.shard_states[shard] - # latest_block_root hasn't changed - assert shard_state.latest_block_root == pre_shard_state.latest_block_root - assert shard_state != pre_shard_state - assert shard_state == shard_transition.shard_states[len(shard_transition.shard_states) - 1] + yield from run_beacon_block_with_shard_blocks(spec, state, shard_blocks, target_len_offset_slot, committee_index) if target_len_offset_slot == 1 and len(shard_blocks) > 0: - assert shard_state.gasprice > pre_gasprice + assert state.shard_states[shard].gasprice > pre_gasprice From b43e24acb610ae75497fdb10633a110bf6cddcfc Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Sat, 2 May 2020 02:04:46 +0800 Subject: [PATCH 42/44] specs/phase1/fraud-proofs.md -> specs/phase1/shard-transition.md --- specs/phase1/{fraud-proofs.md => shard-transition.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename specs/phase1/{fraud-proofs.md => shard-transition.md} (100%) diff --git a/specs/phase1/fraud-proofs.md b/specs/phase1/shard-transition.md similarity index 100% rename from specs/phase1/fraud-proofs.md rename to specs/phase1/shard-transition.md From 4558c7db4ed4600895909420b4d0629cc93e0d56 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Sat, 2 May 2020 02:22:31 +0800 Subject: [PATCH 43/44] Reorg the file structure --- setup.py | 2 +- specs/phase1/shard-transition.md | 107 ++++++++++++++++--------------- 2 files changed, 56 insertions(+), 53 deletions(-) diff --git a/setup.py b/setup.py index d1c62fb72f..e0d6561dd6 100644 --- a/setup.py +++ b/setup.py @@ -375,7 +375,7 @@ def finalize_options(self): specs/phase0/fork-choice.md specs/phase1/custody-game.md specs/phase1/beacon-chain.md - specs/phase1/fraud-proofs.md + specs/phase1/shard-transition.md specs/phase1/fork-choice.md specs/phase1/phase1-fork.md """ diff --git a/specs/phase1/shard-transition.md b/specs/phase1/shard-transition.md index c676e142a4..a8de508fb9 100644 --- a/specs/phase1/shard-transition.md +++ b/specs/phase1/shard-transition.md @@ -1,38 +1,70 @@ +# Ethereum 2.0 Phase 1 -- Shard Transition and Fraud Proofs + +**Notice**: This document is a work-in-progress for researchers and implementers. + +## Table of contents + **Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* -- [Ethereum 2.0 Phase 1 -- Shard Transition and Fraud Proofs](#ethereum-20-phase-1----shard-transition-and-fraud-proofs) - - [Table of contents](#table-of-contents) - - [Introduction](#introduction) - - [Fraud proofs](#fraud-proofs) - - [Shard state transition function](#shard-state-transition-function) - - [Verifying the proof](#verifying-the-proof) - - [Honest committee member behavior](#honest-committee-member-behavior) - - [Helper functions](#helper-functions) - - [Make attestations](#make-attestations) +- [Introduction](#introduction) +- [Helper functions](#helper-functions) + - [Misc](#misc) + - [Shard block verification functions](#shard-block-verification-functions) +- [Shard state transition](#shard-state-transition) +- [Fraud proofs](#fraud-proofs) + - [Verifying the proof](#verifying-the-proof) +- [Honest committee member behavior](#honest-committee-member-behavior) + - [Helper functions](#helper-functions-1) + - [Make attestations](#make-attestations) -# Ethereum 2.0 Phase 1 -- Shard Transition and Fraud Proofs - -**Notice**: This document is a work-in-progress for researchers and implementers. +## Introduction -## Table of contents +This document describes the shard transition function and fraud proofs as part of Phase 1 of Ethereum 2.0. - +## Helper functions - TODO +### Misc - +```python +def compute_shard_transition_digest(beacon_state: BeaconState, + shard_state: ShardState, + beacon_parent_root: Root, + shard_body_root: Root) -> Bytes32: + # TODO: use SSZ hash tree root + return hash( + hash_tree_root(shard_state) + beacon_parent_root + shard_body_root + ) +``` -## Introduction +### Shard block verification functions -This document describes the shard transition function and fraud proofs as part of Phase 1 of Ethereum 2.0. +```python +def verify_shard_block_message(beacon_state: BeaconState, + shard_state: ShardState, + block: ShardBlock, + slot: Slot, + shard: Shard) -> bool: + assert block.shard_parent_root == shard_state.latest_block_root + assert block.slot == slot + assert block.proposer_index == get_shard_proposer_index(beacon_state, slot, shard) + assert 0 < len(block.body) <= MAX_SHARD_BLOCK_SIZE + return True +``` -## Fraud proofs +```python +def verify_shard_block_signature(beacon_state: BeaconState, + signed_block: SignedShardBlock) -> bool: + proposer = beacon_state.validators[signed_block.message.proposer_index] + domain = get_domain(beacon_state, DOMAIN_SHARD_PROPOSAL, compute_epoch_at_slot(signed_block.message.slot)) + signing_root = compute_signing_root(signed_block.message, domain) + return bls.Verify(proposer.pubkey, signing_root, signed_block.signature) +``` -### Shard state transition function +## Shard state transition ```python def shard_state_transition(beacon_state: BeaconState, @@ -56,6 +88,8 @@ def shard_state_transition(beacon_state: BeaconState, shard_state.latest_block_root = latest_block_root ``` +We have a pure function `get_post_shard_state` for describing the fraud proof verification and honest validator behavior. + ```python def get_post_shard_state(beacon_state: BeaconState, shard_state: ShardState, @@ -68,38 +102,7 @@ def get_post_shard_state(beacon_state: BeaconState, return post_state ``` -```python -def compute_shard_transition_digest(beacon_state: BeaconState, - shard_state: ShardState, - beacon_parent_root: Root, - shard_body_root: Root) -> Bytes32: - # TODO: use SSZ hash tree root - return hash( - hash_tree_root(shard_state) + beacon_parent_root + shard_body_root - ) -``` - -```python -def verify_shard_block_message(beacon_state: BeaconState, - shard_state: ShardState, - block: ShardBlock, - slot: Slot, - shard: Shard) -> bool: - assert block.shard_parent_root == shard_state.latest_block_root - assert block.slot == slot - assert block.proposer_index == get_shard_proposer_index(beacon_state, slot, shard) - assert 0 < len(block.body) <= MAX_SHARD_BLOCK_SIZE - return True -``` - -```python -def verify_shard_block_signature(beacon_state: BeaconState, - signed_block: SignedShardBlock) -> bool: - proposer = beacon_state.validators[signed_block.message.proposer_index] - domain = get_domain(beacon_state, DOMAIN_SHARD_PROPOSAL, compute_epoch_at_slot(signed_block.message.slot)) - signing_root = compute_signing_root(signed_block.message, domain) - return bls.Verify(proposer.pubkey, signing_root, signed_block.signature) -``` +## Fraud proofs ### Verifying the proof From 7a770186b5ba576bf14ce496dc2b0381d169840e Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Sat, 2 May 2020 02:25:11 +0800 Subject: [PATCH 44/44] Reorg beacon-chain spec a bit --- specs/phase1/beacon-chain.md | 80 ++++++++++++++++++------------------ 1 file changed, 41 insertions(+), 39 deletions(-) diff --git a/specs/phase1/beacon-chain.md b/specs/phase1/beacon-chain.md index c9d8eeb54c..35f6a6425f 100644 --- a/specs/phase1/beacon-chain.md +++ b/specs/phase1/beacon-chain.md @@ -16,8 +16,8 @@ - [Extended `AttestationData`](#extended-attestationdata) - [Extended `Attestation`](#extended-attestation) - [Extended `PendingAttestation`](#extended-pendingattestation) - - [`IndexedAttestation`](#indexedattestation) - - [Extended `AttesterSlashing`](#extended-attesterslashing) + - [Extended `IndexedAttestation`](#extended-indexedattestation) + - [Extended `AttesterSlashing`](#extended-attesterslashing) - [Extended `Validator`](#extended-validator) - [Extended `BeaconBlockBody`](#extended-beaconblockbody) - [Extended `BeaconBlock`](#extended-beaconblock) @@ -52,9 +52,9 @@ - [`get_latest_slot_for_shard`](#get_latest_slot_for_shard) - [`get_offset_slots`](#get_offset_slots) - [Predicates](#predicates) + - [Updated `is_valid_indexed_attestation`](#updated-is_valid_indexed_attestation) - [`is_shard_attestation`](#is_shard_attestation) - [`is_winning_attestation`](#is_winning_attestation) - - [Updated `is_valid_indexed_attestation`](#updated-is_valid_indexed_attestation) - [Block processing](#block-processing) - [Operations](#operations) - [New Attestation processing](#new-attestation-processing) @@ -154,7 +154,7 @@ class PendingAttestation(Container): crosslink_success: boolean ``` -### `IndexedAttestation` +### Extended `IndexedAttestation` ```python class IndexedAttestation(Container): @@ -162,7 +162,7 @@ class IndexedAttestation(Container): attestation: Attestation ``` -#### Extended `AttesterSlashing` +### Extended `AttesterSlashing` Note that the `attestation_1` and `attestation_2` have a new `IndexedAttestation` definition. @@ -557,39 +557,6 @@ def get_offset_slots(state: BeaconState, shard: Shard) -> Sequence[Slot]: ### Predicates -#### `is_shard_attestation` - -```python -def is_shard_attestation(state: BeaconState, - attestation: Attestation, - committee_index: CommitteeIndex) -> bool: - if not ( - attestation.data.index == committee_index - and attestation.data.slot + MIN_ATTESTATION_INCLUSION_DELAY == state.slot - ): - return False - - return True -``` - -#### `is_winning_attestation` - -```python -def is_winning_attestation(state: BeaconState, - attestation: PendingAttestation, - committee_index: CommitteeIndex, - winning_root: Root) -> bool: - """ - Check if ``attestation`` helped contribute to the successful crosslink of - ``winning_root`` formed by ``committee_index`` committee at the current slot. - """ - return ( - attestation.data.slot == state.slot - and attestation.data.index == committee_index - and attestation.data.shard_transition_root == winning_root - ) -``` - #### Updated `is_valid_indexed_attestation` Note that this replaces the Phase 0 `is_valid_indexed_attestation`. @@ -632,6 +599,40 @@ def is_valid_indexed_attestation(state: BeaconState, indexed_attestation: Indexe return bls.AggregateVerify(zip(all_pubkeys, all_signing_roots), signature=attestation.signature) ``` +#### `is_shard_attestation` + +```python +def is_shard_attestation(state: BeaconState, + attestation: Attestation, + committee_index: CommitteeIndex) -> bool: + if not ( + attestation.data.index == committee_index + and attestation.data.slot + MIN_ATTESTATION_INCLUSION_DELAY == state.slot # Must be on-time attestation + # TODO: MIN_ATTESTATION_INCLUSION_DELAY should always be 1 + ): + return False + + return True +``` + +#### `is_winning_attestation` + +```python +def is_winning_attestation(state: BeaconState, + attestation: PendingAttestation, + committee_index: CommitteeIndex, + winning_root: Root) -> bool: + """ + Check if ``attestation`` helped contribute to the successful crosslink of + ``winning_root`` formed by ``committee_index`` committee at the current slot. + """ + return ( + attestation.data.slot == state.slot + and attestation.data.index == committee_index + and attestation.data.shard_transition_root == winning_root + ) +``` + ### Block processing ```python @@ -942,7 +943,8 @@ def process_light_client_signatures(state: BeaconState, block_body: BeaconBlockB signing_root = compute_signing_root(get_block_root_at_slot(state, slot), get_domain(state, DOMAIN_LIGHT_CLIENT, compute_epoch_at_slot(slot))) if len(signer_pubkeys) == 0: - # TODO: Or disallow empty light_client_signature? + # TODO: handle the empty light_client_signature case? + assert block_body.light_client_signature == BLSSignature() return else: assert bls.FastAggregateVerify(signer_pubkeys, signing_root, signature=block_body.light_client_signature)